docker

How to use LUKS data disk encryption in MapR

techhadoop Uncategorized , , , ,

How to use LUKS data disk encryption in MapR

MapR (now part of HPE Ezmeral) supports encryption at various levels, but using LUKS (Linux Unified Key Setup) encryption for data disks is a system-level operation that must be done outside of MapR’s native encryption features. Here’s a step-by-step guide to set up LUKS disk encryption on a MapR node and ensure MapR can access the encrypted disk after unlocking it.

Steps to Set Up LUKS Disk Encryption for MapR Data Disks

1. Identify the Disk to Encrypt

Find the disk you want to encrypt using the lsblk or fdisk command:

bash

lsblk

fdisk -l

For example, if the disk to be encrypted is /dev/sdb, use that in the following steps.

2. Install the Necessary Packages

Ensure you have the required tools to set up LUKS encryption:

sudo apt-get install cryptsetup  # For Ubuntu/Debian

sudo yum install cryptsetup      # For CentOS/RHEL

3. Set Up LUKS Encryption on the Disk

Run the following command to initialize the disk with LUKS encryption:

sudo cryptsetup luksFormat /dev/sdb

You’ll be prompted to confirm the operation and set a passphrase.

⚠️ Warning: This will erase all existing data on the disk.

4. Open and Map the Encrypted Disk

Unlock the encrypted disk and map it to a device:

sudo cryptsetup open /dev/sdb mapr_data_disk

You can verify that the encrypted device is available:

lsblk

5. Format the Encrypted Disk

Format the newly mapped device with a filesystem that MapR supports (typically ext4 or xfs):

sudo mkfs.ext4 /dev/mapper/mapr_data_disk

6. Mount the Encrypted Disk

Create a mount point and mount the encrypted disk:

sudo mkdir -p /opt/mapr/data

sudo mount /dev/mapper/mapr_data_disk /opt/mapr/data

7. Make the Mount Persistent

Edit the /etc/crypttab file to automatically unlock the disk at boot:

echo “mapr_data_disk /dev/sdb none luks” | sudo tee -a /etc/crypttab

Then, add an entry to /etc/fstab to mount the disk automatically after it is unlocked:

echo “/dev/mapper/mapr_data_disk /opt/mapr/data ext4 defaults 0 0” | sudo tee -a /etc/fstab

8. Ensure MapR Can Access the Disk

Make sure the MapR user has the necessary permissions to access the encrypted disk:

sudo chown -R mapr:mapr /opt/mapr/data

9. Test the Setup

Reboot the system to ensure the encrypted disk is unlocked and mounted correctly:

sudo reboot

After the system reboots, verify that the disk is unlocked and mounted:

lsblk

df -h

10. Verify MapR Storage Pools

After the encrypted disk is mounted, add it to the MapR storage pool:

maprcli disk add -server <server_name> -disks /dev/mapper/mapr_data_disk

Additional Considerations

  • Passphrase Management: Consider integrating with a key management system (KMS) to avoid manual passphrase entry.
  • Performance Impact: Encryption may introduce some performance overhead, so test accordingly.
  • Backup Configuration Files: Ensure you back up /etc/crypttab and /etc/fstab for disaster recovery.

Steps to install HPE Ezmeral 7.x on Linux cluster

techhadoop Uncategorized , , , ,

Installing HPE Ezmeral Data Fabric (formerly MapR) version 7.x on a 12-node Linux cluster requires planning and configuration. Here are the detailed steps to install and configure the cluster:

Step 1: Prerequisites

  1. System Requirements:
    • 64-bit Linux (RHEL/CentOS 7 or 8, or equivalent).
    • Minimum hardware for each node:
      • Memory: At least 16GB RAM.
      • CPU: Quad-core or higher.
      • Disk: Minimum of 500GB of storage.
  2. Network Configuration:
    • Assign static IP addresses or hostnames to all 12 nodes.
    • Configure DNS or update /etc/hosts with the IP and hostname mappings.
    • Ensure nodes can communicate with each other via SSH.
  3. Users and Permissions:
    • Create a dedicated user for HPE Ezmeral (e.g., mapr).
    • Grant the user passwordless SSH access across all nodes.
  4. Firewall and SELinux:
    • Disable or configure the firewall to allow required ports.
    • Set SELinux to permissive mode:

sudo setenforce 0

sudo sed -i ‘s/^SELINUX=.*/SELINUX=permissive/’ /etc/selinux/config

  1. Java Installation:
    • Install Java (OpenJDK 11 recommended):

sudo yum install java-11-openjdk -y

Step 2: Download HPE Ezmeral Data Fabric Software

  1. Obtain Software:
    • Download the HPE Ezmeral 7.x installation packages from the official HPE Ezmeral website.
  2. Distribute Packages:
    • Copy the packages to all 12 nodes using scp or a similar tool.

Step 3: Install Core Services

  1. Install the Core Packages:
    • On each node, install the required packages:

sudo yum install mapr-core mapr-fileserver mapr-cldb mapr-webserver -y

  1. Install Additional Services:
    • Based on your use case, install additional packages (e.g., mapr-zookeeper, mapr-nodemanager, etc.).

Step 4: Configure ZooKeeper

  1. Select ZooKeeper Nodes:
  2. Choose three nodes to run the ZooKeeper service (e.g., node1, node2, node3).
  3. Edit the ZooKeeper Configuration:
  4. Update the ZooKeeper configuration file (/opt/mapr/zookeeper/zookeeper-<version>/conf/zoo.cfg) on the ZooKeeper nodes:

tickTime=2000

dataDir=/var/mapr/zookeeper

clientPort=2181

initLimit=5

syncLimit=2

server.1=node1:2888:3888

server.2=node2:2888:3888

server.3=node3:2888:3888

  1. Initialize ZooKeeper:
    • On each ZooKeeper node, create a myid file:

echo “1” > /var/mapr/zookeeper/myid  # Replace with 2 or 3 for other nodes

  1. Start ZooKeeper:

sudo systemctl start mapr-zookeeper

Step 5: Configure the Cluster

  1. Initialize the Cluster:
    • Run the cluster initialization command from one node:

/opt/mapr/server/configure.sh -C node1,node2,node3 -Z node1,node2,node3

  • Replace node1,node2,node3 with the actual hostnames of the CLDB and ZooKeeper nodes.
  1. Verify Installation:
    • Check the cluster status:

maprcli cluster info

  1. Add Nodes to the Cluster:
    • On each additional node, configure it to join the cluster:

/opt/mapr/server/configure.sh -N <cluster_name> -C node1,node2,node3 -Z node1,node2,node3

Step 6: Start Core Services

  1. Start CLDB:
    • Start the CLDB service on the designated nodes:

sudo systemctl start mapr-cldb

  1. Start FileServer and WebServer:
    • Start the file server and web server services on all nodes:

sudo systemctl start mapr-fileserver

sudo systemctl start mapr-webserver

  1. Start Node Manager:
  2. If using YARN, start the Node Manager service on all nodes:

sudo systemctl start mapr-nodemanager

Step 7: Post-Installation Steps

  1. Access the Web Interface:
    • Open a browser and go to the web interface of your cluster:

http://<CLDB-node-IP&gt;:8443

  • Log in using the mapr user credentials.
  1. Add Storage:
    • Add storage disks to the cluster using the web interface or CLI:

maprcli disk list

maprcli disk add -all

  1. Secure the Cluster:
    • Enable Kerberos or LDAP integration for authentication.
    • Configure SSL/TLS for secure communication.

Step 8: Verify the Cluster

  1. Check Services:
    • Verify that all services are running:

maprcli node list -columns svc

  1. Run a Test:
  2. Create a directory in the file system and test file creation:

hadoop fs -mkdir /test

hadoop fs -put /path/to/local/file /test

Kong Service

techhadoop Uncategorized , , , , 
---
- name: Create Services with Service Paths and Routes in Kong API Gateway
  hosts: localhost
  tasks:

    # Define a list of services, their service paths, and routes
    - name: Define a list of services, service paths, and routes
      set_fact:
        services:
          - { name: service1, service_url: http://example-service1.com:8080/service1, route_name: route1, route_path: /service1 }
          - { name: service2, service_url: http://example-service2.com:8080/service2, route_name: route2, route_path: /service2 }
          - { name: service3, service_url: http://example-service3.com:8080/service3, route_name: route3, route_path: /service3 }

    # Create a Service in Kong for each service defined, including service path
    - name: Create Service in Kong
      uri:
        url: http://localhost:8001/services
        method: POST
        body_format: json
        body:
          name: "{{ item.name }}"
          url: "{{ item.service_url }}"  # Service URL (including the service path)
        status_code: 201
      loop: "{{ services }}"
      register: service_creation

# Create a Route for each Service
    - name: Create Route for the Service
      uri:
        url: http://localhost:8001/routes
        method: POST
        body_format: json
        body:
          service:
            name: "{{ item.name }}"
          name: "{{ item.route_name }}"  # Route name
          paths:
            - "{{ item.route_path }}"  # Route path (external access path)
        status_code: 201
      loop: "{{ services }}"
      when: service_creation is succeeded

    # Optionally verify that services and routes were created
    - name: Verify Service and Route creation
      uri:
        url: http://localhost:8001/services/{{ item.name }}
        method: GET
        status_code: 200
      loop: "{{ services }}"