Uncategorized

mTLS cloud flare integration with kong

techhadoop Uncategorized

integrate kong with cloudflare with mtls

Reasoned about Kong Cloudflare integration for 8 seconds

Integrating Kong with Cloudflare via mTLS

The user is asking about how to integrate Kong with Cloudflare using mTLS, which stands for mutual TLS. Kong is an API gateway, and the goal is to set up mTLS for secure communication with Cloudflare. This likely involves configuring Cloudflare as a reverse proxy for Kong. I’ll need to include steps for setting up mTLS on both ends, including cert management. The answer will detail how to configure mTLS in Cloudflare and how to handle it on the Kong side for a secure connection.

Configuring Kong and Cloudflare with mTLS

To integrate Kong with Cloudflare using mTLS, we need a few key steps. Cloudflare supports mTLS via its Zero Trust/Access solutions, and Kong can use mTLS for upstream services. Here’s a simplified approach:

  1. Generate certificates (either Cloudflare’s Origin or client certificate).
  2. On Kong’s end, configure the mTLS plugin or settings to verify the client certificate.
  3. On Cloudflare, configure mTLS to forward certificates or use “Authenticated Origin Pulls” to validate requests with a client certificate from Cloudflare to Kong.

Below is an overview of how you can integrate Kong with Cloudflare using mutual TLS (mTLS) to secure the connection between Cloudflare (as your edge proxy) and Kong (as your API gateway):

1. Overview of the Integration

  • Cloudflare Authenticated Origin Pulls:
    Cloudflare can be configured to present a client certificate when it connects to your origin server. This proves to your origin that the connection is coming through Cloudflare.
  • Kong as the Origin:
    Kong (which runs on Nginx) can be configured to require and verify client certificates on incoming TLS connections. In this scenario, Kong will be set to trust only the client certificate (or its issuing CA) that Cloudflare uses.

The integration ensures that only requests that have passed through Cloudflare—and that present the proper client certificate—are accepted by Kong.

2. Prerequisites

  • A Cloudflare account with your domain configured.
  • Cloudflare’s Authenticated Origin Pulls enabled (this is available in your Cloudflare dashboard).
  • A publicly trusted server certificate installed on Kong for its TLS listener.
  • Administrative access to your Kong instance so you can modify TLS settings (via configuration file or custom Nginx snippets).

3. Enable Cloudflare Authenticated Origin Pulls

  1. Activate Authenticated Origin Pulls in Cloudflare:
    In the Cloudflare dashboard, go to SSL/TLS → Edge Certificates and ensure that Authenticated Origin Pulls is turned on.
  2. Download the Cloudflare Origin Pull CA Certificate:
    Cloudflare publishes a CA certificate that it uses to sign its client certificates for origin pulls. You can download it from Cloudflare’s documentation (typically named something like origin-pull-ca.pem).

4. Configure Kong to Require mTLS from Cloudflare

Since Kong uses Nginx as its underlying proxy, you can require client certificate verification by adjusting its TLS configuration. There are two common approaches:

A. Using a Custom Nginx Configuration Snippet in Kong

  1. Locate or Create a Custom Nginx Snippet File:
    Depending on your Kong version, you can add custom configuration to Kong’s Nginx configuration (for example, using the nginx_http_ssl_verify_client directive).
  2. Configure Client Certificate Verification:
    In your custom snippet (which might be included via Kong’s configuration file or a custom plugin), add directives similar to the following:# This snippet is applied to the HTTPS server block in Kong’s Nginx config ssl_verify_client on; ssl_client_certificate /path/to/cloudflare-origin-pull-ca.pem;
  3. Reload Kong:
    After updating the configuration, reload or restart Kong so that the new settings take effect.

B. Configuring Kong (Enterprise) with Built-In mTLS Features

If you are using Kong Enterprise or a version that supports mTLS configuration natively for incoming connections, you can use its configuration options (or plugins) to require client certificates. Check the Kong documentation for any native mTLS support where you can specify a trusted CA.

5. Test the Setup

  1. Verify the TLS Handshake:
    Use tools like OpenSSL’s s_client to simulate a connection to Kong and check that it requires a client certificate: openssl s_client -connect your-kong-host:443 -CAfile /path/to/cloudflare-origin-pull-ca.pem Without a proper client certificate, the handshake should fail.
  2. Send a Request Through Cloudflare:
    Ensure that when Cloudflare forwards requests to Kong, it presents its client certificate and that Kong accepts the connection.
  3. Monitor Kong Logs:
    Review Kong (or Nginx) logs to ensure that client certificate verification is working as expected.

6. Additional Considerations

  • Certificate Management:
    Maintain and update your Cloudflare Origin Pull CA certificate on Kong when required (though Cloudflare’s published CA rarely changes).
  • Fallback and Testing:
    During testing, you might want to temporarily set ssl_verify_client optional; so you can observe behavior without blocking all connections, then switch to on for production.
  • Security and Performance:
    Ensure that all certificate files are stored securely on your Kong server and that proper file permissions are set.
  • Documentation and Updates:
    Always refer to the latest Cloudflare documentation on Authenticated Origin Pulls and Kong documentation for any version-specific instructions.

LEAPP upgrade – multiple kernel issue

techhadoop Uncategorized , , , ,

Newest installed kernel not in use

If the newest installed kernel is not in use even after rebooting, it typically means your system is booting into an older kernel version. This happens due to the default GRUB settings or other configuration issues.

Here’s how you can resolve the issue step-by-step:

1. Verify the Currently Running Kernel

  • Check the kernel currently in use:

uname -r

  • Example output:

CopyEdit

3.10.0-1127.el7.x86_64

  • Compare this with the newest installed kernel version (see step 2).

2. List All Installed Kernels

  • List all installed kernel packages:

rpm -qa | grep kernel | grep -v firmware

  • Example output:

kernel-3.10.0-1127.el7.x86_64

kernel-3.10.0-1160.el7.x86_64

  • Identify the newest kernel (e.g., 3.10.0-1160.el7.x86_64).

3. Check GRUB Default Boot Entry

  • Display the current GRUB default boot entry:

grub2-editenv list

  • You should see something like this:

saved_entry=0

  • If the saved_entry is pointing to an older kernel, the system is booting into it by default.

4. Update GRUB to Use the Newest Kernel

Option A: Use the First Kernel (Default GRUB Behavior)

  • Edit the GRUB configuration:

sudo vi /etc/default/grub

  • Set the GRUB_DEFAULT value to 0 (the first kernel in the GRUB list):

GRUB_DEFAULT=0

  • Save and exit the file.
  • Regenerate the GRUB configuration:

sudo grub2-mkconfig -o /boot/grub2/grub.cfg

Option B: Manually Specify the Newest Kernel

  • Run this command to view all available kernel entries:

awk -F\’ ‘$1==”menuentry ” {print i++ ” : ” $2}’ /boot/grub2/grub.cfg

  • Example output:

0 : CentOS Linux (3.10.0-1160.el7.x86_64)

1 : CentOS Linux (3.10.0-1127.el7.x86_64)

2 : CentOS Linux (0-rescue-…)

  • Set GRUB_DEFAULT to the corresponding entry for the newest kernel (e.g., 0):

sudo grub2-set-default 0

  • Verify the setting:

grub2-editenv list

5. Reboot the System

  • Reboot your system to load the correct kernel:

sudo reboot

  • After reboot, confirm that the new kernel is in use:

uname -r

6. Remove Old Kernels (Optional)

  • To prevent confusion in the future, you can remove unused older kernels:

sudo package-cleanup –oldkernels –count=1

  • This retains only the most recent kernel.

7. Troubleshooting

  • Manually Select the Kernel: If the system still boots into the wrong kernel, you can manually select the desired kernel during the GRUB menu at boot time. To enable the GRUB menu:
    1. Edit /etc/default/grub:

sudo vi /etc/default/grub

  • Set GRUB_TIMEOUT to a non-zero value (e.g., 5):

GRUB_TIMEOUT=5

  • Save and regenerate the GRUB configuration:

sudo grub2-mkconfig -o /boot/grub2/grub.cfg

  • Kernel Missing After Update: Ensure the new kernel is properly installed. Reinstall it if necessary:

sudo yum install kernel

Istio

techhadoop Uncategorized

What is istio ?

Istio is an open-source service mesh platform that helps manage, secure, and monitor communication between microservices in distributed systems, such as Kubernetes-based applications. It abstracts the complexity of service-to-service communication and provides powerful capabilities without requiring changes to application code.

Key Features of Istio

  1. Traffic Management
    • Provides fine-grained control over service traffic.
    • Features include:
      • Load balancing.
      • Traffic routing (e.g., blue-green deployments, canary releases).
      • Traffic splitting by percentage.
      • Retry and failover mechanisms.
  2. Security
    • Enables mutual TLS (mTLS) for secure communication between services.
    • Provides service-level identity, authentication, and authorization.
    • Supports encryption of data in transit.
  3. Observability
    • Provides monitoring and tracing capabilities for service communication.
    • Integrates with tools like Prometheus, Grafana, and Jaeger for metrics and distributed tracing.
    • Offers detailed logs and dashboards for understanding service health and performance.
  4. Service Resilience
    • Provides fault injection to test resilience.
    • Circuit breakers and timeouts for handling failing services gracefully.
    • Automatic retries for transient failures.
  5. Policy Enforcement
    • Allows applying policies at runtime (e.g., rate limiting, quotas).
    • Helps enforce compliance with organizational or regulatory requirements.

How Istio Works

Istio operates by introducing sidecars and control planes:

  1. Sidecar Proxy
    • Istio uses Envoy as a sidecar proxy.
    • A proxy container is injected alongside each service container in a pod.
    • The proxy intercepts and manages all inbound and outbound traffic for the service.
  2. Control Plane
    • The control plane (typically managed by the Istiod component) oversees configuration, policy enforcement, and secure communication setup.
    • It communicates with all sidecar proxies to enforce desired traffic behavior.

Use Cases for Istio

  1. Service Mesh for Microservices
    • Manage the complexity of service communication in large microservices environments.
  2. Zero-Trust Security
    • Implement mTLS and fine-grained access control between services.
  3. Traffic Splitting
    • Perform canary deployments or A/B testing by routing a percentage of traffic to different service versions.
  4. Observability
    • Gain deep insights into service interactions with metrics, logs, and traces.
  5. Policy Enforcement
    • Apply runtime policies to ensure reliability and security.

Istio Architecture

Istio’s architecture consists of the following components:

  1. Envoy Proxy:
    • Data plane component responsible for service-to-service communication.
    • Handles traffic routing, load balancing, and security (e.g., mTLS).
  2. Istiod:
    • Central control plane that configures proxies and enforces policies.
    • Provides service discovery, certificate management, and telemetry data.
  3. Telemetry Tools:
    • Istio collects metrics, logs, and traces to integrate with monitoring tools like Prometheus, Grafana, and Jaeger.

Istio vs Traditional Load Balancers

  • Istio provides layer 7 (application layer) traffic management, unlike traditional load balancers which mostly focus on layer 4 (transport layer).
  • It can dynamically route traffic based on HTTP headers, cookies, or other data, which makes it ideal for modern microservice architectures.

When to Use Istio

Use Istio if:

  • Your application involves multiple microservices that need traffic management and security.
  • You need advanced observability for troubleshooting and monitoring.
  • You’re running Kubernetes or containerized workloads at scale.
  • You want to implement fine-grained traffic controls, such as canary or blue-green deployments.

Ansible – host list

techhadoop Uncategorized , , , ,

The error “host list declined parsing host file as it did not pass its verify_file() method” occurs when Ansible cannot properly parse your inventory file. This is typically due to an invalid format or incorrect file structure. Here’s how to troubleshoot and fix it:

Step 1: Verify Inventory File Format

Correct INI Format Example

Ensure your inventory file uses the proper INI-style syntax:

[webservers]

web1 ansible_host=192.168.1.10 ansible_user=root

web2 ansible_host=192.168.1.11 ansible_user=root

[dbservers]

db1 ansible_host=192.168.1.20 ansible_user=root

Correct YAML Format Example

If using a YAML-based inventory file, ensure it follows the correct structure:

all:

  hosts:

    web1:

      ansible_host: 192.168.1.10

      ansible_user: root

    web2:

      ansible_host: 192.168.1.11

      ansible_user: root

  children:

    dbservers:

      hosts:

        db1:

          ansible_host: 192.168.1.20

          ansible_user: root

Step 2: Check File Extension

  • For INI-style inventory, use .ini or no extension (e.g., inventory).
  • For YAML-style inventory, use .yaml or .yml.

Step 3: Test the Inventory File

Use the ansible-inventory command to validate the inventory file:

ansible-inventory -i inventory –list

  • Replace inventory with the path to your inventory file.
  • If there’s an error, it will provide details about what’s wrong.

 Step 4: Use Explicit Inventory Path

When running an Ansible command or playbook, explicitly specify the inventory file:

ansible all -i /path/to/inventory -m ping

 Step 5: Check Syntax Errors

  1. Ensure there are no trailing spaces or unexpected characters in your inventory file.
  2. Use a linter for YAML files if you suspect a YAML formatting issue:

yamllint /path/to/inventory.yaml

 Step 6: Set Permissions

Ensure the inventory file has the correct permissions so that Ansible can read it:

chmod 644 /path/to/inventory

Leapp

techhadoop Uncategorized

Leapp is a tool provided by Red Hat for performing in-place upgrades of Red Hat Enterprise Linux (RHEL) systems. It helps users upgrade from one major version of RHEL to another without needing to reinstall the entire system1. This means you can move from, say, RHEL 7 to RHEL 8 or RHEL 8 to RHEL 9, while keeping your system configurations, custom repositories, and third-party applications intact.

Ansible playbook to run a shell script on your remote hosts:

techhadoop Uncategorized , , , , 

#!/bin/bash
# example_script.sh

echo "Hello, this is a sample script."
echo "Current date and time: $(date)"

ansible 

---
- name: Run Shell Script on Remote Hosts
  hosts: all
  become: yes
  tasks:
    - name: Copy Shell Script to Remote Hosts
      copy:
        src: /path/to/example_script.sh
        dest: /tmp/example_script.sh
        mode: '0755'

    - name: Run Shell Script
      command: /tmp/example_script.sh

    - name: Remove Shell Script from Remote Hosts
      file:
        path: /tmp/example_script.sh
        state: absent

Explanation:
Copy Shell Script: The copy module is used to transfer the shell script from the local machine to the remote hosts. The mode parameter ensures that the script is executable.

Run Shell Script: The command module is used to execute the shell script on the remote hosts.

Remove Shell Script: The file module is used to delete the shell script from the remote hosts after execution to clean up.

Running the Playbook:
To run the playbook, use the following command:

initramfs

techhadoop Uncategorized , , , ,

What is initramfs ?

initramfs stands for initial RAM filesystem. It plays a crucial role in the Linux boot process by providing a temporary root filesystem that is loaded into memory. This temporary root filesystem contains the necessary drivers, tools, and scripts needed to mount the real root filesystem and continue the boot process.

In simpler terms, it acts as a bridge between the bootloader and the main operating system, ensuring that the system has everything it needs to boot successfully.

Key Concepts of initramfs:

FeatureDescription
Temporary FilesystemIt’s loaded into memory as a temporary root filesystem.
Kernel ModulesContains drivers (kernel modules) required to access disks, filesystems, and other hardware.
ScriptsContains initialization scripts to prepare the system for booting the real root filesystem.
Critical FilesIncludes essential tools like mount, udev, bash, and libraries.

Key Functions of initramfs:

  1. Kernel Initialization: During the boot process, the Linux kernel loads the initramfs into memory.
  2. Loading Drivers: initramfs includes essential drivers needed to access hardware components, such as storage devices and filesystems.
  3. Mounting Root Filesystem: The primary function of initramfs is to mount the real root filesystem from a storage device (e.g., hard drive, SSD).
  4. Transitioning to Real Root: Once the real root filesystem is mounted, the initramfs transitions control to the system’s main init process, allowing the boot process to continue.

How initramfs Works:

  1. Bootloader Stage: The bootloader (e.g., GRUB) loads the Linux kernel and initramfs into memory.
  2. Kernel Stage: The kernel initializes and mounts the initramfs as the root filesystem.
  3. Init Stage: The init script or program within initramfs runs, performing tasks such as loading additional drivers, mounting filesystems, and locating the real root filesystem.
  4. Switch Root: The initramfs mounts the real root filesystem and switches control to it, allowing the system to boot normally.

Customizing initramfs:

You can customize the initramfs by including specific drivers, tools, and scripts. This is useful for scenarios where the default initramfs does not include the necessary components for your system.

Tools for Managing initramfs:

  • mkinitramfs: A tool to create initramfs images.
  • update-initramfs: A tool to update existing initramfs images.

Difference Between initramfs and initrd

Featureinitramfsinitrd
Formatcpio archive (compressed)Disk image (block device)
MountingExtracted directly into RAM as a rootfsMounted as a loop device
FlexibilityMore flexible and fasterLess flexible, older technology

Location of initramfs

On most Linux distributions, the initramfs file is located in the /boot directory:

ls /boot/initramfs-*.img

How to Rebuild initramfs

If you’ve made changes to the kernel, /etc/fstab, or storage configuration (e.g., LUKS, LVM), you may need to rebuild the initramfs.

Rebuild initramfs on RHEL/CentOS:

sudo dracut -f

Rebuild initramfs on Ubuntu/Debian:

sudo update-initramfs -u

Common Issues Related to initramfs

IssueCauseSolution
Dropped into initramfs shellKernel can’t find the root filesystemCheck /etc/fstab, rebuild initramfs, or fix missing drivers.
Boot failure after kernel updateMissing or corrupt initramfsRebuild initramfs.
Filesystem not mountingIncorrect or missing drivers in initramfsEnsure necessary drivers are included and rebuild.

By understanding how initramfs works, you can better appreciate its role in the Linux boot process and customize it to suit your needs.

Upgrade RHEL 7 to RHEL 8 – ansible playbook

techhadoop Uncategorized , , , , 

---
- name: Upgrade RHEL 7 to RHEL 8
  hosts: all
  become: true
  vars:
    leapp_repo: "rhel-7-server-extras-rpms"

  tasks:
    - name: Ensure system is up to date
      yum:
        name: '*'
        state: latest

    - name: Install the Leapp utility
      yum:
        name: leapp
        state: present
        enablerepo: "{{ leapp_repo }}"

    - name: Run the pre-upgrade check with Leapp
      shell: leapp preupgrade
      register: leapp_preupgrade
      ignore_errors: true

    - name: Check for pre-upgrade issues
      debug:
        msg: "Leapp pre-upgrade check output: {{ leapp_preupgrade.stdout }}"

    - name: Fix any issues identified by Leapp (manual step)
      pause:
        prompt: "Please review the pre-upgrade report at /var/log/leapp/leapp-report.txt and fix any blocking issues. Press Enter to continue."

    - name: Proceed with the upgrade
      shell: leapp upgrade
      ignore_errors: true

    - name: Reboot the server to complete the upgrade
      reboot:
        reboot_timeout: 1200

    - name: Verify the OS version after reboot
      shell: cat /etc/redhat-release
      register: os_version

    - name: Display the new OS version
      debug:
        msg: "The server has been successfully upgraded to: {{ os_version.stdout }}"

How to use LUKS data disk encryption in MapR

techhadoop Uncategorized , , , ,

How to use LUKS data disk encryption in MapR

MapR (now part of HPE Ezmeral) supports encryption at various levels, but using LUKS (Linux Unified Key Setup) encryption for data disks is a system-level operation that must be done outside of MapR’s native encryption features. Here’s a step-by-step guide to set up LUKS disk encryption on a MapR node and ensure MapR can access the encrypted disk after unlocking it.

Steps to Set Up LUKS Disk Encryption for MapR Data Disks

1. Identify the Disk to Encrypt

Find the disk you want to encrypt using the lsblk or fdisk command:

bash

lsblk

fdisk -l

For example, if the disk to be encrypted is /dev/sdb, use that in the following steps.

2. Install the Necessary Packages

Ensure you have the required tools to set up LUKS encryption:

sudo apt-get install cryptsetup  # For Ubuntu/Debian

sudo yum install cryptsetup      # For CentOS/RHEL

3. Set Up LUKS Encryption on the Disk

Run the following command to initialize the disk with LUKS encryption:

sudo cryptsetup luksFormat /dev/sdb

You’ll be prompted to confirm the operation and set a passphrase.

⚠️ Warning: This will erase all existing data on the disk.

4. Open and Map the Encrypted Disk

Unlock the encrypted disk and map it to a device:

sudo cryptsetup open /dev/sdb mapr_data_disk

You can verify that the encrypted device is available:

lsblk

5. Format the Encrypted Disk

Format the newly mapped device with a filesystem that MapR supports (typically ext4 or xfs):

sudo mkfs.ext4 /dev/mapper/mapr_data_disk

6. Mount the Encrypted Disk

Create a mount point and mount the encrypted disk:

sudo mkdir -p /opt/mapr/data

sudo mount /dev/mapper/mapr_data_disk /opt/mapr/data

7. Make the Mount Persistent

Edit the /etc/crypttab file to automatically unlock the disk at boot:

echo “mapr_data_disk /dev/sdb none luks” | sudo tee -a /etc/crypttab

Then, add an entry to /etc/fstab to mount the disk automatically after it is unlocked:

echo “/dev/mapper/mapr_data_disk /opt/mapr/data ext4 defaults 0 0” | sudo tee -a /etc/fstab

8. Ensure MapR Can Access the Disk

Make sure the MapR user has the necessary permissions to access the encrypted disk:

sudo chown -R mapr:mapr /opt/mapr/data

9. Test the Setup

Reboot the system to ensure the encrypted disk is unlocked and mounted correctly:

sudo reboot

After the system reboots, verify that the disk is unlocked and mounted:

lsblk

df -h

10. Verify MapR Storage Pools

After the encrypted disk is mounted, add it to the MapR storage pool:

maprcli disk add -server <server_name> -disks /dev/mapper/mapr_data_disk

Additional Considerations

  • Passphrase Management: Consider integrating with a key management system (KMS) to avoid manual passphrase entry.
  • Performance Impact: Encryption may introduce some performance overhead, so test accordingly.
  • Backup Configuration Files: Ensure you back up /etc/crypttab and /etc/fstab for disaster recovery.

Kong – add custom plugins (ansible playbook)

techhadoop Uncategorized , , , , 

---
- name: Deploy and enable a custom plugin in Kong
  hosts: kong_servers
  become: yes
  vars:
    plugin_name: "my_custom_plugin"
    plugin_source_path: "/path/to/local/plugin" # Local path to the plugin code
    kong_plugin_dir: "/usr/local/share/lua/5.1/kong/plugins" # Default Kong plugin directory
  tasks:

    - name: Ensure Kong plugin directory exists
      file:
        path: "{{ kong_plugin_dir }}/{{ plugin_name }}"
        state: directory
        mode: '0755'

    - name: Copy plugin files to Kong plugin directory
      copy:
        src: "{{ plugin_source_path }}/"
        dest: "{{ kong_plugin_dir }}/{{ plugin_name }}/"
        mode: '0644'

    - name: Verify plugin files were copied
      shell: ls -la "{{ kong_plugin_dir }}/{{ plugin_name }}"
      register: verify_plugin_copy
    - debug:
        var: verify_plugin_copy.stdout

    - name: Update Kong configuration to include the custom plugin
      lineinfile:
        path: "/etc/kong/kong.conf"
        regexp: "^plugins ="
        line: "plugins = bundled,{{ plugin_name }}"
        state: present
      notify: restart kong

    - name: Verify the plugin is enabled
      shell: kong config parse /etc/kong/kong.conf
      register: config_check
    - debug:
        var: config_check.stdout

  handlers:
    - name: restart kong
      service:
        name: kong
        state: restarted