security

Kong – HIGH-LEVEL AUTH FLOW

techhadoop , , , ,

Here’s a clear breakdown of how authentication works in a Kong + Ping Identity + IAM setup, which is common in enterprise environments where Kong acts as an API gateway and Ping Identity (like PingFederate/PingOne) handles user authentication and token issuance.

HIGH-LEVEL AUTH FLOW

Scenario:

Client → Kong Gateway → Protected Service
Kong integrates with Ping Identity for OIDC authentication or JWT validation, often backed by a central IAM system.

COMPONENT ROLES

ComponentRole
Kong GatewayAPI gateway that enforces authentication & authorization plugins
Ping IdentityIdentity Provider (IdP) – handles login, token issuance, and federation
IAM (e.g., Ping IAM, LDAP, AD)Stores users, groups, permissions, policies

AUTHENTICATION FLOW (OIDC Plugin)

OpenID Connect (Authorization Code Flow):

  1. Client → Kong
    Tries to access a protected API.
  2. Kong (OIDC Plugin)
    Redirects client to Ping Identity (Authorization Endpoint).
  3. Ping Identity (PingFederate or PingOne)
    • Authenticates user (UI, MFA, etc.).
    • Issues authorization_code.
  4. Kong → Ping Identity (Token Endpoint)
    Exchanges code for access_token (and optionally id_token, refresh_token).
  5. Kong (OIDC Plugin)
    • Validates token.
    • Optionally maps user to Kong consumer.
    • Passes request to backend with enriched headers (e.g., X-Consumer-Username).
  6. Backend Service
    Receives authenticated request with headers.

JWT Token Validation (Alt. Flow)

If Ping Identity issues JWT tokens, you can use Kong’s JWT plugin to validate them without redirect:

  1. Client gets access_token from Ping (out of band or SPA).
  2. Client sends request with Authorization: Bearer <token>.
  3. Kong JWT plugin verifies:
    • Signature using Ping’s public key / JWKS.
    • Claims like iss, aud, exp, etc.
  4. If valid, forward to upstream.

Where IAM Comes In

IAM sits behind Ping Identity and handles:

  • User/group storage (via LDAP, AD, DB, etc.)
  • Role mapping (e.g., admin/user)
  • Authorization policies (PingAccess, policy processor)
  • MFA/SSO rules

Ping Identity federates those identities into SSO tokens and passes them to Kong.

Plugin Example: OIDC Config

curl -X POST http://<admin>:8001/services/my-service/plugins \
  --data "name=openid-connect" \
  --data "config.issuer=https://ping-idp.com" \
  --data "config.client_id=my-client" \
  --data "config.client_secret=abc123" \
  --data "config.redirect_uri=https://my-kong.com/callback" \
  --data "config.scopes=openid profile email"

Kong Headers Passed to Backend

After successful auth, Kong can pass:

  • X-Consumer-Username
  • X-Authenticated-Userid
  • Authorization: Bearer <token> (if configured)

Optional Enhancements

  • Use ACL plugin to enforce group-level access.
  • Use OIDC group claim mapping to Kong consumer groups.
  • Enable rate limiting per consumer.
  • Log authenticated user ID to Splunk or ELK.

KONG 401 & 400 error

techhadoop , , ,

If you’re getting 401 and 400 intermittently from the upstream service, that strongly suggests authentication-related issues, often tied to token forwarding, expiration, or format mismatch.

🔁 Quick Summary of Key Differences

StatusMeaningCommon Cause
401UnauthorizedMissing/invalid/expired credentials
400Bad RequestMalformed or incomplete request (e.g. OIDC token request)

🧠 Intermittent 401 + 400: Common Root Causes

🔸 1. Expired or Reused Tokens

  • Kong gets a token once, caches it, and keeps using it—but upstream expects a fresh one.
  • Especially common with client credentials or authorization code flows.

Solution:

  • Set token caching to short duration or disable it in the OIDC plugin: config: cache_ttl: 0 # Or a very short TTL like 5

🔸 2. Multiple Consumers with Invalid Secrets

  • One client (consumer) is configured correctly, others are not.
  • You see 401/400 when the bad client makes a request.

Solution:

  • Enable verbose logging in Kong: export KONG_LOG_LEVEL=debug kong reload Then correlate consumer_id with the error.

🔸 3. Kong Not Forwarding Tokens Correctly

  • Kong authenticates but doesn’t forward Authorization header to the upstream.
  • Some plugins strip headers by default.

Solution:

  • Add request-transformer plugin to pass the token: curl -X POST http://localhost:8001/services/YOUR_SERVICE/plugins \ --data "name=request-transformer" \ --data "config.add.headers=Authorization:Bearer $(jwt_token)"

🔸 4. OIDC Plugin Misconfiguration

If you’re using the OpenID Connect plugin:

  • grant_type, client_id, or redirect_uri may be wrong or missing intermittently.
  • Kong might request a new token but fail to pass a correct one.

Check:

  • Kong OIDC plugin config
  • Errors like: error=invalid_request error_description="Unsupported client authentication method"

🧪 How to Debug Effectively

  1. Set KONG_LOG_LEVEL=debug, reload Kong, and tail the logs: tail -f /usr/local/kong/logs/error.log
  2. Inspect Upstream Request:
    • Look for what headers/body Kong is sending.
    • Especially Authorization, Content-Type, and request body if OIDC is involved.
  3. Track Errors to a Specific Consumer:
    • Use consumer_id in the access log to trace.
    • Maybe only some consumers are misconfigured.
  4. Try Curling the Upstream Directly with the exact payload Kong sends (use Postman or curl): curl -X POST https://upstream/token \ -H "Authorization: Bearer <your_token>" \ -H "Content-Type: application/x-www-form-urlencoded" \ -d "grant_type=client_credentials&client_id=...&client_secret=..."

to check

  • Kong plugin configs (especially OIDC/JWT)
  • A few lines from Kong’s debug logs showing the upstream request/response
  • Whether you’re using Ping Identity or a custom upstream

Kong error

techhadoop , , ,

The error:

no credentials found for given iss

in the Kong OIDC plugin means:

Kong received a token or request with an issuer (iss) claim, but it cannot find a matching configuration for that issuer.

🔍 Why This Happens

This usually occurs in one of these scenarios:

1. Missing or Incorrect config.issuer in the OIDC plugin

  • You did not set the exact issuer URL from Ping Identity in the plugin config.
  • Or you set it, but it’s spelled or formatted differently than what’s in the JWT’s iss claim.

Example:
If the token contains:

"iss": "https://auth.pingone.com/12345/as"

Then your OIDC plugin must be configured with:

config.issuer=https://auth.pingone.com/12345/as

2. Wrong Discovery Document

Kong uses the issuer to retrieve:

<issuer>/.well-known/openid-configuration

If the URL is incorrect or inaccessible, it can’t discover metadata and fails.

3. Token Issuer Doesn’t Match Plugin Config

  • You’re sending a JWT token manually (e.g., in Authorization: Bearer) and it’s not issued by the issuer you configured.
  • Kong expects the iss claim to match the config.issuer.

✅ Fix Steps

Step 1: Extract the Token’s Issuer

Run:

jwt decode <your-token>

Or paste the JWT into jwt.io and check the iss field.

Step 2: Update Plugin Config

Ensure the OIDC plugin config matches the exact iss value:

curl -X PATCH http://localhost:8001/plugins/<plugin-id> \
  --data "config.issuer=https://auth.pingone.com/12345/as"

Or if creating fresh:

curl -X POST http://localhost:8001/services/my-service/plugins \
  --data "name=openid-connect" \
  --data "config.issuer=https://auth.pingone.com/12345/as" \
  ...

🧪 Test Again

Once updated:

  1. Clear browser session or test in Postman
  2. Trigger the login flow again
  3. Ensure Kong receives a token from Ping with matching iss

Would you like help decoding the token or verifying your current plugin setup? You can paste the token header (without the signature) and I’ll check it.

A 401 Unauthorized error in Kong Gateway typically indicates that the request lacks valid authentication credentials required to access the resource. Here’s how you can troubleshoot and resolve this issue:

techhadoop , , , ,

🔍 Common Causes and Solutions

1. Missing or Invalid API Key (Key Authentication Plugin)

If you’re using the Key Authentication plugin, Kong expects a valid API key in the request.(Kong Docs)

  • Symptom: Receiving a 401 error with the message "No API key found in request" or "Invalid authentication credentials".(Turbolab Technologies)
  • Solution:
    • Create a Consumer: curl -i -X POST http://localhost:8001/consumers/ --data "username=your_username"
    • Assign an API Key to the Consumer: curl -i -X POST http://localhost:8001/consumers/your_username/key-auth --data "key=your_api_key"
    • Include the API Key in Your Request: curl -i http://lo calhost:8000/your_service \ -H "apikey: your_api_key"
    • Note: The header name (apikey) should match the key_names configured in your Key Authentication plugin.(Kong Docs)

2. Expired or Invalid JWT Token (JWT Plugin)

If you’re using the JWT plugin, ensure that the token is valid and not expired.

  • Symptom: Receiving a 401 error with the message "Invalid token" or "Token expired".
  • Solution:
    • Verify the Token: Use tools like jwt.io to decode and inspect the token.
    • Check Expiration: Ensure the exp claim has not passed.
    • Validate Signature: Confirm that the token’s signature matches the expected signing key.(Auth0 Community)

3. OpenID Connect (OIDC) Plugin Issues

When using the OIDC plugin, a 401 error can occur if the plugin’s cache is outdated after changes to the Identity Provider (IdP).(support.konghq.com)

  • Symptom: Receiving a 401 error after updating the IdP’s signing certificate.
  • Solution:
    • Clear Plugin Cache: Restart Kong or clear the plugin’s cache to fetch the updated signing keys from the IdP.
    • Verify IdP Configuration: Ensure that the IdP’s metadata URL is correctly configured in the plugin settings.

4. Plugin Configuration on Specific Routes or Services

Ensure that authentication plugins are correctly configured on the intended routes or services.

  • Symptom: Receiving a 401 error on specific endpoints.
  • Solution:
    • Check Plugin Association: Use the Admin API to list plugins associated with your routes or services. curl http://localhost:8001/plugins
    • Adjust Configuration: Ensure that the plugin is enabled and correctly configured for the desired routes or services.

🛠️ Additional Troubleshooting Steps

  • Inspect Kong Logs: Check the Kong error logs for detailed information.
  tail -f /usr/local/kong/logs/error.log
  • Use Debug Headers: Add the X-Kong-Debug: 1 header to your requests to get more insight into how Kong processes them.
  • Verify Plugin Order: Ensure that authentication plugins are executed in the correct order, especially when multiple plugins are used.

If you can provide more details about your setup—such as the authentication plugins in use, specific routes or services affected, and sample requests—I can offer more targeted assistance.

Allow LDAP users to access the Kong Manager GUI in Kong Gateway

techhadoop , , , ,

To allow LDAP users to access the Kong Manager GUI in Kong Gateway Enterprise 3.4, you’ll need to integrate LDAP authentication via the Kong Enterprise Role-Based Access Control (RBAC) system.

Here’s how you can get it working step-by-step 👇

👤 Step 1: Configure LDAP Authentication for Kong Manager

Edit your kong.conf or pass these as environment variables if you’re using a container setup.

admin_gui_auth = ldap-auth
admin_gui_auth_conf = {
  "ldap_host": "ldap.example.com",
  "ldap_port": 389,
  "ldap_base_dn": "dc=example,dc=com",
  "ldap_attribute": "uid",
  "ldap_bind_dn": "cn=admin,dc=example,dc=com",
  "ldap_password": "adminpassword",
  "start_tls": false,
  "verify_ldap_host": false
}

✅ If you’re using LDAPS, set ldap_port = 636 and start_tls = false or configure accordingly.

Restart Kong after updating this config.

👥 Step 2: Create an RBAC User Linked to the LDAP Username

Kong still needs an RBAC user that maps to the LDAP-authenticated identity.

curl -i -X POST http://localhost:8001/rbac/users \
  --data "name=jdoe" \
  --data "user_token=jdoe-admin-token"

The name here must match the LDAP uid or whatever attribute you configured with ldap_attribute.

🔐 Step 3: Assign a Role to the RBAC User

curl -i -X POST http://localhost:8001/rbac/users/jdoe/roles \
  --data "roles=read-only"  # Or "admin", "super-admin", etc.

Available roles: read-only, admin, super-admin, or your own custom roles.

🔓 Step 4: Log into Kong Manager with LDAP User

Go to your Kong Manager GUI:

https://<KONG_MANAGER_URL>:8445

Enter:

  • Username: jdoe (LDAP uid)
  • Password: LDAP user’s actual password (Kong will bind to LDAP and verify it)

🛠️ Optional: Test LDAP Config from CLI

You can test the LDAP binding from Kong CLI:

curl -i -X POST http://localhost:8001/rbac/users \
  --data "name=testuser" \
  --data "user_token=test123"

Then try logging into Kong Manager with testuser using their LDAP password.

LEAPP upgrade – multiple kernel issue

techhadoop , , , ,

Newest installed kernel not in use

If the newest installed kernel is not in use even after rebooting, it typically means your system is booting into an older kernel version. This happens due to the default GRUB settings or other configuration issues.

Here’s how you can resolve the issue step-by-step:

1. Verify the Currently Running Kernel

  • Check the kernel currently in use:

uname -r

  • Example output:

CopyEdit

3.10.0-1127.el7.x86_64

  • Compare this with the newest installed kernel version (see step 2).

2. List All Installed Kernels

  • List all installed kernel packages:

rpm -qa | grep kernel | grep -v firmware

  • Example output:

kernel-3.10.0-1127.el7.x86_64

kernel-3.10.0-1160.el7.x86_64

  • Identify the newest kernel (e.g., 3.10.0-1160.el7.x86_64).

3. Check GRUB Default Boot Entry

  • Display the current GRUB default boot entry:

grub2-editenv list

  • You should see something like this:

saved_entry=0

  • If the saved_entry is pointing to an older kernel, the system is booting into it by default.

4. Update GRUB to Use the Newest Kernel

Option A: Use the First Kernel (Default GRUB Behavior)

  • Edit the GRUB configuration:

sudo vi /etc/default/grub

  • Set the GRUB_DEFAULT value to 0 (the first kernel in the GRUB list):

GRUB_DEFAULT=0

  • Save and exit the file.
  • Regenerate the GRUB configuration:

sudo grub2-mkconfig -o /boot/grub2/grub.cfg

Option B: Manually Specify the Newest Kernel

  • Run this command to view all available kernel entries:

awk -F\’ ‘$1==”menuentry ” {print i++ ” : ” $2}’ /boot/grub2/grub.cfg

  • Example output:

0 : CentOS Linux (3.10.0-1160.el7.x86_64)

1 : CentOS Linux (3.10.0-1127.el7.x86_64)

2 : CentOS Linux (0-rescue-…)

  • Set GRUB_DEFAULT to the corresponding entry for the newest kernel (e.g., 0):

sudo grub2-set-default 0

  • Verify the setting:

grub2-editenv list

5. Reboot the System

  • Reboot your system to load the correct kernel:

sudo reboot

  • After reboot, confirm that the new kernel is in use:

uname -r

6. Remove Old Kernels (Optional)

  • To prevent confusion in the future, you can remove unused older kernels:

sudo package-cleanup –oldkernels –count=1

  • This retains only the most recent kernel.

7. Troubleshooting

  • Manually Select the Kernel: If the system still boots into the wrong kernel, you can manually select the desired kernel during the GRUB menu at boot time. To enable the GRUB menu:
    1. Edit /etc/default/grub:

sudo vi /etc/default/grub

  • Set GRUB_TIMEOUT to a non-zero value (e.g., 5):

GRUB_TIMEOUT=5

  • Save and regenerate the GRUB configuration:

sudo grub2-mkconfig -o /boot/grub2/grub.cfg

  • Kernel Missing After Update: Ensure the new kernel is properly installed. Reinstall it if necessary:

sudo yum install kernel

Ansible playbook to run a shell script on your remote hosts:

techhadoop , , , , 

#!/bin/bash
# example_script.sh

echo "Hello, this is a sample script."
echo "Current date and time: $(date)"

ansible 

---
- name: Run Shell Script on Remote Hosts
  hosts: all
  become: yes
  tasks:
    - name: Copy Shell Script to Remote Hosts
      copy:
        src: /path/to/example_script.sh
        dest: /tmp/example_script.sh
        mode: '0755'

    - name: Run Shell Script
      command: /tmp/example_script.sh

    - name: Remove Shell Script from Remote Hosts
      file:
        path: /tmp/example_script.sh
        state: absent

Explanation:
Copy Shell Script: The copy module is used to transfer the shell script from the local machine to the remote hosts. The mode parameter ensures that the script is executable.

Run Shell Script: The command module is used to execute the shell script on the remote hosts.

Remove Shell Script: The file module is used to delete the shell script from the remote hosts after execution to clean up.

Running the Playbook:
To run the playbook, use the following command:

initramfs

techhadoop , , , ,

What is initramfs ?

initramfs stands for initial RAM filesystem. It plays a crucial role in the Linux boot process by providing a temporary root filesystem that is loaded into memory. This temporary root filesystem contains the necessary drivers, tools, and scripts needed to mount the real root filesystem and continue the boot process.

In simpler terms, it acts as a bridge between the bootloader and the main operating system, ensuring that the system has everything it needs to boot successfully.

Key Concepts of initramfs:

FeatureDescription
Temporary FilesystemIt’s loaded into memory as a temporary root filesystem.
Kernel ModulesContains drivers (kernel modules) required to access disks, filesystems, and other hardware.
ScriptsContains initialization scripts to prepare the system for booting the real root filesystem.
Critical FilesIncludes essential tools like mount, udev, bash, and libraries.

Key Functions of initramfs:

  1. Kernel Initialization: During the boot process, the Linux kernel loads the initramfs into memory.
  2. Loading Drivers: initramfs includes essential drivers needed to access hardware components, such as storage devices and filesystems.
  3. Mounting Root Filesystem: The primary function of initramfs is to mount the real root filesystem from a storage device (e.g., hard drive, SSD).
  4. Transitioning to Real Root: Once the real root filesystem is mounted, the initramfs transitions control to the system’s main init process, allowing the boot process to continue.

How initramfs Works:

  1. Bootloader Stage: The bootloader (e.g., GRUB) loads the Linux kernel and initramfs into memory.
  2. Kernel Stage: The kernel initializes and mounts the initramfs as the root filesystem.
  3. Init Stage: The init script or program within initramfs runs, performing tasks such as loading additional drivers, mounting filesystems, and locating the real root filesystem.
  4. Switch Root: The initramfs mounts the real root filesystem and switches control to it, allowing the system to boot normally.

Customizing initramfs:

You can customize the initramfs by including specific drivers, tools, and scripts. This is useful for scenarios where the default initramfs does not include the necessary components for your system.

Tools for Managing initramfs:

  • mkinitramfs: A tool to create initramfs images.
  • update-initramfs: A tool to update existing initramfs images.

Difference Between initramfs and initrd

Featureinitramfsinitrd
Formatcpio archive (compressed)Disk image (block device)
MountingExtracted directly into RAM as a rootfsMounted as a loop device
FlexibilityMore flexible and fasterLess flexible, older technology

Location of initramfs

On most Linux distributions, the initramfs file is located in the /boot directory:

ls /boot/initramfs-*.img

How to Rebuild initramfs

If you’ve made changes to the kernel, /etc/fstab, or storage configuration (e.g., LUKS, LVM), you may need to rebuild the initramfs.

Rebuild initramfs on RHEL/CentOS:

sudo dracut -f

Rebuild initramfs on Ubuntu/Debian:

sudo update-initramfs -u

Common Issues Related to initramfs

IssueCauseSolution
Dropped into initramfs shellKernel can’t find the root filesystemCheck /etc/fstab, rebuild initramfs, or fix missing drivers.
Boot failure after kernel updateMissing or corrupt initramfsRebuild initramfs.
Filesystem not mountingIncorrect or missing drivers in initramfsEnsure necessary drivers are included and rebuild.

By understanding how initramfs works, you can better appreciate its role in the Linux boot process and customize it to suit your needs.

How to use LUKS data disk encryption in MapR

techhadoop , , , ,

How to use LUKS data disk encryption in MapR

MapR (now part of HPE Ezmeral) supports encryption at various levels, but using LUKS (Linux Unified Key Setup) encryption for data disks is a system-level operation that must be done outside of MapR’s native encryption features. Here’s a step-by-step guide to set up LUKS disk encryption on a MapR node and ensure MapR can access the encrypted disk after unlocking it.

Steps to Set Up LUKS Disk Encryption for MapR Data Disks

1. Identify the Disk to Encrypt

Find the disk you want to encrypt using the lsblk or fdisk command:

bash

lsblk

fdisk -l

For example, if the disk to be encrypted is /dev/sdb, use that in the following steps.

2. Install the Necessary Packages

Ensure you have the required tools to set up LUKS encryption:

sudo apt-get install cryptsetup  # For Ubuntu/Debian

sudo yum install cryptsetup      # For CentOS/RHEL

3. Set Up LUKS Encryption on the Disk

Run the following command to initialize the disk with LUKS encryption:

sudo cryptsetup luksFormat /dev/sdb

You’ll be prompted to confirm the operation and set a passphrase.

⚠️ Warning: This will erase all existing data on the disk.

4. Open and Map the Encrypted Disk

Unlock the encrypted disk and map it to a device:

sudo cryptsetup open /dev/sdb mapr_data_disk

You can verify that the encrypted device is available:

lsblk

5. Format the Encrypted Disk

Format the newly mapped device with a filesystem that MapR supports (typically ext4 or xfs):

sudo mkfs.ext4 /dev/mapper/mapr_data_disk

6. Mount the Encrypted Disk

Create a mount point and mount the encrypted disk:

sudo mkdir -p /opt/mapr/data

sudo mount /dev/mapper/mapr_data_disk /opt/mapr/data

7. Make the Mount Persistent

Edit the /etc/crypttab file to automatically unlock the disk at boot:

echo “mapr_data_disk /dev/sdb none luks” | sudo tee -a /etc/crypttab

Then, add an entry to /etc/fstab to mount the disk automatically after it is unlocked:

echo “/dev/mapper/mapr_data_disk /opt/mapr/data ext4 defaults 0 0” | sudo tee -a /etc/fstab

8. Ensure MapR Can Access the Disk

Make sure the MapR user has the necessary permissions to access the encrypted disk:

sudo chown -R mapr:mapr /opt/mapr/data

9. Test the Setup

Reboot the system to ensure the encrypted disk is unlocked and mounted correctly:

sudo reboot

After the system reboots, verify that the disk is unlocked and mounted:

lsblk

df -h

10. Verify MapR Storage Pools

After the encrypted disk is mounted, add it to the MapR storage pool:

maprcli disk add -server <server_name> -disks /dev/mapper/mapr_data_disk

Additional Considerations

  • Passphrase Management: Consider integrating with a key management system (KMS) to avoid manual passphrase entry.
  • Performance Impact: Encryption may introduce some performance overhead, so test accordingly.
  • Backup Configuration Files: Ensure you back up /etc/crypttab and /etc/fstab for disaster recovery.

Kong – add custom plugins (ansible playbook)

techhadoop , , , , 

---
- name: Deploy and enable a custom plugin in Kong
  hosts: kong_servers
  become: yes
  vars:
    plugin_name: "my_custom_plugin"
    plugin_source_path: "/path/to/local/plugin" # Local path to the plugin code
    kong_plugin_dir: "/usr/local/share/lua/5.1/kong/plugins" # Default Kong plugin directory
  tasks:

    - name: Ensure Kong plugin directory exists
      file:
        path: "{{ kong_plugin_dir }}/{{ plugin_name }}"
        state: directory
        mode: '0755'

    - name: Copy plugin files to Kong plugin directory
      copy:
        src: "{{ plugin_source_path }}/"
        dest: "{{ kong_plugin_dir }}/{{ plugin_name }}/"
        mode: '0644'

    - name: Verify plugin files were copied
      shell: ls -la "{{ kong_plugin_dir }}/{{ plugin_name }}"
      register: verify_plugin_copy
    - debug:
        var: verify_plugin_copy.stdout

    - name: Update Kong configuration to include the custom plugin
      lineinfile:
        path: "/etc/kong/kong.conf"
        regexp: "^plugins ="
        line: "plugins = bundled,{{ plugin_name }}"
        state: present
      notify: restart kong

    - name: Verify the plugin is enabled
      shell: kong config parse /etc/kong/kong.conf
      register: config_check
    - debug:
        var: config_check.stdout

  handlers:
    - name: restart kong
      service:
        name: kong
        state: restarted