kong – 2 services same route

In Kong Gateway, you cannot have two services bound to the exact same route — a route must be unique in terms of its combination of matching rules (such as paths, hosts, methods, etc.).


🚫 Why You Can’t Have Duplicate Routes

Kong uses the route’s matching criteria to determine which service should handle a request. If two routes have the same matching rules, Kong can’t disambiguate, which leads to a conflict error when trying to create the second route.

For example:

# Service A
curl -i -X POST http://localhost:8001/services/ --data name=service-a --data url=http://api-a
curl -i -X POST http://localhost:8001/routes --data name=route-a --data service.name=service-a --data paths[]=/api

# Service B
curl -i -X POST http://localhost:8001/services/ --data name=service-b --data url=http://api-b
curl -i -X POST http://localhost:8001/routes --data name=route-b --data service.name=service-b --data paths[]=/api

⛔ The second POST /routes will fail with:

HTTP 409 Conflict – duplicate entry


✅ Workarounds / Alternatives

GoalSolution
Blue/Green or Canary DeploymentsUse Kong Plugins (e.g., traffic-split, canary) or upstreams and targets instead of duplicate routes.
Different consumers/users hitting different backendsUse request transformers, ACLs, or Kong Enterprise Route-by-header/Route-by-consumer plugins.
Same path, different method or hostYou can differentiate routes by methods[], hosts[], or headers.

🧪 Example: Two Routes with Same Path, Different Hosts

# Route 1
curl -i -X POST http://localhost:8001/routes \
  --data name=api-v1 \
  --data paths[]=/api \
  --data hosts[]=v1.example.com \
  --data service.name=service-a

# Route 2
curl -i -X POST http://localhost:8001/routes \
  --data name=api-v2 \
  --data paths[]=/api \
  --data hosts[]=v2.example.com \
  --data service.name=service-b

These can coexist because their hosts[] fields are different.


🧠 Summary

Route Matching RuleMust Be Unique For
paths[]Same host/methods/headers
hosts[]If combined with same path
methods[], headersCan disambiguate routes with same path

If you’re trying to achieve load balancing, blue-green deployment, or AB testing between services at the same route — I can help you set that up using upstreams + targets or the traffic-split plugin.

Would you like an example?

Kong error

The error:

no credentials found for given iss

in the Kong OIDC plugin means:

Kong received a token or request with an issuer (iss) claim, but it cannot find a matching configuration for that issuer.


🔍 Why This Happens

This usually occurs in one of these scenarios:

1. Missing or Incorrect config.issuer in the OIDC plugin

  • You did not set the exact issuer URL from Ping Identity in the plugin config.
  • Or you set it, but it’s spelled or formatted differently than what’s in the JWT’s iss claim.

Example:
If the token contains:

"iss": "https://auth.pingone.com/12345/as"

Then your OIDC plugin must be configured with:

config.issuer=https://auth.pingone.com/12345/as


2. Wrong Discovery Document

Kong uses the issuer to retrieve:

<issuer>/.well-known/openid-configuration

If the URL is incorrect or inaccessible, it can’t discover metadata and fails.


3. Token Issuer Doesn’t Match Plugin Config

  • You’re sending a JWT token manually (e.g., in Authorization: Bearer) and it’s not issued by the issuer you configured.
  • Kong expects the iss claim to match the config.issuer.

✅ Fix Steps

Step 1: Extract the Token’s Issuer

Run:

jwt decode <your-token>

Or paste the JWT into jwt.io and check the iss field.

Step 2: Update Plugin Config

Ensure the OIDC plugin config matches the exact iss value:

curl -X PATCH http://localhost:8001/plugins/<plugin-id> \
  --data "config.issuer=https://auth.pingone.com/12345/as"

Or if creating fresh:

curl -X POST http://localhost:8001/services/my-service/plugins \
  --data "name=openid-connect" \
  --data "config.issuer=https://auth.pingone.com/12345/as" \
  ...


🧪 Test Again

Once updated:

  1. Clear browser session or test in Postman
  2. Trigger the login flow again
  3. Ensure Kong receives a token from Ping with matching iss

Would you like help decoding the token or verifying your current plugin setup? You can paste the token header (without the signature) and I’ll check it.

Kong troubleshooting

A 502 Bad Gateway error in your setup:

Client → LB → KONG → LB → Server

means that Kong Gateway cannot successfully connect to the upstream server behind the second load balancer. Here’s how to troubleshoot and resolve it step by step:


🔁 Understanding the Flow

  1. Client hits the first LB (e.g., AWS ELB, NGINX).
  2. That LB forwards traffic to Kong Gateway.
  3. Kong receives the request and proxies it to a second LB.
  4. That LB routes to your actual upstream service.

🛠️ Troubleshooting Checklist

✅ 1. Check Upstream URL in Kong

  • Run: curl -s http://<KONG_ADMIN>:8001/services/<service-name>
  • Ensure the upstream URL (url or host) points to a valid, resolvable address (DNS or IP) and correct port.

Example:

{
  "host": "internal-lb.yourdomain.local",
  "port": 8080,
  "protocol": "http"
}

✅ 2. Test Kong’s Network Reachability

From the Kong container or host:

curl -v http://internal-lb.yourdomain.local:8080/

If this fails:

  • DNS resolution might be broken
  • Port might be blocked
  • LB might not route correctly

✅ 3. Enable Debug Logs in Kong

In kong.conf or via environment variable:

log_level = debug

Then check:

tail -f /usr/local/kong/logs/error.log

Look for messages like:

  • upstream timed out
  • could not resolve host
  • connection refused

✅ 4. Check Health of Second LB and Backend

  • Ensure second LB is up
  • Verify backend servers are healthy and accepting connections
  • Check if Kong’s IP is allowed (firewall or security group)

✅ 5. Check Kong Route & Service Configuration

Validate route is defined correctly:

curl -s http://<KONG_ADMIN>:8001/routes

Make sure paths, hosts, or methods match the request.


🧪 Example Kong Service & Route Setup

# Service pointing to internal load balancer
curl -i -X POST http://localhost:8001/services \
  --data name=upstream-service \
  --data url=http://internal-lb.yourdomain.local:8080

# Route for the service
curl -i -X POST http://localhost:8001/services/upstream-service/routes \
  --data paths[]=/api


🚫 Common Causes of 502 with LB Behind Kong

ProblemSolution
DNS resolution failureUse IP or fix /etc/resolv.conf or CoreDNS
Port not exposed or wrongConfirm port with nc or curl
Second LB not forwarding correctlyCheck LB target groups and health checks
Kong plugins (e.g., OIDC, rate-limit) errorDisable plugins temporarily to isolate
HTTP vs HTTPS mismatchEnsure protocol matches (http vs https)
Timeout too shortIncrease proxy_read_timeout or similar

✅ Final Tips

  • Try curl directly from Kong to the backend server.
  • Use Kong’s health check endpoint if you’re using upstream targets: curl http://localhost:8001/upstreams/<name>/health

If you share:

  • the exact curl call to Kong
  • the relevant Kong service/route config
  • error.log content from Kong

The error message “upstream prematurely closed connection while reading response header from upstream” in Kong Gateway indicates that Kong attempted to read the response headers from the upstream service, but the connection was closed unexpectedly before the headers were fully received. This typically results in a 502 Bad Gateway error.

Common Causes

  1. Upstream Service Crashes or Terminates Connection Early:
    • The upstream application may crash, encounter an error, or intentionally close the connection before sending a complete response.
  2. Timeouts:
    • The upstream service takes too long to respond, exceeding Kong’s configured timeouts.
  3. Keepalive Connection Issues:
    • Persistent connections (keepalive) between Kong and the upstream service may be closed unexpectedly by the upstream, leading to this error.
  4. Protocol Mismatch:
    • Kong expects a certain protocol (e.g., HTTP/1.1), but the upstream service responds differently or uses an incompatible protocol.
  5. Large Response Headers:
    • The upstream service sends headers that exceed Kong’s buffer sizes, causing the connection to be closed prematurely.

es, a mismatch between the protocol specified in Kong’s service configuration and the actual protocol used by the upstream service can lead to the error:

“upstream prematurely closed connection while reading response header from upstream”

This typically occurs when Kong attempts to communicate with an upstream service over HTTP, but the upstream expects HTTPS, or vice versa.


🔍 Understanding the Issue

When Kong is configured to connect to an upstream service, it uses the protocol specified in the service’s configuration. If the upstream service expects HTTPS connections and Kong is configured to use HTTP, the SSL/TLS handshake will fail, leading to the connection being closed prematurely.

Yes, a mismatch between the protocol specified in Kong’s service configuration and the actual protocol used by the upstream service can lead to the error:

“upstream prematurely closed connection while reading response header from upstream”

This typically occurs when Kong attempts to communicate with an upstream service over HTTP, but the upstream expects HTTPS, or vice versa.


🔍 Understanding the Issue

When Kong is configured to connect to an upstream service, it uses the protocol specified in the service’s configuration. If the upstream service expects HTTPS connections and Kong is configured to use HTTP, the SSL/TLS handshake will fail, leading to the connection being closed prematurely.

For example, if your upstream service is accessible at https://api.example.com, but Kong is configured with:

bashCopyEditcurl -i -X POST http://localhost:8001/services \
  --data name=example-service \
  --data url=http://api.example.com

Kong will attempt to connect over HTTP, resulting in a failed connection.


✅ Solution

Ensure that the protocol in Kong’s service configuration matches the protocol expected by the upstream service.

If the upstream service requires HTTPS, configure the service in Kong accordingly:

bashCopyEditcurl -i -X POST http://localhost:8001/services \
  --data name=example-service \
  --data url=https://api.example.com

This ensures that Kong establishes a secure connection using HTTPS, aligning with the upstream service’s expectations.

Allow LDAP users to access the Kong Manager GUI in Kong Gateway

To allow LDAP users to access the Kong Manager GUI in Kong Gateway Enterprise 3.4, you’ll need to integrate LDAP authentication via the Kong Enterprise Role-Based Access Control (RBAC) system.

Here’s how you can get it working step-by-step 👇


👤 Step 1: Configure LDAP Authentication for Kong Manager

Edit your kong.conf or pass these as environment variables if you’re using a container setup.

admin_gui_auth = ldap-auth
admin_gui_auth_conf = {
  "ldap_host": "ldap.example.com",
  "ldap_port": 389,
  "ldap_base_dn": "dc=example,dc=com",
  "ldap_attribute": "uid",
  "ldap_bind_dn": "cn=admin,dc=example,dc=com",
  "ldap_password": "adminpassword",
  "start_tls": false,
  "verify_ldap_host": false
}

✅ If you’re using LDAPS, set ldap_port = 636 and start_tls = false or configure accordingly.

Restart Kong after updating this config.


👥 Step 2: Create an RBAC User Linked to the LDAP Username

Kong still needs an RBAC user that maps to the LDAP-authenticated identity.

curl -i -X POST http://localhost:8001/rbac/users \
  --data "name=jdoe" \
  --data "user_token=jdoe-admin-token"

The name here must match the LDAP uid or whatever attribute you configured with ldap_attribute.


🔐 Step 3: Assign a Role to the RBAC User

curl -i -X POST http://localhost:8001/rbac/users/jdoe/roles \
  --data "roles=read-only"  # Or "admin", "super-admin", etc.

Available roles: read-only, admin, super-admin, or your own custom roles.


🔓 Step 4: Log into Kong Manager with LDAP User

Go to your Kong Manager GUI:

https://<KONG_MANAGER_URL>:8445

Enter:

  • Username: jdoe (LDAP uid)
  • Password: LDAP user’s actual password (Kong will bind to LDAP and verify it)

🛠️ Optional: Test LDAP Config from CLI

You can test the LDAP binding from Kong CLI:

curl -i -X POST http://localhost:8001/rbac/users \
  --data "name=testuser" \
  --data "user_token=test123"

Then try logging into Kong Manager with testuser using their LDAP password.


Kong HA

Setting up Kong Gateway with high availability (HA) on-premise on bare metal servers involves several steps. Below is a comprehensive guide to achieve this setup:

Prerequisites

  1. Bare Metal Servers: Ensure you have multiple physical servers available.
  2. Network Configuration: Ensure all servers are on the same network and can communicate with each other.
  3. Data Store: Kong Gateway requires a shared data store like PostgreSQL or Cassandra. Ensure you have a highly available setup for your data store.
  4. Load Balancer: A hardware or software load balancer to distribute traffic across Kong Gateway nodes.

Step-by-Step Guide

1. Install PostgreSQL for the Shared Data Store

  1. Install PostgreSQL:

sudo apt-get update

sudo apt-get install -y postgresql postgresql-contrib

  1. Configure PostgreSQL for High Availability:
    • Set up replication between multiple PostgreSQL instances.
    • Ensure that the primary and standby instances are configured correctly.
  2. Create a Kong Database:

sudo -u postgres psql

CREATE DATABASE kong;

CREATE USER kong WITH PASSWORD ‘yourpassword’;

GRANT ALL PRIVILEGES ON DATABASE kong TO kong;

\q

2. Install Kong Gateway on Each Server

  1. Install Kong Gateway:

sudo apt-get update

sudo apt-get install -y apt-transport-https

curl -s https://packages.konghq.com/keys/kong.key | sudo apt-key add –

echo “deb https://packages.konghq.com/debian/ $(lsb_release -sc) main” | sudo tee -a /etc/apt/sources.list

sudo apt-get update

sudo apt-get install -y kong

  1. Configure Kong Gateway:
    • Create a kong.conf file on each server with the following configuration:

database = postgres

pg_host = <primary_postgresql_host>

pg_port = 5432

pg_user = kong

pg_password = yourpassword

pg_database = kong

  1. Start Kong Gateway:

kong migrations bootstrap

kong start

3. Configure Load Balancer

  1. Set Up a Load Balancer:
    • Configure your load balancer to distribute traffic across the Kong Gateway nodes.
    • Ensure the load balancer is set up for high availability (e.g., using a failover IP or DNS).
  2. Configure Health Checks:
    • Configure health checks on the load balancer to monitor the health of each Kong Gateway node.
    • Ensure that traffic is only sent to healthy nodes.

4. Set Up Failover Mechanism

  1. Database Failover:
    • Ensure your PostgreSQL setup has a failover mechanism in place (e.g., using Patroni or pgpool-II).
  2. Kong Gateway Failover:
    • Ensure that the load balancer can detect when a Kong Gateway node is down and redirect traffic to other nodes.

5. Implement Monitoring and Alerts

  1. Set Up Monitoring:
    • Use tools like Prometheus and Grafana to monitor the health and performance of your Kong Gateway nodes and PostgreSQL database.
  2. Set Up Alerts:
    • Configure alerts to notify you of any issues with the Kong Gateway nodes or the PostgreSQL database.

Example Configuration Files

PostgreSQL Configuration (pg_hba.conf):

# TYPE  DATABASE        USER            ADDRESS                 METHOD

host    kong            kong            192.168.1.0/24          md5

Kong Gateway Configuration (kong.conf):

database = postgres

pg_host = 192.168.1.10

pg_port = 5432

pg_user = kong

pg_password = yourpassword

pg_database = kong

Summary

By following these steps, you can set up a highly available Kong Gateway on bare metal servers. This setup ensures that your API gateway remains reliable and performs well under various conditions. Make sure to thoroughly test your setup to ensure that failover and load balancing work as expected.

LUKS – disks encrypt

#!/bin/bash

# Variables
DISKS=("/dev/sdb" "/dev/sdc") # List of disks to encrypt
KEYFILE="/etc/luks/keyfile"   # Keyfile path
MOUNT_POINTS=("/mnt/disk1" "/mnt/disk2") # Corresponding mount points

# Check for root privileges
if [ "$(id -u)" -ne 0 ]; then
    echo "This script must be run as root. Exiting."
    exit 1
fi

# Create the keyfile if it doesn't exist
if [ ! -f "$KEYFILE" ]; then
    echo "Creating LUKS keyfile..."
    mkdir -p "$(dirname "$KEYFILE")"
    dd if=/dev/urandom of="$KEYFILE" bs=4096 count=1
    chmod 600 "$KEYFILE"
fi

# Function to encrypt and set up a disk
encrypt_disk() {
    local DISK=$1
    local MAPPER_NAME=$2
    local MOUNT_POINT=$3

    echo "Processing $DISK..."
    
    # Check if the disk is already encrypted
    if cryptsetup isLuks "$DISK"; then
        echo "$DISK is already encrypted. Skipping."
        return
    fi

    # Format the disk with LUKS encryption
    echo "Encrypting $DISK..."
    cryptsetup luksFormat "$DISK" "$KEYFILE"
    if [ $? -ne 0 ]; then
        echo "Failed to encrypt $DISK. Exiting."
        exit 1
    fi

    # Open the encrypted disk
    echo "Opening $DISK..."
    cryptsetup luksOpen "$DISK" "$MAPPER_NAME" --key-file "$KEYFILE"

    # Create a filesystem on the encrypted disk
    echo "Creating filesystem on /dev/mapper/$MAPPER_NAME..."
    mkfs.ext4 "/dev/mapper/$MAPPER_NAME"

    # Create the mount point if it doesn't exist
    mkdir -p "$MOUNT_POINT"

    # Add entry to /etc/fstab for automatic mounting
    echo "Adding $DISK to /etc/fstab..."
    UUID=$(blkid -s UUID -o value "/dev/mapper/$MAPPER_NAME")
    echo "UUID=$UUID $MOUNT_POINT ext4 defaults 0 2" >> /etc/fstab

    # Mount the disk
    echo "Mounting $MOUNT_POINT..."
    mount "$MOUNT_POINT"
}

# Loop through disks and encrypt each one
for i in "${!DISKS[@]}"; do
    DISK="${DISKS[$i]}"
    MAPPER_NAME="luks_disk_$i"
    MOUNT_POINT="${MOUNT_POINTS[$i]}"

    encrypt_disk "$DISK" "$MAPPER_NAME" "$MOUNT_POINT"
done

echo "All disks have been encrypted and mounted."

EKS – subnet size

To determine the appropriate subnet class for an Amazon EKS (Elastic Kubernetes Service) cluster with 5 nodes, it’s important to account for both the nodes and the additional IP addresses needed for pods and other resources. Here’s a recommended approach:

Calculation and Considerations:

  1. EKS Node IP Addresses:
    • Each node will need its own IP address.
    • For 5 nodes, that’s 5 IP addresses.
  2. Pod IP Addresses:
    • By default, the Amazon VPC CNI plugin assigns one IP address per pod from the node’s subnet.
    • The number of pods per node depends on your instance type and the configuration of your Kubernetes cluster.
    • For example, if you expect each node to host around 20 pods, you’ll need approximately 100 IP addresses for pods.
  3. Additional Resources:
    • Include IP addresses for other resources like load balancers, services, etc.

Subnet Size Recommendation:

A /24 subnet provides 254 usable IP addresses, which is typically sufficient for a small EKS cluster with 5 nodes.

Example Calculation:

  • Nodes: 5 IP addresses
  • Pods: 100 IP addresses (assuming 20 pods per node)
  • Additional Resources: 10 IP addresses (for services, load balancers, etc.)

Total IP Addresses Needed: 5 (nodes) + 100 (pods) + 10 (resources) = 115 IP addresses.

Recommended Subnet Size:

A /24 subnet should be sufficient for this setup:

  • CIDR Notation: 192.168.0.0/24
  • Total IP Addresses: 256
  • Usable IP Addresses: 254

Example Configuration:

  • Subnet 1: 192.168.0.0/24

Reasons to Choose a Bigger Subnet (e.g., /22 or /20):

  1. Future Scalability: If you anticipate significant growth in the number of nodes or pods, a larger subnet will provide ample IP addresses for future expansion without the need to reconfigure your network.
  2. Flexibility: More IP addresses give you flexibility to add additional resources such as load balancers, services, or new applications.
  3. Avoiding Exhaustion: Ensuring you have a large pool of IP addresses can prevent issues related to IP address exhaustion, which can disrupt your cluster’s operations.

Example Subnet Sizes:

  • /22 Subnet:
    • Total IP Addresses: 1,024
    • Usable IP Addresses: 1,022
  • /20 Subnet:
    • Total IP Addresses: 4,096
    • Usable IP Addresses: 4,094

When to Consider Smaller Subnets (e.g., /24):

  1. Small Deployments: If your EKS cluster is small and you do not expect significant growth, a /24 subnet might be sufficient.
  2. Cost Efficiency: Smaller subnets can sometimes be more cost-effective in environments where IP address scarcity is not a concern.

For an EKS cluster with 5 nodes, I would recommend going with a /22 subnet. This gives you a healthy margin of IP addresses for your nodes, pods, and additional resources while providing room for future growth.

Encrypt multiple disks with LUKS

---
- name: Encrypt multiple disks with LUKS
  hosts: all
  become: yes
  vars:
    luks_disks:            # List of disks to encrypt
      - /dev/sdb
      - /dev/sdc
    luks_password: secret_password  # Replace or use a vault/encrypted variable
    mount_points:          # List of mount points corresponding to the disks
      - /mnt/disk1
      - /mnt/disk2

  tasks:
    - name: Ensure required packages are installed
      ansible.builtin.yum:
        name:
          - cryptsetup
        state: present

    - name: Create LUKS encryption on disks
      ansible.builtin.command:
        cmd: "echo {{ luks_password }} | cryptsetup luksFormat {{ item }} -q"
      loop: "{{ luks_disks }}"
      ignore_errors: no

    - name: Open LUKS-encrypted disks
      ansible.builtin.command:
        cmd: "echo {{ luks_password }} | cryptsetup luksOpen {{ item }} luks_{{ item | regex_replace('/dev/', '') }}"
      loop: "{{ luks_disks }}"

    - name: Format the LUKS-encrypted devices with ext4 filesystem
      ansible.builtin.command:
        cmd: "mkfs.ext4 /dev/mapper/luks_{{ item | regex_replace('/dev/', '') }}"
      loop: "{{ luks_disks }}"

    - name: Create mount points
      ansible.builtin.file:
        path: "{{ item }}"
        state: directory
      loop: "{{ mount_points }}"

    - name: Mount the LUKS devices to mount points
      ansible.builtin.mount:
        path: "{{ item.1 }}"
        src: "/dev/mapper/luks_{{ item.0 | regex_replace('/dev/', '') }}"
        fstype: ext4
        state: mounted
      loop: "{{ luks_disks | zip(mount_points) | list }}"

    - name: Add entries to /etc/crypttab
      ansible.builtin.lineinfile:
        path: /etc/crypttab
        line: "luks_{{ item | regex_replace('/dev/', '') }} {{ item }} none luks"
      loop: "{{ luks_disks }}"
      create: yes

    - name: Add entries to /etc/fstab
      ansible.builtin.lineinfile:
        path: /etc/fstab
        line: "/dev/mapper/luks_{{ item.0 | regex_replace('/dev/', '') }} {{ item.1 }} ext4 defaults 0 0"
      loop: "{{ luks_disks | zip(mount_points) | list }}"
      create: yes
a

## output 

Processing /dev/sdc...
Encrypting /dev/sdc...

WARNING!
========
This will overwrite data on /dev/sdc irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
Opening /dev/sdc...
Device luks_disk_0 already exists.
Creating filesystem on /dev/mapper/luks_disk_0...
mke2fs 1.46.5 (30-Dec-2021)
/dev/mapper/luks_disk_0 is mounted; will not make a filesystem here!
Adding /dev/sdc to /etc/fstab...
Mounting /mnt/disk2...
mount: (hint) your fstab has been modified, but systemd still uses
       the old version; use 'systemctl daemon-reload' to reload.
Processing /dev/sdd...
Encrypting /dev/sdd...

WARNING!
========
This will overwrite data on /dev/sdd irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
Opening /dev/sdd...
Creating filesystem on /dev/mapper/luks_disk_1...
mke2fs 1.46.5 (30-Dec-2021)
Creating filesystem with 2617344 4k blocks and 655360 inodes
Filesystem UUID: d0bb5504-abf9-4e00-8670-59d8fa92b883
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done 

Adding /dev/sdd to /etc/fstab...
Mounting /mnt/disk3...
mount: (hint) your fstab has been modified, but systemd still uses
       the old version; use 'systemctl daemon-reload' to reload.
All disks have been encrypted and mounted.

Rack awareness in Hadoop

Rack awareness in Hadoop is a concept used to improve data availability and network efficiency within a Hadoop cluster. Here’s a breakdown of what it entails:

What is Rack Awareness?

Rack awareness is the ability of Hadoop to recognize the physical network topology of the cluster. This means that Hadoop knows the location of each DataNode (the nodes that store data) within the network2.

Why is Rack Awareness Important?

  1. Fault Tolerance: By placing replicas of data blocks on different racks, Hadoop ensures that even if an entire rack fails, the data is still available from another rack.
  2. Network Efficiency: Hadoop tries to place replicas on the same rack or nearby racks to reduce network traffic and improve read/write performance.
  3. High Availability: Ensures that data is available even in the event of network failures or partitions within the cluster.

How Does Rack Awareness Work?

  • NameNode: The NameNode, which manages the file system namespace and metadata, maintains the rack information for each DataNode.
  • Block Placement Policy: When Hadoop stores data blocks, it uses a block placement policy that considers rack information to place replicas on different racks.
  • Topology Script or Java Class: Hadoop can use either an external topology script or a Java class to obtain rack information. The configuration file specifies which method to use3.

Example Configuration

Here’s an example of how to configure rack awareness in Hadoop:

  1. Create a Topology Script: Write a script that maps IP addresses to rack identifiers.
  2. Configure Hadoop: Set the net.topology.script.file.name parameter in the Hadoop configuration file to point to your script.
  3. Restart Hadoop Services: Restart the Hadoop services to apply the new configuration.

By implementing rack awareness, Hadoop can optimize data placement and improve the overall performance and reliability of the cluster.

Topology Script Example

This script maps IP addresses to rack IDs. Let’s assume we have a few DataNodes with specific IP addresses, and we want to assign them to different racks.

  1. Create the Script: Save the following script as topology-script.sh.

#!/bin/bash

# Script to map IP addresses to rack identifiers

# Default rack if no match is found

DEFAULT_RACK=”/default-rack”

# Function to map IP to rack

map_ip_to_rack() {

  case $1 in

    192.168.1.1) echo “/rack1” ;;

    192.168.1.2) echo “/rack1” ;;

    192.168.1.3) echo “/rack2” ;;

    192.168.1.4) echo “/rack2” ;;

    192.168.1.5) echo “/rack3” ;;

    192.168.1.6) echo “/rack3” ;;

    *) echo $DEFAULT_RACK ;;

  esac

}

# Read IP addresses from stdin

while read -r line; do

  map_ip_to_rack “$line”

done

  1. Make the Script Executable:

chmod +x topology-script.sh

  1. Configure Hadoop: Update your Hadoop configuration to use this script. Add the following line to your hdfs-site.xml file:

<property>

  <name>net.topology.script.file.name</name>

  <value>/path/to/topology-script.sh</value>

</property>

  1. Restart Hadoop Services: Restart your Hadoop services to apply the new configuration.

This script maps specific IP addresses to rack IDs and uses a default rack if no match is found. Adjust the IP addresses and rack IDs according to your cluster setup.

High Availability (HA) for two Kong API Gateway instances

To set up High Availability (HA) for two Kong API Gateway instances, you need to configure them to work seamlessly, ensuring reliability and fault tolerance. Below are the steps in detail:


1. HA Architecture Overview

In an HA setup, two Kong Gateway instances are deployed behind a load balancer. Both instances share the same configuration data, stored in a database (e.g., PostgreSQL or Cassandra), or operate in DB-less mode if configuration is managed via automation.

Components of the Architecture

  • Kong API Gateway Instances: Two or more Kong nodes deployed on separate servers.
  • Load Balancer: Distributes traffic to Kong nodes.
  • Database (Optional): A shared PostgreSQL or Cassandra instance for storing configuration if not using DB-less mode.
  • Health Monitoring: Ensures requests are routed only to healthy Kong nodes.

2. Setup Steps

Step 1: Install Kong on Two Nodes

  1. Follow the Kong installation guide for your operating system.
    • Kong Installation Guide
  2. Ensure both nodes are installed with the same version of Kong.

Step 2: Configure a Shared Database (If Not Using DB-less Mode)

Database Setup:

  1. Install PostgreSQL or Cassandra on a separate server (or cluster for HA).
  2. Create a Kong database user and database.
    • Example for PostgreSQL:

CREATE USER kong WITH PASSWORD ‘kong’;

CREATE DATABASE kong OWNER kong;

  1. Update the kong.conf file on both nodes to point to the shared database:

database = postgres

pg_host = <DATABASE_IP>

pg_port = 5432

pg_user = kong

pg_password = kong

pg_database = kong

  1. Run the Kong migrations (only on one node):

kong migrations bootstrap


Step 3: DB-less Mode (Optional)

If you prefer DB-less mode for better scalability and faster failover:

  1. Use declarative configuration with a YAML file (kong.yml).
  2. Place the configuration file on both Kong nodes.
  3. Set the kong.conf file to use DB-less mode:

database = off

declarative_config = /path/to/kong.yml


Step 4: Configure Load Balancer

Set up a load balancer to distribute traffic between the two Kong instances.

Options:

  • F5, HAProxy, or NGINX for on-premises environments.
  • AWS Elastic Load Balancer (ELB) for cloud-based setups.

Configuration Example:

  • Backend pool: Add both Kong instances (Node1_IP and Node2_IP).
  • Health checks: Use HTTP health checks to monitor the /status endpoint of Kong.

curl -X GET http://<KONG_INSTANCE&gt;:8001/status


Step 5: Synchronize Configuration Across Nodes

For consistency, ensure both Kong nodes share the same configuration.

DB-Mode Synchronization:

  • Configurations are automatically synchronized via the shared database.

DB-less Mode Synchronization:

  • Use configuration management tools like Ansible, Terraform, or CI/CD pipelines to deploy the kong.yml file to all nodes.

Step 6: Enable Kong Clustering (Optional)

If using Cassandra as the database, configure Kong clustering for HA.

  1. Enable clustering in the kong.conf file:

cluster_listen = 0.0.0.0:7946

cluster_listen_rpc = 127.0.0.1:7373

  1. Ensure that ports 7946 (for gossip communication) and 7373 (for RPC communication) are open between Kong nodes.

Step 7: Configure SSL (TLS)

  1. Generate SSL certificates for your domain.
  2. Configure Kong to use these certificates for the gateway.

curl -X POST http://<KONG_ADMIN_API>/certificates \

  –data “cert=@/path/to/cert.pem” \

  –data “key=@/path/to/key.pem” \

  –data “snis[]=example.com”


Step 8: Test the Setup

Health Check:

  • Verify the /status endpoint on both nodes:

curl -X GET http://<KONG_NODE_IP&gt;:8001/status

Request Routing:

  • Send a test request through the load balancer:

curl -X GET http://<LOAD_BALANCER_IP&gt;:8000/your-api

  • Verify logs on both Kong instances to ensure traffic is distributed.

Example HA Diagram

plaintext

Copy code

                       +———————–+

                       |    Load Balancer      |

                       | (F5, ELB, HAProxy)    |

                       +———-+————+

                                  |

                +—————–+—————–+

                |                                    |

    +———–+———–+           +———–+———–+

    |   Kong Instance 1       |           |   Kong Instance 2      |

    |    (Node1_IP)           |           |    (Node2_IP)           |

+———–+————+           +———–+————+

                |                                   |

    +———–+————+           +———–+————+

    | Shared Database (DB-Mode) OR Shared Config File (DB-less Mode)|

    +———————–+—————————————-+


Best Practices for Kong HA

  1. Load Balancer Health Checks:
    • Ensure your load balancer only forwards requests to healthy Kong instances.
  2. Database High Availability:
    • Use a clustered database for the shared configuration.
  3. Monitoring:
    • Integrate Kong with monitoring tools (e.g., Prometheus, Grafana) to track performance.
  4. Rate Limiting:
    • Configure plugins for rate limiting to prevent node overload.
  5. Session Persistence:
    • Use sticky sessions if required by your application.

By following these steps, you’ll achieve a robust, highly available Kong Gateway setup. Let me know if you need help with specific configurations!