kong – 2 services same route

In Kong Gateway, you cannot have two services bound to the exact same route β€” a route must be unique in terms of its combination of matching rules (such as paths, hosts, methods, etc.).


🚫 Why You Can’t Have Duplicate Routes

Kong uses the route’s matching criteria to determine which service should handle a request. If two routes have the same matching rules, Kong can’t disambiguate, which leads to a conflict error when trying to create the second route.

For example:

# Service A
curl -i -X POST http://localhost:8001/services/ --data name=service-a --data url=http://api-a
curl -i -X POST http://localhost:8001/routes --data name=route-a --data service.name=service-a --data paths[]=/api

# Service B
curl -i -X POST http://localhost:8001/services/ --data name=service-b --data url=http://api-b
curl -i -X POST http://localhost:8001/routes --data name=route-b --data service.name=service-b --data paths[]=/api

β›” The second POST /routes will fail with:

HTTP 409 Conflict – duplicate entry


βœ… Workarounds / Alternatives

GoalSolution
Blue/Green or Canary DeploymentsUse Kong Plugins (e.g., traffic-split, canary) or upstreams and targets instead of duplicate routes.
Different consumers/users hitting different backendsUse request transformers, ACLs, or Kong Enterprise Route-by-header/Route-by-consumer plugins.
Same path, different method or hostYou can differentiate routes by methods[], hosts[], or headers.

πŸ§ͺ Example: Two Routes with Same Path, Different Hosts

# Route 1
curl -i -X POST http://localhost:8001/routes \
  --data name=api-v1 \
  --data paths[]=/api \
  --data hosts[]=v1.example.com \
  --data service.name=service-a

# Route 2
curl -i -X POST http://localhost:8001/routes \
  --data name=api-v2 \
  --data paths[]=/api \
  --data hosts[]=v2.example.com \
  --data service.name=service-b

These can coexist because their hosts[] fields are different.


🧠 Summary

Route Matching RuleMust Be Unique For
paths[]Same host/methods/headers
hosts[]If combined with same path
methods[], headersCan disambiguate routes with same path

If you’re trying to achieve load balancing, blue-green deployment, or AB testing between services at the same route β€” I can help you set that up using upstreams + targets or the traffic-split plugin.

Would you like an example?

LEAPP upgrade – multiple kernel issue

Newest installed kernel not in use

If the newest installed kernel is not in use even after rebooting, it typically means your system is booting into an older kernel version. This happens due to the default GRUB settings or other configuration issues.

Here’s how you can resolve the issue step-by-step:


1. Verify the Currently Running Kernel

  • Check the kernel currently in use:

uname -r

  • Example output:

CopyEdit

3.10.0-1127.el7.x86_64

  • Compare this with the newest installed kernel version (see step 2).

2. List All Installed Kernels

  • List all installed kernel packages:

rpm -qa | grep kernel | grep -v firmware

  • Example output:

kernel-3.10.0-1127.el7.x86_64

kernel-3.10.0-1160.el7.x86_64

  • Identify the newest kernel (e.g., 3.10.0-1160.el7.x86_64).

3. Check GRUB Default Boot Entry

  • Display the current GRUB default boot entry:

grub2-editenv list

  • You should see something like this:

saved_entry=0

  • If the saved_entry is pointing to an older kernel, the system is booting into it by default.

4. Update GRUB to Use the Newest Kernel

Option A: Use the First Kernel (Default GRUB Behavior)

  • Edit the GRUB configuration:

sudo vi /etc/default/grub

  • Set the GRUB_DEFAULT value to 0 (the first kernel in the GRUB list):

GRUB_DEFAULT=0

  • Save and exit the file.
  • Regenerate the GRUB configuration:

sudo grub2-mkconfig -o /boot/grub2/grub.cfg

Option B: Manually Specify the Newest Kernel

  • Run this command to view all available kernel entries:

awk -F\’ ‘$1==”menuentry ” {print i++ ” : ” $2}’ /boot/grub2/grub.cfg

  • Example output:

0 : CentOS Linux (3.10.0-1160.el7.x86_64)

1 : CentOS Linux (3.10.0-1127.el7.x86_64)

2 : CentOS Linux (0-rescue-…)

  • Set GRUB_DEFAULT to the corresponding entry for the newest kernel (e.g., 0):

sudo grub2-set-default 0

  • Verify the setting:

grub2-editenv list


5. Reboot the System

  • Reboot your system to load the correct kernel:

sudo reboot

  • After reboot, confirm that the new kernel is in use:

uname -r


6. Remove Old Kernels (Optional)

  • To prevent confusion in the future, you can remove unused older kernels:

sudo package-cleanup –oldkernels –count=1

  • This retains only the most recent kernel.

7. Troubleshooting

  • Manually Select the Kernel: If the system still boots into the wrong kernel, you can manually select the desired kernel during the GRUB menu at boot time. To enable the GRUB menu:
    1. Edit /etc/default/grub:

sudo vi /etc/default/grub

  • Set GRUB_TIMEOUT to a non-zero value (e.g., 5):

GRUB_TIMEOUT=5

  • Save and regenerate the GRUB configuration:

sudo grub2-mkconfig -o /boot/grub2/grub.cfg

  • Kernel Missing After Update: Ensure the new kernel is properly installed. Reinstall it if necessary:

sudo yum install kernel

Securing an HPE Ezmeral Data Fabric

Security

Securing an HPE Ezmeral Data Fabric (formerly MapR) Hadoop cluster involves implementing a multi-layered security strategy that covers authentication, authorization, encryption, and monitoring. Below is a comprehensive guide to securing your HPE Ezmeral Hadoop cluster:


1. Authentication

Implement strong authentication mechanisms to ensure that only authorized users and applications can access the cluster.

  • Kerberos Integration:
    • Use Kerberos for secure authentication of users and services.
    • Configure Kerberos key distribution centers (KDCs) and set up service principals for all Hadoop components.
  • LDAP/AD Integration:
    • Integrate the cluster with LDAP or Active Directory (AD) for centralized user authentication.
    • Use Pluggable Authentication Modules (PAM) to synchronize user credentials.
  • Token-based Authentication:
    • Enable token-based authentication for inter-service communication to enhance security and reduce Kerberos dependency.

2. Authorization

Implement role-based access control (RBAC) to manage user and application permissions.

  • Access Control Lists (ACLs):
    • Configure ACLs for Hadoop Distributed File System (HDFS), YARN, and other services.
    • Restrict access to sensitive data directories.
  • Apache Ranger Integration:
    • Use Apache Ranger for centralized authorization management.
    • Define fine-grained policies for HDFS, Hive, and other components.
  • Group-based Permissions:
    • Assign users to appropriate groups and define group-level permissions for ease of management.

3. Encryption

Protect data at rest and in transit to prevent unauthorized access.

  • Data-at-Rest Encryption:
    • Use dm-crypt/LUKS for disk-level encryption of storage volumes.
    • Enable HDFS Transparent Data Encryption (TDE) for encrypting data blocks.
  • Data-in-Transit Encryption:
    • Configure TLS/SSL for all inter-service communication.
    • Use certificates signed by a trusted certificate authority (CA).
  • Key Management:
    • Implement a secure key management system, such as HPE Ezmeral Data Fabric’s built-in key management service or an external solution like HashiCorp Vault.

4. Network Security

Restrict network access to the cluster and its services.

  • Firewall Rules:
    • Limit inbound and outbound traffic to required ports only.
    • Use network segmentation to isolate the Hadoop cluster.
  • Private Networking:
    • Deploy the cluster in a private network (e.g., VPC on AWS or Azure).
    • Use VPN or Direct Connect for secure remote access.
  • Gateway Nodes:
    • Restrict direct access to Hadoop cluster nodes by using gateway or edge nodes.

5. Auditing and Monitoring

Monitor cluster activity and audit logs to detect and respond to security incidents.

  • Log Management:
    • Enable and centralize audit logging for HDFS, YARN, Hive, and other components.
    • Use tools like Splunk, Elasticsearch, or Fluentd for log aggregation and analysis.
  • Intrusion Detection:
    • Deploy intrusion detection systems (IDS) or intrusion prevention systems (IPS) to monitor network traffic.
  • Real-time Alerts:
    • Set up alerts for anomalous activities using monitoring tools like Prometheus, Grafana, or Nagios.

6. Secure Cluster Configuration

Ensure that the cluster components are securely configured.

  • Hadoop Configuration Files:
    • Disable unnecessary services and ports.
    • Set secure defaults for core-site.xml, hdfs-site.xml, and yarn-site.xml.
  • Service Accounts:
    • Run Hadoop services under dedicated user accounts with minimal privileges.
  • Regular Updates:
    • Keep the Hadoop distribution and all dependencies updated with the latest security patches.

7. User Security Awareness

Educate users on secure practices.

  • Strong Passwords:
    • Enforce password complexity requirements and periodic password changes.
  • Access Reviews:
    • Conduct regular access reviews to ensure that only authorized users have access.
  • Security Training:
    • Provide security awareness training to users and administrators.

8. Backup and Disaster Recovery

Ensure the availability and integrity of your data.

  • Backup Policy:
    • Regularly back up metadata and critical data to secure storage.
  • Disaster Recovery:
    • Implement a disaster recovery plan with off-site replication.

9. Compliance

Ensure the cluster complies with industry standards and regulations.

  • Data Protection Regulations:
    • Adhere to GDPR, HIPAA, PCI DSS, or other relevant standards.
    • Implement data masking and anonymization where required.
  • Third-party Audits:
    • Conduct periodic security assessments and audits.

By following these practices, you can ensure a robust security posture for your HPE Ezmeral Hadoop cluster.

Rack awareness in Hadoop

Rack awareness in Hadoop is a concept used to improve data availability and network efficiency within a Hadoop cluster. Here’s a breakdown of what it entails:

What is Rack Awareness?

Rack awareness is the ability of Hadoop to recognize the physical network topology of the cluster. This means that Hadoop knows the location of each DataNode (the nodes that store data) within the network2.

Why is Rack Awareness Important?

  1. Fault Tolerance: By placing replicas of data blocks on different racks, Hadoop ensures that even if an entire rack fails, the data is still available from another rack.
  2. Network Efficiency: Hadoop tries to place replicas on the same rack or nearby racks to reduce network traffic and improve read/write performance.
  3. High Availability: Ensures that data is available even in the event of network failures or partitions within the cluster.

How Does Rack Awareness Work?

  • NameNode: The NameNode, which manages the file system namespace and metadata, maintains the rack information for each DataNode.
  • Block Placement Policy: When Hadoop stores data blocks, it uses a block placement policy that considers rack information to place replicas on different racks.
  • Topology Script or Java Class: Hadoop can use either an external topology script or a Java class to obtain rack information. The configuration file specifies which method to use3.

Example Configuration

Here’s an example of how to configure rack awareness in Hadoop:

  1. Create a Topology Script: Write a script that maps IP addresses to rack identifiers.
  2. Configure Hadoop: Set the net.topology.script.file.name parameter in the Hadoop configuration file to point to your script.
  3. Restart Hadoop Services: Restart the Hadoop services to apply the new configuration.

By implementing rack awareness, Hadoop can optimize data placement and improve the overall performance and reliability of the cluster.

Topology Script Example

This script maps IP addresses to rack IDs. Let’s assume we have a few DataNodes with specific IP addresses, and we want to assign them to different racks.

  1. Create the Script: Save the following script as topology-script.sh.

#!/bin/bash

# Script to map IP addresses to rack identifiers

# Default rack if no match is found

DEFAULT_RACK=”/default-rack”

# Function to map IP to rack

map_ip_to_rack() {

  case $1 in

    192.168.1.1) echo “/rack1” ;;

    192.168.1.2) echo “/rack1” ;;

    192.168.1.3) echo “/rack2” ;;

    192.168.1.4) echo “/rack2” ;;

    192.168.1.5) echo “/rack3” ;;

    192.168.1.6) echo “/rack3” ;;

    *) echo $DEFAULT_RACK ;;

  esac

}

# Read IP addresses from stdin

while read -r line; do

  map_ip_to_rack “$line”

done

  1. Make the Script Executable:

chmod +x topology-script.sh

  1. Configure Hadoop: Update your Hadoop configuration to use this script. Add the following line to your hdfs-site.xml file:

<property>

  <name>net.topology.script.file.name</name>

  <value>/path/to/topology-script.sh</value>

</property>

  1. Restart Hadoop Services: Restart your Hadoop services to apply the new configuration.

This script maps specific IP addresses to rack IDs and uses a default rack if no match is found. Adjust the IP addresses and rack IDs according to your cluster setup.

Ping auth plugin

The Ping Auth plugin for Kong API Gateway integrates with Ping Identity’s PingAuthorize to provide attribute-based access control and policy decisions. This plugin allows Kong to utilize Ping products for dynamic authorization, making it easier to control access to your API resources1.

Key Features:

  • Attribute-Based Access Control (ABAC): Allows fine-grained access control based on user attributes and policies.
  • Dynamic Authorization: Policies can be updated in real-time without redeploying the API gateway.
  • Mutual TLS (mTLS): Supports client certificate authentication using mTLS.
  • Sideband API Protocol: Communicates with Ping services to retrieve authorization decisions.

Installation:

  1. Download the Plugin: Get the ping-auth plugin from Luarocks.
  2. Install the Plugin: Use LuaRocks to install the plugin:

luarocks install kong-plugin-ping-auth

  1. Configure Kong: Add the plugin to your Kong configuration:

plugins = bundled,ping-auth

  1. Apply the Plugin: Enable and configure the plugin via Kong’s admin UI or API.

Example Configuration:

plugins:

  – name: ping-auth

    config:

      service_url: “https://your-ping-service/policy&#8221;

This setup allows Kong to communicate with Ping services to handle authorization decisions.

Would you like more details on configuring this plugin or any other specific information?

https://github.com/pingidentity/kong-plugin-ping-auth