aws

A 401 Unauthorized error in Kong Gateway typically indicates that the request lacks valid authentication credentials required to access the resource. Here’s how you can troubleshoot and resolve this issue:

techhadoop , , , ,

🔍 Common Causes and Solutions

1. Missing or Invalid API Key (Key Authentication Plugin)

If you’re using the Key Authentication plugin, Kong expects a valid API key in the request.(Kong Docs)

  • Symptom: Receiving a 401 error with the message "No API key found in request" or "Invalid authentication credentials".(Turbolab Technologies)
  • Solution:
    • Create a Consumer: curl -i -X POST http://localhost:8001/consumers/ --data "username=your_username"
    • Assign an API Key to the Consumer: curl -i -X POST http://localhost:8001/consumers/your_username/key-auth --data "key=your_api_key"
    • Include the API Key in Your Request: curl -i http://lo calhost:8000/your_service \ -H "apikey: your_api_key"
    • Note: The header name (apikey) should match the key_names configured in your Key Authentication plugin.(Kong Docs)

2. Expired or Invalid JWT Token (JWT Plugin)

If you’re using the JWT plugin, ensure that the token is valid and not expired.

  • Symptom: Receiving a 401 error with the message "Invalid token" or "Token expired".
  • Solution:
    • Verify the Token: Use tools like jwt.io to decode and inspect the token.
    • Check Expiration: Ensure the exp claim has not passed.
    • Validate Signature: Confirm that the token’s signature matches the expected signing key.(Auth0 Community)

3. OpenID Connect (OIDC) Plugin Issues

When using the OIDC plugin, a 401 error can occur if the plugin’s cache is outdated after changes to the Identity Provider (IdP).(support.konghq.com)

  • Symptom: Receiving a 401 error after updating the IdP’s signing certificate.
  • Solution:
    • Clear Plugin Cache: Restart Kong or clear the plugin’s cache to fetch the updated signing keys from the IdP.
    • Verify IdP Configuration: Ensure that the IdP’s metadata URL is correctly configured in the plugin settings.

4. Plugin Configuration on Specific Routes or Services

Ensure that authentication plugins are correctly configured on the intended routes or services.

  • Symptom: Receiving a 401 error on specific endpoints.
  • Solution:
    • Check Plugin Association: Use the Admin API to list plugins associated with your routes or services. curl http://localhost:8001/plugins
    • Adjust Configuration: Ensure that the plugin is enabled and correctly configured for the desired routes or services.

🛠️ Additional Troubleshooting Steps

  • Inspect Kong Logs: Check the Kong error logs for detailed information.
  tail -f /usr/local/kong/logs/error.log
  • Use Debug Headers: Add the X-Kong-Debug: 1 header to your requests to get more insight into how Kong processes them.
  • Verify Plugin Order: Ensure that authentication plugins are executed in the correct order, especially when multiple plugins are used.

If you can provide more details about your setup—such as the authentication plugins in use, specific routes or services affected, and sample requests—I can offer more targeted assistance.

Kong HA

techhadoop , , , ,

Setting up Kong Gateway with high availability (HA) on-premise on bare metal servers involves several steps. Below is a comprehensive guide to achieve this setup:

Prerequisites

  1. Bare Metal Servers: Ensure you have multiple physical servers available.
  2. Network Configuration: Ensure all servers are on the same network and can communicate with each other.
  3. Data Store: Kong Gateway requires a shared data store like PostgreSQL or Cassandra. Ensure you have a highly available setup for your data store.
  4. Load Balancer: A hardware or software load balancer to distribute traffic across Kong Gateway nodes.

Step-by-Step Guide

1. Install PostgreSQL for the Shared Data Store

  1. Install PostgreSQL:

sudo apt-get update

sudo apt-get install -y postgresql postgresql-contrib

  1. Configure PostgreSQL for High Availability:
    • Set up replication between multiple PostgreSQL instances.
    • Ensure that the primary and standby instances are configured correctly.
  2. Create a Kong Database:

sudo -u postgres psql

CREATE DATABASE kong;

CREATE USER kong WITH PASSWORD ‘yourpassword’;

GRANT ALL PRIVILEGES ON DATABASE kong TO kong;

\q

2. Install Kong Gateway on Each Server

  1. Install Kong Gateway:

sudo apt-get update

sudo apt-get install -y apt-transport-https

curl -s https://packages.konghq.com/keys/kong.key | sudo apt-key add –

echo “deb https://packages.konghq.com/debian/ $(lsb_release -sc) main” | sudo tee -a /etc/apt/sources.list

sudo apt-get update

sudo apt-get install -y kong

  1. Configure Kong Gateway:
    • Create a kong.conf file on each server with the following configuration:

database = postgres

pg_host = <primary_postgresql_host>

pg_port = 5432

pg_user = kong

pg_password = yourpassword

pg_database = kong

  1. Start Kong Gateway:

kong migrations bootstrap

kong start

3. Configure Load Balancer

  1. Set Up a Load Balancer:
    • Configure your load balancer to distribute traffic across the Kong Gateway nodes.
    • Ensure the load balancer is set up for high availability (e.g., using a failover IP or DNS).
  2. Configure Health Checks:
    • Configure health checks on the load balancer to monitor the health of each Kong Gateway node.
    • Ensure that traffic is only sent to healthy nodes.

4. Set Up Failover Mechanism

  1. Database Failover:
    • Ensure your PostgreSQL setup has a failover mechanism in place (e.g., using Patroni or pgpool-II).
  2. Kong Gateway Failover:
    • Ensure that the load balancer can detect when a Kong Gateway node is down and redirect traffic to other nodes.

5. Implement Monitoring and Alerts

  1. Set Up Monitoring:
    • Use tools like Prometheus and Grafana to monitor the health and performance of your Kong Gateway nodes and PostgreSQL database.
  2. Set Up Alerts:
    • Configure alerts to notify you of any issues with the Kong Gateway nodes or the PostgreSQL database.

Example Configuration Files

PostgreSQL Configuration (pg_hba.conf):

# TYPE  DATABASE        USER            ADDRESS                 METHOD

host    kong            kong            192.168.1.0/24          md5

Kong Gateway Configuration (kong.conf):

database = postgres

pg_host = 192.168.1.10

pg_port = 5432

pg_user = kong

pg_password = yourpassword

pg_database = kong

Summary

By following these steps, you can set up a highly available Kong Gateway on bare metal servers. This setup ensures that your API gateway remains reliable and performs well under various conditions. Make sure to thoroughly test your setup to ensure that failover and load balancing work as expected.

EKS – subnet size

techhadoop , , , ,

To determine the appropriate subnet class for an Amazon EKS (Elastic Kubernetes Service) cluster with 5 nodes, it’s important to account for both the nodes and the additional IP addresses needed for pods and other resources. Here’s a recommended approach:

Calculation and Considerations:

  1. EKS Node IP Addresses:
    • Each node will need its own IP address.
    • For 5 nodes, that’s 5 IP addresses.
  2. Pod IP Addresses:
    • By default, the Amazon VPC CNI plugin assigns one IP address per pod from the node’s subnet.
    • The number of pods per node depends on your instance type and the configuration of your Kubernetes cluster.
    • For example, if you expect each node to host around 20 pods, you’ll need approximately 100 IP addresses for pods.
  3. Additional Resources:
    • Include IP addresses for other resources like load balancers, services, etc.

Subnet Size Recommendation:

A /24 subnet provides 254 usable IP addresses, which is typically sufficient for a small EKS cluster with 5 nodes.

Example Calculation:

  • Nodes: 5 IP addresses
  • Pods: 100 IP addresses (assuming 20 pods per node)
  • Additional Resources: 10 IP addresses (for services, load balancers, etc.)

Total IP Addresses Needed: 5 (nodes) + 100 (pods) + 10 (resources) = 115 IP addresses.

Recommended Subnet Size:

A /24 subnet should be sufficient for this setup:

  • CIDR Notation: 192.168.0.0/24
  • Total IP Addresses: 256
  • Usable IP Addresses: 254

Example Configuration:

  • Subnet 1: 192.168.0.0/24

Reasons to Choose a Bigger Subnet (e.g., /22 or /20):

  1. Future Scalability: If you anticipate significant growth in the number of nodes or pods, a larger subnet will provide ample IP addresses for future expansion without the need to reconfigure your network.
  2. Flexibility: More IP addresses give you flexibility to add additional resources such as load balancers, services, or new applications.
  3. Avoiding Exhaustion: Ensuring you have a large pool of IP addresses can prevent issues related to IP address exhaustion, which can disrupt your cluster’s operations.

Example Subnet Sizes:

  • /22 Subnet:
    • Total IP Addresses: 1,024
    • Usable IP Addresses: 1,022
  • /20 Subnet:
    • Total IP Addresses: 4,096
    • Usable IP Addresses: 4,094

When to Consider Smaller Subnets (e.g., /24):

  1. Small Deployments: If your EKS cluster is small and you do not expect significant growth, a /24 subnet might be sufficient.
  2. Cost Efficiency: Smaller subnets can sometimes be more cost-effective in environments where IP address scarcity is not a concern.

For an EKS cluster with 5 nodes, I would recommend going with a /22 subnet. This gives you a healthy margin of IP addresses for your nodes, pods, and additional resources while providing room for future growth.

High Availability (HA) for two Kong API Gateway instances

techhadoop , , , ,

To set up High Availability (HA) for two Kong API Gateway instances, you need to configure them to work seamlessly, ensuring reliability and fault tolerance. Below are the steps in detail:

1. HA Architecture Overview

In an HA setup, two Kong Gateway instances are deployed behind a load balancer. Both instances share the same configuration data, stored in a database (e.g., PostgreSQL or Cassandra), or operate in DB-less mode if configuration is managed via automation.

Components of the Architecture

  • Kong API Gateway Instances: Two or more Kong nodes deployed on separate servers.
  • Load Balancer: Distributes traffic to Kong nodes.
  • Database (Optional): A shared PostgreSQL or Cassandra instance for storing configuration if not using DB-less mode.
  • Health Monitoring: Ensures requests are routed only to healthy Kong nodes.

2. Setup Steps

Step 1: Install Kong on Two Nodes

  1. Follow the Kong installation guide for your operating system.
    • Kong Installation Guide
  2. Ensure both nodes are installed with the same version of Kong.

Step 2: Configure a Shared Database (If Not Using DB-less Mode)

Database Setup:

  1. Install PostgreSQL or Cassandra on a separate server (or cluster for HA).
  2. Create a Kong database user and database.
    • Example for PostgreSQL:

CREATE USER kong WITH PASSWORD ‘kong’;

CREATE DATABASE kong OWNER kong;

  1. Update the kong.conf file on both nodes to point to the shared database:

database = postgres

pg_host = <DATABASE_IP>

pg_port = 5432

pg_user = kong

pg_password = kong

pg_database = kong

  1. Run the Kong migrations (only on one node):

kong migrations bootstrap

Step 3: DB-less Mode (Optional)

If you prefer DB-less mode for better scalability and faster failover:

  1. Use declarative configuration with a YAML file (kong.yml).
  2. Place the configuration file on both Kong nodes.
  3. Set the kong.conf file to use DB-less mode:

database = off

declarative_config = /path/to/kong.yml

Step 4: Configure Load Balancer

Set up a load balancer to distribute traffic between the two Kong instances.

Options:

  • F5, HAProxy, or NGINX for on-premises environments.
  • AWS Elastic Load Balancer (ELB) for cloud-based setups.

Configuration Example:

  • Backend pool: Add both Kong instances (Node1_IP and Node2_IP).
  • Health checks: Use HTTP health checks to monitor the /status endpoint of Kong.

curl -X GET http://<KONG_INSTANCE&gt;:8001/status

Step 5: Synchronize Configuration Across Nodes

For consistency, ensure both Kong nodes share the same configuration.

DB-Mode Synchronization:

  • Configurations are automatically synchronized via the shared database.

DB-less Mode Synchronization:

  • Use configuration management tools like Ansible, Terraform, or CI/CD pipelines to deploy the kong.yml file to all nodes.

Step 6: Enable Kong Clustering (Optional)

If using Cassandra as the database, configure Kong clustering for HA.

  1. Enable clustering in the kong.conf file:

cluster_listen = 0.0.0.0:7946

cluster_listen_rpc = 127.0.0.1:7373

  1. Ensure that ports 7946 (for gossip communication) and 7373 (for RPC communication) are open between Kong nodes.

Step 7: Configure SSL (TLS)

  1. Generate SSL certificates for your domain.
  2. Configure Kong to use these certificates for the gateway.

curl -X POST http://<KONG_ADMIN_API>/certificates \

  –data “cert=@/path/to/cert.pem” \

  –data “key=@/path/to/key.pem” \

  –data “snis[]=example.com”

Step 8: Test the Setup

Health Check:

  • Verify the /status endpoint on both nodes:

curl -X GET http://<KONG_NODE_IP&gt;:8001/status

Request Routing:

  • Send a test request through the load balancer:

curl -X GET http://<LOAD_BALANCER_IP&gt;:8000/your-api

  • Verify logs on both Kong instances to ensure traffic is distributed.

Example HA Diagram

plaintext

Copy code

                       +———————–+

                       |    Load Balancer      |

                       | (F5, ELB, HAProxy)    |

                       +———-+————+

                                  |

                +—————–+—————–+

                |                                    |

    +———–+———–+           +———–+———–+

    |   Kong Instance 1       |           |   Kong Instance 2      |

    |    (Node1_IP)           |           |    (Node2_IP)           |

+———–+————+           +———–+————+

                |                                   |

    +———–+————+           +———–+————+

    | Shared Database (DB-Mode) OR Shared Config File (DB-less Mode)|

    +———————–+—————————————-+

Best Practices for Kong HA

  1. Load Balancer Health Checks:
    • Ensure your load balancer only forwards requests to healthy Kong instances.
  2. Database High Availability:
    • Use a clustered database for the shared configuration.
  3. Monitoring:
    • Integrate Kong with monitoring tools (e.g., Prometheus, Grafana) to track performance.
  4. Rate Limiting:
    • Configure plugins for rate limiting to prevent node overload.
  5. Session Persistence:
    • Use sticky sessions if required by your application.

By following these steps, you’ll achieve a robust, highly available Kong Gateway setup. Let me know if you need help with specific configurations!

Setup services and route in Kong API Gateway

techhadoop , , , ,
  1. Shell script

<code>

#!/bin/bash

#Set Kong Admin API URL

KONG_ADMIN_URL=”http://localhost:8001&#8243;

#Define an array of services and routes

declare -A services
services=(
[“service11″]=”http://example11.com:8080&#8221;
[“service12″]=”http://example12.com:8080&#8221;
[“service13″]=”http://example13.com:8080&#8221;
)

Define routes corresponding to the services

declare -A routes
routes=(
[“service11″]=”/example11”
[“service12″]=”/example12”
[“service13″]=”/example13”
)

Loop through the services and create them in Kong

for service in “${!services[@]}”; do
# Create each service
echo “Creating service: $service with URL: ${services[$service]}”
curl -i -X POST $KONG_ADMIN_URL/services \
–data name=$service \
–data url=${services[$service]}

# Create a route for each service
echo “Creating route for service: $service with path: ${routes[$service]}”
curl -i -X POST $KONG_ADMIN_URL/routes \
–data paths[]=${routes[$service]} \
–data service.name=$service

# Optionally, add a plugin (e.g., key-auth) to each route

echo “Adding key-auth plugin to route for service: $service”

curl -i -X POST $KONG_ADMIN_URL/routes/${service}/plugins \

–data name=key-auth

done

echo “All services and routes have been configured.

</code>

</code>
  • name: Automate Kong API Mapping for Multiple Services with Different Ports hosts: localhost tasks:

    • name: Define a list of services and routes with different ports set_fact: services: – { name: service6, url: http://service6.com:8086, path: /service6 } – { name: service7, url: http://service7.com:8087, path: /service7 } – { name: service8, url: http://service8.com:8088, path: /service8 } – { name: service9, url: http://service9.com:8089, path: /service9 } – { name: service10, url: http://service10.com:8090, path: /service10 }

    • name: Create a Service in Kong for each service with different ports uri: url: http://localhost:8001/services method: POST body_format: json body: name: “{{ item.name }}” url: “{{ item.url }}” status_code: 201 with_items: “{{ services }}” register: service_creation

    • name: Create a Route for each Service uri: url: http://localhost:8001/routes method: POST body_format: json body: service: name: “{{ item.name }}” paths: – “{{ item.path }}” status_code: 201 with_items: “{{ services }}”

</code>

How to send Kong logs to splunk

techhadoop , , , ,

To send Kong API Gateway logs to Splunk, you can leverage several approaches based on the logging mechanism Kong uses, such as:

  1. HTTP Logging Plugin (sending logs via HTTP to Splunk’s HTTP Event Collector)
  2. Syslog Logging Plugin (sending logs to a syslog server integrated with Splunk)
  3. File-based Logging (sending logs using Splunk Universal Forwarder)

Here’s how you can achieve this:

1. Using the HTTP Logging Plugin (Recommended for Splunk HEC)

You can use Kong’s HTTP Log Plugin to send logs directly to Splunk HTTP Event Collector (HEC) over HTTP(S).

Steps:

1.1. Set Up HTTP Event Collector (HEC) in Splunk

  1. Go to your Splunk Web Interface.
  2. Navigate to Settings > Data Inputs > HTTP Event Collector.
  3. Create a new HEC token.
    • Set a source type (e.g., kong_logs).
    • Note down the token and HEC URL (e.g., http://<SPLUNK_URL&gt;:8088/services/collector).
  4. Ensure that HEC is enabled and configured to accept data.

1.2. Install the HTTP Log Plugin in Kong

The HTTP Log Plugin sends Kong logs to a specified HTTP endpoint (in this case, Splunk HEC).

  • You can configure the plugin at the service, route, or global level.

Example Configuration (using curl):

curl -X POST http://<KONG_ADMIN_URL&gt;:8001/services/<service-id>/plugins \

    –data “name=http-log” \

    –data “config.http_endpoint=http://<SPLUNK_HEC_URL>:8088/services/collector” \

    –data “config.method=POST” \

    –data “config.timeout=10000” \

    –data “config.keepalive=10000” \

    –data “config.headers.Splunk”=”<SPLUNK_HEC_TOKEN>”

  • Replace <KONG_ADMIN_URL> with your Kong Admin URL.
  • Replace <SPLUNK_HEC_URL> with your Splunk HEC endpoint.
  • Replace <SPLUNK_HEC_TOKEN> with the HEC token from Splunk.

You can customize the headers, format, and log levels as per your needs.

1.3. Log Format Configuration (Optional)

You can customize the log format that Kong sends to Splunk by configuring the log_format property of the HTTP Log Plugin.

bash

Copy code

–data “config.log_format={‘message’: ‘Kong Log: $request_uri $status’, ‘client_ip’: ‘$remote_addr’}”

Splunk will now start receiving the logs sent by Kong via the HTTP Event Collector.

2. Using the Syslog Logging Plugin

Kong can send logs to a syslog server that can be monitored by Splunk.

Steps:

2.1. Set Up Syslog Logging Plugin in Kong

  1. Install the Syslog Logging Plugin on your Kong instance:

bash

Copy code

curl -X POST http://<KONG_ADMIN_URL&gt;:8001/services/<service-id>/plugins \

    –data “name=syslog” \

    –data “config.host=<SYSLOG_SERVER_IP>” \

    –data “config.port=514” \

    –data “config.facility=user” \

    –data “config.log_level=info”

  1. Replace <SYSLOG_SERVER_IP> with your syslog server IP or domain.

2.2. Configure Splunk to Receive Syslog Data

  1. On your Splunk instance, configure a new data input for receiving syslog data:
    • Go to Settings > Data Inputs > UDP.
    • Create a new UDP input on port 514 (or another port if you’re using a different one).
    • Set a source type like syslog or a custom type like kong_logs.
  2. You can also use a dedicated syslog server (like rsyslog or syslog-ng) to forward syslog messages from Kong to Splunk.

3. Using the File Log Plugin and Splunk Universal Forwarder

If you’re using file-based logging in Kong, you can set up the File Log Plugin and use the Splunk Universal Forwarder to monitor and send log files to Splunk.

Steps:

3.1. Set Up the File Log Plugin in Kong

  1. Install the File Log Plugin in Kong and configure it to log to a specific file.

Example configuration:

bash

Copy code

curl -X POST http://<KONG_ADMIN_URL&gt;:8001/services/<service-id>/plugins \

    –data “name=file-log” \

    –data “config.path=/var/log/kong/kong.log”

  • Replace /var/log/kong/kong.log with the path where you want the log files stored.

3.2. Install and Configure Splunk Universal Forwarder

  1. Install the Splunk Universal Forwarder on the server where Kong logs are stored.
  2. Configure the forwarder to monitor the log file:

In the inputs.conf file, specify the log file you want to forward:

bash

Copy code

[monitor:///var/log/kong/kong.log]

index = kong

sourcetype = kong:logs

  1. In the outputs.conf file, configure the forwarder to send logs to your main Splunk indexer:

bash

Copy code

[tcpout]

defaultGroup = indexers

[tcpout:indexers]

server = <SPLUNK_INDEXER_IP>:9997

  1. Start the Splunk Universal Forwarder to begin sending logs.

4. Using AWS Lambda (If Kong is in AWS)

If you’re using Kong on AWS (e.g., on EC2), and your logs are stored in AWS CloudWatch, you can set up an AWS Lambda function to forward logs to Splunk HTTP Event Collector (HEC).

Steps:

  1. Set up CloudWatch Logs to capture Kong logs.
  2. Create an AWS Lambda function to forward logs from CloudWatch to Splunk.
  3. Use the Splunk-provided AWS Lambda blueprint to send logs to Splunk HEC.

Summary of Methods:

  • HTTP Log Plugin: Send logs directly to Splunk’s HTTP Event Collector (HEC). This is the easiest and most efficient method.
  • Syslog Logging Plugin: Send logs to a syslog server that can be ingested by Splunk.
  • File Log Plugin + Splunk Universal Forwarder: Write logs to a file and forward them to Splunk using the Splunk Universal Forwarder.
  • AWS Lambda (CloudWatch Logs): Use Lambda to stream logs from CloudWatch to Splunk HEC (for AWS-hosted Kong).

Choose the method based on your infrastructure and logging requirements