security

ansible-playbook

techhadoop , , , , 
---
- name: Delete multiple routes in Kong API Gateway
  hosts: localhost
  gather_facts: false
  tasks:
    - name: Get all routes from Kong
      uri:
        url: "http://localhost:8001/routes"
        method: GET
        return_content: yes
        status_code: 200
      register: kong_routes

    - name: Extract the list of route IDs to delete
      set_fact:
        route_ids: "{{ kong_routes.json.data | map(attribute='id') | list }}"

    - name: Show the route IDs to delete
      debug:
        msg: "Route ID: {{ item }}"
      loop: "{{ route_ids }}"

    - name: Delete each route by its ID
      uri:
        url: "http://localhost:8001/routes/{{ item }}"
        method: DELETE
        status_code: 204
      loop: "{{ route_ids }}"
      when: item is defined
---
- name: Delete multiple services in Kong API Gateway
  hosts: localhost
  gather_facts: false
  tasks:
    - name: Get all services from Kong
      uri:
        url: "http://localhost:8001/services"
        method: GET
        return_content: yes
        status_code: 200
      register: kong_services

    - name: Extract the list of route IDs to delete
      set_fact:
        service_ids: "{{ kong_services.json.data | map(attribute='id') | list }}"

    - name: Show the service IDs to delete
      debug:
        msg: "Service ID: {{ item }}"
      loop: "{{ service_ids }}"

    - name: Delete each service by its ID
      uri:
        url: "http://localhost:8001/services/{{ item }}"
        method: DELETE
        status_code: 204
      loop: "{{ service_ids }}"
      when: item is defined

Kong Service

techhadoop , , , , 
---
- name: Create Services with Service Paths and Routes in Kong API Gateway
  hosts: localhost
  tasks:

    # Define a list of services, their service paths, and routes
    - name: Define a list of services, service paths, and routes
      set_fact:
        services:
          - { name: service1, service_url: http://example-service1.com:8080/service1, route_name: route1, route_path: /service1 }
          - { name: service2, service_url: http://example-service2.com:8080/service2, route_name: route2, route_path: /service2 }
          - { name: service3, service_url: http://example-service3.com:8080/service3, route_name: route3, route_path: /service3 }

    # Create a Service in Kong for each service defined, including service path
    - name: Create Service in Kong
      uri:
        url: http://localhost:8001/services
        method: POST
        body_format: json
        body:
          name: "{{ item.name }}"
          url: "{{ item.service_url }}"  # Service URL (including the service path)
        status_code: 201
      loop: "{{ services }}"
      register: service_creation

# Create a Route for each Service
    - name: Create Route for the Service
      uri:
        url: http://localhost:8001/routes
        method: POST
        body_format: json
        body:
          service:
            name: "{{ item.name }}"
          name: "{{ item.route_name }}"  # Route name
          paths:
            - "{{ item.route_path }}"  # Route path (external access path)
        status_code: 201
      loop: "{{ services }}"
      when: service_creation is succeeded

    # Optionally verify that services and routes were created
    - name: Verify Service and Route creation
      uri:
        url: http://localhost:8001/services/{{ item.name }}
        method: GET
        status_code: 200
      loop: "{{ services }}"

unable to verify the first certificate – KONG

techhadoop , , , ,

unable to verify the first certificate

The error “unable to verify the first certificate” typically indicates that the client (or server) cannot verify the certificate because it does not have the correct root certificate, intermediate certificate, or the certificate chain is incomplete. Here’s how you can troubleshoot and resolve this issue:

Common Causes of the Error:

  1. Missing Root or Intermediate Certificates
    • The server or client lacks the necessary CA certificates to complete the chain of trust.
  2. Self-Signed Certificate
    • If the certificate is self-signed, the server or client must explicitly trust the certificate.
  3. Incomplete Certificate Chain
    • The server might not be sending the entire certificate chain (intermediate certificates) along with the server certificate.
  4. Incorrect Client/Server Configuration
    • The client or server may not be configured to trust the CA that issued the certificate.

Steps to Fix the Issue:

1. Verify the Certificate Chain

Check whether the certificate chain is complete. You can use openssl to check the chain from the client side:

openssl s_client -connect your-kong-api:443 -showcerts

This will show the certificates presented by the server. Verify if the server provides the full chain, including the intermediate certificates.

2. Install Missing CA Certificates on the Client

If the client does not trust the CA that issued the server certificate, you need to install the root CA certificate on the client. For example, on most systems, you can install the CA certificates by adding them to the trusted certificate store.

  • Linux (e.g., Ubuntu/Debian): Copy the CA certificate (e.g., ca.crt) to the /usr/local/share/ca-certificates/ directory and then update the certificate store:

sudo cp ca.crt /usr/local/share/ca-certificates/

sudo update-ca-certificates

  • Windows: Import the root certificate into the Trusted Root Certification Authorities store via the Certificate Manager.
  • macOS: You can import the CA certificate into Keychain Access and mark it as trusted.

3. Verify the Server-Side Configuration

If you’re managing the server (e.g., Kong API Gateway), ensure that the server is sending the complete certificate chain. You can provide both the server certificate and any intermediate certificates in the configuration.

For example, in Kong, you can configure SSL certificates like this:

curl -i -X POST http://localhost:8001/certificates \

–form “cert=@/path/to/server.crt” \

–form “key=@/path/to/server.key” \

–form “cert_alt=@/path/to/intermediate.crt”

Ensure that the server certificate and intermediate certificates are included in the cert field.

4. Test the Connection with the Correct CA

When testing using curl, ensure you include the correct root CA or intermediate CA:

curl -v –cacert /path/to/ca.crt https://your-kong-api/your-route

This will make curl use the specified CA certificate for validation.

5. Check for Self-Signed Certificates

If you’re using self-signed certificates, you’ll need to make sure that both the client and server are explicitly configured to trust the self-signed certificate.

For example, when using curl:

curl -v –key client.key –cert client.crt –cacert ca.crt https://your-kong-api/

If the certificate is self-signed and you want to bypass certificate verification (not recommended in production), you can use:

curl -v –insecure https://your-kong-api/

6. Include the Correct Intermediate Certificates

If the server is not sending the intermediate certificate (or you forgot to add it), make sure that the intermediate certificate is included in the chain. You can concatenate the server certificate and intermediate certificate into one file:

cat server.crt intermediate.crt > full_chain.crt

Then, use the full chain certificate for your server configuration.

Summary of Steps:

  1. Check the certificate chain using openssl s_client to see if all necessary certificates are presented.
  2. Ensure the client trusts the root CA by installing the root or intermediate CA on the client.
  3. Ensure the server presents the full certificate chain (server certificate + intermediate certificates).
  4. Test the connection using the correct CA with curl or another tool.
  5. Handle self-signed certificates by explicitly trusting them or bypassing verification in non-production environments.

By following these steps, you should be able to resolve the “unable to verify the first certificate” issue.

com.pingidentity.pf.datastore an error occured while testing the connection Error PKIX path building failed

techhadoop , , , ,

The error you’re encountering, PKIX path building failed, is related to SSL certificate validation in PingIdentity while attempting to establish a connection. Specifically, it indicates that the system could not verify the certificate path back to a trusted root certificate authority (CA). This is a common issue when a server certificate is either self-signed or not recognized by the trust store.

Here’s a breakdown of the error:

  1. PKIX path building failed: This means that Java’s SSL/TLS system could not build a valid certificate chain back to a trusted root certificate authority.
  2. SunCertPathBuilderException: unable to find valid certification path to requested target: This suggests that the certificate presented by the remote system is not trusted by the client making the request. The client’s trust store (Java keystore) does not have the necessary CA certificates to validate the server certificate.

Causes:

  • Self-signed certificate: If the server you’re trying to connect to uses a self-signed certificate, it won’t be trusted automatically.
  • Untrusted CA: The certificate is signed by a CA that’s not included in the default trust store of the Java Virtual Machine (JVM).
  • Missing intermediate certificates: The certificate chain might be incomplete, missing intermediate certificates between the server’s certificate and the trusted root.
  • Expired or revoked certificates: The server certificate could be expired or revoked, leading to validation failure.

Solutions:

1. Import the Server’s Certificate to the Java Truststore

You need to import the server’s certificate (or the intermediate CA certificates) into the Java trust store to ensure it’s recognized as a trusted certificate.

Steps:

  • Obtain the server certificate:

openssl s_client -connect <server-host>:<port> -showcerts

This will return the server’s SSL certificate chain. Copy the relevant certificate (in PEM format).

  • Save the certificate as server.crt.
  • Import the certificate into the Java trust store:

keytool -import -alias <alias_name> -keystore <path_to_java_home>/lib/security/cacerts -file server.crt

The default password for the trust store is usually changeit, but this can vary.

  • Restart your PingFederate server or application to ensure the new trust store is loaded.

2. Use a Valid SSL Certificate

  • If the server is using a self-signed certificate, consider replacing it with one signed by a trusted public CA (e.g., Let’s Encrypt, GlobalSign, etc.).
  • Ensure the entire certificate chain, including intermediate certificates, is properly configured on the server.

3. Disable SSL Validation (Not Recommended in Production)

You can temporarily disable SSL validation to allow connections without certificate verification. This is usually done in testing environments or when working with self-signed certificates.

Example: In Java, you can disable SSL certificate validation by setting a custom trust manager, but this approach is not secure and should be avoided in production:

import javax.net.ssl.*;

import java.security.SecureRandom;

import java.security.cert.X509Certificate;

TrustManager[] trustAllCerts = new TrustManager[]{

    new X509TrustManager() {

        public X509Certificate[] getAcceptedIssuers() {

            return null;

        }

        public void checkClientTrusted(X509Certificate[] certs, String authType) {}

        public void checkServerTrusted(X509Certificate[] certs, String authType) {}

    }

};

SSLContext sc = SSLContext.getInstance(“SSL”);

sc.init(null, trustAllCerts, new SecureRandom());

HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());

This solution is not recommended for production environments.

4. Verify the Server Certificate

Ensure that the certificate on the server is not expired, properly configured, and includes all intermediate certificates. You can use tools like:

openssl s_client -connect <server-host>:<port> -showcerts

5. Update the JVM Truststore (if outdated)

If you are using an old version of Java, the default trust store may not include modern root CAs. You can update your JVM or manually update the CA certificates:

  • Download a fresh version of the cacerts file from the latest JVM or a trusted source.
  • Replace your current cacerts file (located in JAVA_HOME/lib/security) with the new one.

6. Proxy Configuration

If you’re using a proxy, make sure the proxy server has the necessary CA certificates and that PingFederate is properly configured to connect through the proxy.

Conclusion

To resolve the PKIX path building failed error, you will likely need to add the server certificate (or its CA) to your Java trust store. Ensure the server’s certificate chain is correctly configured and, if using self-signed certificates, import them into the trust store. Avoid disabling SSL validation in production environments due to security risks.

PING Identity

techhadoop , , , ,

The error you’re encountering is related to certificate validation and occurs when the system is unable to establish a valid certificate chain to the requested target (usually an external service or API). Specifically, the error message:

SunCertPathBuilderException: unable to find valid certification path to requested target

indicates that the Java application (in this case, Ping Identity) is trying to connect to an HTTPS service, but the service’s SSL/TLS certificate is either:

  • Self-signed, or
  • Issued by a Certificate Authority (CA) that is not trusted by the Java trust store.

Here’s how you can troubleshoot and resolve this issue:

Step 1: Verify the SSL/TLS Certificate

  1. Check the service’s certificate: Use a browser or a tool like openssl to verify the service’s certificate chain:

openssl s_client -connect <hostname>:<port> -showcerts

This will display the certificate chain used by the server. Ensure that the server certificate is valid and that intermediate and root certificates are also included.

  1. Check if the certificate is self-signed: If the service is using a self-signed certificate or a certificate from a CA not included in the default Java trust store, you’ll need to manually trust it.

Step 2: Add the Certificate to the Java Trust Store

You’ll need to import the certificate into the Java trust store so that Java can trust it.

  1. Export the server certificate:
    • Save the certificate to a file using your browser or the openssl command.

For example:

echo | openssl s_client -connect <hostname>:<port> | sed -ne ‘/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p’ > server-cert.pem

  1. Import the certificate into Java’s trust store: Use the keytool command to import the certificate:

sudo keytool -import -alias <alias-name> -file <path-to-cert> -keystore $JAVA_HOME/lib/security/cacerts

Replace:

  1. <alias-name>: A unique alias for the certificate.
  2. <path-to-cert>: The path to the certificate file you saved.

The default password for the Java trust store is usually changeit.

  1. Verify the certificate import: You can verify if the certificate has been successfully imported by listing the contents of the trust store:

sudo keytool -list -keystore $JAVA_HOME/lib/security/cacerts

Step 3: Test the Connection Again

After importing the certificate, restart your application and test the connection again to ensure the error is resolved.

Additional Considerations:

  • Custom Trust Store: If your application is using a custom trust store (not the default Java trust store), ensure that the certificate is added to that trust store instead.
  • CA Certificates: If the certificate is from a trusted CA, ensure that your system has the correct root CA certificates in its trust store.

By importing the certificate into the Java trust store, you should be able to resolve the PKIX path building failed error and establish a successful connection.

How to verify mtls client cert via curl in Kong API gateway

techhadoop , , , ,

To verify mTLS (Mutual TLS) client certificates in Kong API Gateway using curl, you will need to have:

  1. A valid client certificate and key that the server can verify.
  2. The server’s CA certificate to trust the server’s certificate.

Here’s a step-by-step guide to verify the mTLS setup using curl:

1. Ensure mTLS is enabled in Kong

Make sure that the mTLS authentication plugin is enabled on the route, service, or globally in your Kong instance. For example, you can check if the mtls-auth plugin is applied:

curl http://localhost:8001/plugins

If it’s not enabled, you can add the mTLS plugin to a specific route or service:

curl -i -X POST http://localhost:8001/services/{service-id}/plugins \

    –data “name=mtls-auth”

2. Ensure the Client Certificate is Associated with a Consumer

To match the client certificate with a consumer in Kong, you need to associate the Common Name (CN) or Subject Alternative Name (SAN) in the certificate with a consumer.

curl -i -X POST http://localhost:8001/consumers/{consumer-id}/mtls_auth \

    –data “subject_name=client.example.com”

This ensures that when the client presents the certificate, Kong will map the request to the correct consumer based on the CN.

3. Verify mTLS using curl

To perform mTLS using curl, you will need:

  • The client certificate and client key.
  • The CA certificate that Kong uses to verify the client certificate.

Use the following curl command to verify mTLS:

bash

Copy code

curl -v https://{KONG_HOST}:{KONG_PORT}/{route-path} \

    –cert /path/to/client-cert.pem \

    –key /path/to/client-key.pem \

    –cacert /path/to/ca-cert.pem

  • –cert /path/to/client-cert.pem: Path to the client certificate file.
  • –key /path/to/client-key.pem: Path to the client private key file.
  • –cacert /path/to/ca-cert.pem: Path to the CA certificate that Kong uses to verify the client certificate.
  • -v: Verbose output to see the handshake details.

4. Example Command

For example, if Kong is running on localhost, the route is /secure-service, and you have the client certificate and key:

curl -v https://localhost:8443/secure-service \

    –cert /etc/ssl/client-cert.pem \

    –key /etc/ssl/client-key.pem \

    –cacert /etc/ssl/ca-cert.pem

5. Check the Response

  • If the client certificate is valid and trusted, Kong will allow the request, and you will see a 200 OK response (or the relevant service response).
  • If the certificate validation fails, you may see errors like:
    • 400 Bad Request: Indicates issues with the certificate verification.
    • 403 Forbidden: Indicates the certificate was valid but the client was not authorized for the route.

6. Common Errors and Troubleshooting

  • 400 Bad Request: SSL certificate validation failed: This usually happens when the client certificate is not signed by the trusted CA or doesn’t match the subject name expected by Kong.
  • 403 Forbidden: mTLS authentication failed: This happens if the certificate’s CN or SAN does not match any configured consumers in Kong.

Conclusion

Verifying mTLS in Kong API Gateway via curl involves ensuring that the client certificate and key are properly configured and Kong is set up to validate them. If configured correctly, Kong will authenticate the client using the certificate, and the request will proceed.

Hadoop ha Active/Active vs Active/Passive

techhadoop , , , ,

Hadoop High Availability (HA): Active/Active vs. Active/Passive

When designing a Hadoop High Availability (HA) solution, two common approaches are Active/Active and Active/Passive. These strategies help ensure data and service availability across failures and disasters. Let’s compare them in detail to help you understand their differences, benefits, challenges, and use cases.

1. Active/Active Hadoop Architecture

Overview:

  • Both sites are fully operational and handling workloads simultaneously.
  • Both clusters actively serve requests, and the load can be distributed between them.
  • Data is replicated between the sites, ensuring both sites are synchronized.

Key Components:

  • HDFS Federation: Each site has its own NameNode that manages a portion of the HDFS namespace.
  • YARN ResourceManager: Each site runs its own ResourceManager, coordinating job execution locally, but the jobs can be balanced between sites.
  • Zookeeper & JournalNodes Quorum: Spread across both sites to provide consistency and manage service coordination.
  • Cross-Site Replication: Hadoop’s DistCp or HDFS replication is used to replicate data across sites.
  • Hive/Impala Metastore: Shared between sites, ensuring consistent metadata.

Advantages:

  1. Load Balancing: Traffic and workloads can be distributed between the two active sites, reducing pressure on a single site.
  2. Low Recovery Time: In case of a site failure, the other site can immediately handle all workloads without downtime.
  3. Improved Resource Utilization: Both sites are fully operational, utilizing available resources efficiently.
  4. Fast Failover: If one site fails, the remaining site continues operating without needing to bring up services.

Challenges:

  1. Increased Complexity: Managing two active sites involves more complex setup, including federation, data replication, and synchronization.
  2. Data Consistency: Ensuring both sites have up-to-date data requires robust replication mechanisms and careful coordination.
  3. Conflict Resolution: Handling conflicting updates across both sites requires careful planning and automated conflict resolution strategies.

Operational Considerations:

  • Synchronization of Data: Ensure real-time or near real-time data replication across both sites.
  • Federated HDFS: Requires splitting data across multiple namespaces with NameNodes in each site.
  • Network Requirements: Reliable, high-bandwidth network links are essential for cross-site replication and synchronization.
  • Monitoring and Automation: Continuous monitoring of job failures, resource usage, and automatic load balancing/failover processes.

Best Use Cases:

  • Mission-Critical Workloads: Where zero downtime and continuous availability are essential.
  • Geographically Distributed Sites: When there is a need for global load balancing or when sites are geographically distant but still need to function as one.
  • High Load Systems: Systems that need to distribute workloads across multiple data centers to balance processing power.

2. Active/Passive Hadoop Architecture

Overview:

  • The Primary (Active) site handles all the workloads, while the Secondary (Passive) site is on standby.
  • In case of failure or disaster, the passive site takes over and becomes the active one.
  • The secondary site is synchronized with the active site, but it does not actively serve any workloads until failover occurs.

Key Components:

  • Active and Standby NameNodes: The active site runs the main NameNode, while the passive site hosts a standby NameNode.
  • YARN ResourceManager: Active ResourceManager at the primary site, standby ResourceManager at the secondary site.
  • Zookeeper & JournalNode Quorum: Distributed across both sites for fault tolerance and coordination.
  • HDFS Replication: Ensures data is replicated across both sites using HDFS data blocks.
  • Hive/Impala Metastore: Either synchronized or replicated between the two sites for metadata consistency.

Advantages:

  1. Simpler Setup: Easier to configure and manage compared to Active/Active architecture.
  2. Cost-Efficient: Since the passive site is not active until failover, fewer resources are consumed.
  3. Data Integrity: With a single active site at a time, data conflicts and consistency issues are less likely.
  4. Disaster Recovery: Ensures quick recovery of services in the event of failure or disaster in the primary site.

Challenges:

  1. Failover Time: There can be a delay in switching over from the active site to the passive site.
  2. Underutilized Resources: The passive site is mostly idle, which can lead to inefficient resource use.
  3. Single Point of Failure: Until failover occurs, there is a reliance on the primary site, creating a risk of downtime.
  4. Data Replication: You need to ensure that the passive site has the latest data in case of a failover.

Operational Considerations:

  • Automated Failover: Implement automated failover mechanisms using Zookeeper and JournalNodes to reduce downtime.
  • Data Synchronization: Ensure regular and real-time synchronization between the two sites to avoid data loss.
  • Disaster Recovery Testing: Regularly test the failover process to ensure that the passive site can take over with minimal downtime.
  • Backup and Monitoring: Maintain backups and monitor the status of both sites to detect any potential failures early.

Best Use Cases:

  • Cost-Conscious Environments: When you need a disaster recovery solution but don’t want the expense of running both sites at full capacity.
  • Disaster Recovery Scenarios: When one site is meant purely for recovery in case of major failure or disaster at the primary site.
  • Low-Volume Operations: When your workloads don’t justify the complexity and overhead of an active/active setup.

Setup services and route in Kong API Gateway

techhadoop , , , ,
  1. Shell script

<code>

#!/bin/bash

#Set Kong Admin API URL

KONG_ADMIN_URL=”http://localhost:8001&#8243;

#Define an array of services and routes

declare -A services
services=(
[“service11″]=”http://example11.com:8080&#8221;
[“service12″]=”http://example12.com:8080&#8221;
[“service13″]=”http://example13.com:8080&#8221;
)

Define routes corresponding to the services

declare -A routes
routes=(
[“service11″]=”/example11”
[“service12″]=”/example12”
[“service13″]=”/example13”
)

Loop through the services and create them in Kong

for service in “${!services[@]}”; do
# Create each service
echo “Creating service: $service with URL: ${services[$service]}”
curl -i -X POST $KONG_ADMIN_URL/services \
–data name=$service \
–data url=${services[$service]}

# Create a route for each service
echo “Creating route for service: $service with path: ${routes[$service]}”
curl -i -X POST $KONG_ADMIN_URL/routes \
–data paths[]=${routes[$service]} \
–data service.name=$service

# Optionally, add a plugin (e.g., key-auth) to each route

echo “Adding key-auth plugin to route for service: $service”

curl -i -X POST $KONG_ADMIN_URL/routes/${service}/plugins \

–data name=key-auth

done

echo “All services and routes have been configured.

</code>

</code>
  • name: Automate Kong API Mapping for Multiple Services with Different Ports hosts: localhost tasks:

    • name: Define a list of services and routes with different ports set_fact: services: – { name: service6, url: http://service6.com:8086, path: /service6 } – { name: service7, url: http://service7.com:8087, path: /service7 } – { name: service8, url: http://service8.com:8088, path: /service8 } – { name: service9, url: http://service9.com:8089, path: /service9 } – { name: service10, url: http://service10.com:8090, path: /service10 }

    • name: Create a Service in Kong for each service with different ports uri: url: http://localhost:8001/services method: POST body_format: json body: name: “{{ item.name }}” url: “{{ item.url }}” status_code: 201 with_items: “{{ services }}” register: service_creation

    • name: Create a Route for each Service uri: url: http://localhost:8001/routes method: POST body_format: json body: service: name: “{{ item.name }}” paths: – “{{ item.path }}” status_code: 201 with_items: “{{ services }}”

</code>

How to integrate mTLS Kong certificate with secrets management infrastructure

techhadoop , , , ,

To integrate mTLS (Mutual TLS) certificates used in Kong API Gateway with a secrets management infrastructure (such as HashiCorp Vault, AWS Secrets Manager, or other secret management tools), you can follow a systematic approach to store, retrieve, and rotate the certificates securely.

Key Steps:

  1. Generate mTLS certificates.
  2. Store certificates securely in the secrets management infrastructure.
  3. Configure Kong to retrieve and use certificates for mTLS.
  4. Automate certificate rotation for secure management.

Step 1: Generate mTLS Certificates

You need to generate the client and server certificates that Kong will use for mTLS.

  1. Generate a Certificate Authority (CA): First, generate a CA to sign the certificates.

openssl genrsa -out ca.key 2048

openssl req -x509 -new -nodes -key ca.key -sha256 -days 365 -out ca.crt \

-subj “/CN=Kong-CA”

  1. Generate the Server Certificate: Generate a private key and a certificate signing request (CSR) for Kong, and sign it with your CA.

openssl genrsa -out kong-server.key 2048

openssl req -new -key kong-server.key -out kong-server.csr -subj “/CN=kong-server”

openssl x509 -req -in kong-server.csr -CA ca.crt -CAkey ca.key -CAcreateserial \

-out kong-server.crt -days 365 -sha256

  1. Generate the Client Certificate: You also need a client certificate to authenticate incoming requests.

openssl genrsa -out kong-client.key 2048

openssl req -new -key kong-client.key -out kong-client.csr -subj “/CN=kong-client”

openssl x509 -req -in kong-client.csr -CA ca.crt -CAkey ca.key -CAcreateserial \

-out kong-client.crt -days 365 -sha256

Step 2: Store Certificates Securely in a Secrets Management Infrastructure

Use a secrets management service like HashiCorp Vault, AWS Secrets Manager, or another system to store the mTLS certificates securely.

Example: Store Certificates in HashiCorp Vault

  1. Start by enabling the secrets engine in Vault:

vault secrets enable -path=pki pki

vault secrets tune -max-lease-ttl=87600h pki

  1. Store the CA certificate in Vault:

vault write pki/config/ca pem_bundle=@ca.crt

  1. Store the server certificate and key in Vault:

vault kv put secret/kong/server cert=@kong-server.crt key=@kong-server.key

  1. Store the client certificate and key:

vault kv put secret/kong/client cert=@kong-client.crt key=@kong-client.key

Example: Store Certificates in AWS Secrets Manager

  1. Use AWS CLI to store the server certificate:

aws secretsmanager create-secret –name kong/server \

–secret-string file://kong-server.json

The kong-server.json file contains:

{

  “certificate”: “—–BEGIN CERTIFICATE—– …”,

  “private_key”: “—–BEGIN PRIVATE KEY—– …”

}

  1. Store the client certificate similarly in AWS Secrets Manager:

aws secretsmanager create-secret –name kong/client \

–secret-string file://kong-client.json

Step 3: Configure Kong to Retrieve and Use Certificates for mTLS

Once the certificates are securely stored, you need to configure Kong to retrieve and use these certificates from your secrets management infrastructure.

Option 1: Use HashiCorp Vault with Kong

  1. Install the Vault Plugin for Kong (or use an external script to retrieve certificates from Vault).
  2. Write a Lua script or custom plugin to dynamically retrieve the certificates from Vault.
    • Use the vault kv get API to retrieve certificates from Vault.
    • Load the certificates into Kong’s SSL configuration dynamically.
  3. Configure Kong to use the certificates:
    • Add the mTLS plugin to your service or route to enable mutual authentication using the retrieved certificates.

Example to configure the mTLS plugin:

curl -i -X POST http://localhost:8001/services/my-service/plugins \

–data “name=mtls-auth” \

–data “config.trusted_ca_ids=ca_id” \

–data “config.client_verify=true”

Option 2: Use AWS Secrets Manager with Kong

  1. Install the AWS SDK (or a script) on your Kong instances to fetch the certificates from Secrets Manager.
  2. Create a script or custom plugin to:
    • Retrieve the server and client certificates from AWS Secrets Manager using the AWS CLI or SDK:

aws secretsmanager get-secret-value –secret-id kong/server

aws secretsmanager get-secret-value –secret-id kong/client

  1. Dynamically load the certificates into Kong’s SSL configuration using Lua or custom logic.
  2. Configure mTLS in Kong:
    • Set up the mTLS plugin in Kong to verify client certificates:

curl -i -X POST http://localhost:8001/services/my-service/plugins \

–data “name=mtls-auth” \

–data “config.trusted_ca_ids=<ca-id>”

Step 4: Automate Certificate Rotation

To ensure secure and automated certificate management, integrate certificate rotation.

  1. Automated Certificate Renewal in Vault:
    • Configure Vault’s PKI secrets engine to automate certificate issuance and rotation.

Example:

vault write pki/roles/kong \

allowed_domains=example.com \

allow_subdomains=true \

max_ttl=72h

Use Vault’s pki/issue endpoint to automatically rotate certificates and replace them in Kong.

  1. Automate AWS Secrets Manager Rotation:
    • Set up AWS Secrets Manager’s built-in rotation feature for SSL certificates.
  2. Trigger Updates in Kong:
    • Use a periodic task (e.g., a cron job or Ansible playbook) to update the certificates in Kong without restarting the gateway:

kong reload

This ensures Kong always uses the latest mTLS certificates from the secrets manager.

Conclusion

To integrate Kong API Gateway’s mTLS certificates with a secrets management infrastructure, follow these steps:

  1. Generate the mTLS certificates and store them securely in a secrets management tool (e.g., HashiCorp Vault, AWS Secrets Manager).
  2. Configure Kong to retrieve the certificates dynamically from the secret manager.
  3. Implement automation for certificate renewal and rotation to ensure that Kong always uses up-to-date certificates without manual intervention.

This approach enhances security by managing sensitive SSL certificates in a centralized and automated manner.

How to send Kong logs to splunk

techhadoop , , , ,

To send Kong API Gateway logs to Splunk, you can leverage several approaches based on the logging mechanism Kong uses, such as:

  1. HTTP Logging Plugin (sending logs via HTTP to Splunk’s HTTP Event Collector)
  2. Syslog Logging Plugin (sending logs to a syslog server integrated with Splunk)
  3. File-based Logging (sending logs using Splunk Universal Forwarder)

Here’s how you can achieve this:

1. Using the HTTP Logging Plugin (Recommended for Splunk HEC)

You can use Kong’s HTTP Log Plugin to send logs directly to Splunk HTTP Event Collector (HEC) over HTTP(S).

Steps:

1.1. Set Up HTTP Event Collector (HEC) in Splunk

  1. Go to your Splunk Web Interface.
  2. Navigate to Settings > Data Inputs > HTTP Event Collector.
  3. Create a new HEC token.
    • Set a source type (e.g., kong_logs).
    • Note down the token and HEC URL (e.g., http://<SPLUNK_URL&gt;:8088/services/collector).
  4. Ensure that HEC is enabled and configured to accept data.

1.2. Install the HTTP Log Plugin in Kong

The HTTP Log Plugin sends Kong logs to a specified HTTP endpoint (in this case, Splunk HEC).

  • You can configure the plugin at the service, route, or global level.

Example Configuration (using curl):

curl -X POST http://<KONG_ADMIN_URL&gt;:8001/services/<service-id>/plugins \

    –data “name=http-log” \

    –data “config.http_endpoint=http://<SPLUNK_HEC_URL>:8088/services/collector” \

    –data “config.method=POST” \

    –data “config.timeout=10000” \

    –data “config.keepalive=10000” \

    –data “config.headers.Splunk”=”<SPLUNK_HEC_TOKEN>”

  • Replace <KONG_ADMIN_URL> with your Kong Admin URL.
  • Replace <SPLUNK_HEC_URL> with your Splunk HEC endpoint.
  • Replace <SPLUNK_HEC_TOKEN> with the HEC token from Splunk.

You can customize the headers, format, and log levels as per your needs.

1.3. Log Format Configuration (Optional)

You can customize the log format that Kong sends to Splunk by configuring the log_format property of the HTTP Log Plugin.

bash

Copy code

–data “config.log_format={‘message’: ‘Kong Log: $request_uri $status’, ‘client_ip’: ‘$remote_addr’}”

Splunk will now start receiving the logs sent by Kong via the HTTP Event Collector.

2. Using the Syslog Logging Plugin

Kong can send logs to a syslog server that can be monitored by Splunk.

Steps:

2.1. Set Up Syslog Logging Plugin in Kong

  1. Install the Syslog Logging Plugin on your Kong instance:

bash

Copy code

curl -X POST http://<KONG_ADMIN_URL&gt;:8001/services/<service-id>/plugins \

    –data “name=syslog” \

    –data “config.host=<SYSLOG_SERVER_IP>” \

    –data “config.port=514” \

    –data “config.facility=user” \

    –data “config.log_level=info”

  1. Replace <SYSLOG_SERVER_IP> with your syslog server IP or domain.

2.2. Configure Splunk to Receive Syslog Data

  1. On your Splunk instance, configure a new data input for receiving syslog data:
    • Go to Settings > Data Inputs > UDP.
    • Create a new UDP input on port 514 (or another port if you’re using a different one).
    • Set a source type like syslog or a custom type like kong_logs.
  2. You can also use a dedicated syslog server (like rsyslog or syslog-ng) to forward syslog messages from Kong to Splunk.

3. Using the File Log Plugin and Splunk Universal Forwarder

If you’re using file-based logging in Kong, you can set up the File Log Plugin and use the Splunk Universal Forwarder to monitor and send log files to Splunk.

Steps:

3.1. Set Up the File Log Plugin in Kong

  1. Install the File Log Plugin in Kong and configure it to log to a specific file.

Example configuration:

bash

Copy code

curl -X POST http://<KONG_ADMIN_URL&gt;:8001/services/<service-id>/plugins \

    –data “name=file-log” \

    –data “config.path=/var/log/kong/kong.log”

  • Replace /var/log/kong/kong.log with the path where you want the log files stored.

3.2. Install and Configure Splunk Universal Forwarder

  1. Install the Splunk Universal Forwarder on the server where Kong logs are stored.
  2. Configure the forwarder to monitor the log file:

In the inputs.conf file, specify the log file you want to forward:

bash

Copy code

[monitor:///var/log/kong/kong.log]

index = kong

sourcetype = kong:logs

  1. In the outputs.conf file, configure the forwarder to send logs to your main Splunk indexer:

bash

Copy code

[tcpout]

defaultGroup = indexers

[tcpout:indexers]

server = <SPLUNK_INDEXER_IP>:9997

  1. Start the Splunk Universal Forwarder to begin sending logs.

4. Using AWS Lambda (If Kong is in AWS)

If you’re using Kong on AWS (e.g., on EC2), and your logs are stored in AWS CloudWatch, you can set up an AWS Lambda function to forward logs to Splunk HTTP Event Collector (HEC).

Steps:

  1. Set up CloudWatch Logs to capture Kong logs.
  2. Create an AWS Lambda function to forward logs from CloudWatch to Splunk.
  3. Use the Splunk-provided AWS Lambda blueprint to send logs to Splunk HEC.

Summary of Methods:

  • HTTP Log Plugin: Send logs directly to Splunk’s HTTP Event Collector (HEC). This is the easiest and most efficient method.
  • Syslog Logging Plugin: Send logs to a syslog server that can be ingested by Splunk.
  • File Log Plugin + Splunk Universal Forwarder: Write logs to a file and forward them to Splunk using the Splunk Universal Forwarder.
  • AWS Lambda (CloudWatch Logs): Use Lambda to stream logs from CloudWatch to Splunk HEC (for AWS-hosted Kong).

Choose the method based on your infrastructure and logging requirements