azure

Hadoop ha Active/Active vs Active/Passive

techhadoop , , , ,

Hadoop High Availability (HA): Active/Active vs. Active/Passive

When designing a Hadoop High Availability (HA) solution, two common approaches are Active/Active and Active/Passive. These strategies help ensure data and service availability across failures and disasters. Let’s compare them in detail to help you understand their differences, benefits, challenges, and use cases.

1. Active/Active Hadoop Architecture

Overview:

  • Both sites are fully operational and handling workloads simultaneously.
  • Both clusters actively serve requests, and the load can be distributed between them.
  • Data is replicated between the sites, ensuring both sites are synchronized.

Key Components:

  • HDFS Federation: Each site has its own NameNode that manages a portion of the HDFS namespace.
  • YARN ResourceManager: Each site runs its own ResourceManager, coordinating job execution locally, but the jobs can be balanced between sites.
  • Zookeeper & JournalNodes Quorum: Spread across both sites to provide consistency and manage service coordination.
  • Cross-Site Replication: Hadoop’s DistCp or HDFS replication is used to replicate data across sites.
  • Hive/Impala Metastore: Shared between sites, ensuring consistent metadata.

Advantages:

  1. Load Balancing: Traffic and workloads can be distributed between the two active sites, reducing pressure on a single site.
  2. Low Recovery Time: In case of a site failure, the other site can immediately handle all workloads without downtime.
  3. Improved Resource Utilization: Both sites are fully operational, utilizing available resources efficiently.
  4. Fast Failover: If one site fails, the remaining site continues operating without needing to bring up services.

Challenges:

  1. Increased Complexity: Managing two active sites involves more complex setup, including federation, data replication, and synchronization.
  2. Data Consistency: Ensuring both sites have up-to-date data requires robust replication mechanisms and careful coordination.
  3. Conflict Resolution: Handling conflicting updates across both sites requires careful planning and automated conflict resolution strategies.

Operational Considerations:

  • Synchronization of Data: Ensure real-time or near real-time data replication across both sites.
  • Federated HDFS: Requires splitting data across multiple namespaces with NameNodes in each site.
  • Network Requirements: Reliable, high-bandwidth network links are essential for cross-site replication and synchronization.
  • Monitoring and Automation: Continuous monitoring of job failures, resource usage, and automatic load balancing/failover processes.

Best Use Cases:

  • Mission-Critical Workloads: Where zero downtime and continuous availability are essential.
  • Geographically Distributed Sites: When there is a need for global load balancing or when sites are geographically distant but still need to function as one.
  • High Load Systems: Systems that need to distribute workloads across multiple data centers to balance processing power.

2. Active/Passive Hadoop Architecture

Overview:

  • The Primary (Active) site handles all the workloads, while the Secondary (Passive) site is on standby.
  • In case of failure or disaster, the passive site takes over and becomes the active one.
  • The secondary site is synchronized with the active site, but it does not actively serve any workloads until failover occurs.

Key Components:

  • Active and Standby NameNodes: The active site runs the main NameNode, while the passive site hosts a standby NameNode.
  • YARN ResourceManager: Active ResourceManager at the primary site, standby ResourceManager at the secondary site.
  • Zookeeper & JournalNode Quorum: Distributed across both sites for fault tolerance and coordination.
  • HDFS Replication: Ensures data is replicated across both sites using HDFS data blocks.
  • Hive/Impala Metastore: Either synchronized or replicated between the two sites for metadata consistency.

Advantages:

  1. Simpler Setup: Easier to configure and manage compared to Active/Active architecture.
  2. Cost-Efficient: Since the passive site is not active until failover, fewer resources are consumed.
  3. Data Integrity: With a single active site at a time, data conflicts and consistency issues are less likely.
  4. Disaster Recovery: Ensures quick recovery of services in the event of failure or disaster in the primary site.

Challenges:

  1. Failover Time: There can be a delay in switching over from the active site to the passive site.
  2. Underutilized Resources: The passive site is mostly idle, which can lead to inefficient resource use.
  3. Single Point of Failure: Until failover occurs, there is a reliance on the primary site, creating a risk of downtime.
  4. Data Replication: You need to ensure that the passive site has the latest data in case of a failover.

Operational Considerations:

  • Automated Failover: Implement automated failover mechanisms using Zookeeper and JournalNodes to reduce downtime.
  • Data Synchronization: Ensure regular and real-time synchronization between the two sites to avoid data loss.
  • Disaster Recovery Testing: Regularly test the failover process to ensure that the passive site can take over with minimal downtime.
  • Backup and Monitoring: Maintain backups and monitor the status of both sites to detect any potential failures early.

Best Use Cases:

  • Cost-Conscious Environments: When you need a disaster recovery solution but don’t want the expense of running both sites at full capacity.
  • Disaster Recovery Scenarios: When one site is meant purely for recovery in case of major failure or disaster at the primary site.
  • Low-Volume Operations: When your workloads don’t justify the complexity and overhead of an active/active setup.

Setup services and route in Kong API Gateway

techhadoop , , , ,
  1. Shell script

<code>

#!/bin/bash

#Set Kong Admin API URL

KONG_ADMIN_URL=”http://localhost:8001&#8243;

#Define an array of services and routes

declare -A services
services=(
[“service11″]=”http://example11.com:8080&#8221;
[“service12″]=”http://example12.com:8080&#8221;
[“service13″]=”http://example13.com:8080&#8221;
)

Define routes corresponding to the services

declare -A routes
routes=(
[“service11″]=”/example11”
[“service12″]=”/example12”
[“service13″]=”/example13”
)

Loop through the services and create them in Kong

for service in “${!services[@]}”; do
# Create each service
echo “Creating service: $service with URL: ${services[$service]}”
curl -i -X POST $KONG_ADMIN_URL/services \
–data name=$service \
–data url=${services[$service]}

# Create a route for each service
echo “Creating route for service: $service with path: ${routes[$service]}”
curl -i -X POST $KONG_ADMIN_URL/routes \
–data paths[]=${routes[$service]} \
–data service.name=$service

# Optionally, add a plugin (e.g., key-auth) to each route

echo “Adding key-auth plugin to route for service: $service”

curl -i -X POST $KONG_ADMIN_URL/routes/${service}/plugins \

–data name=key-auth

done

echo “All services and routes have been configured.

</code>

</code>
  • name: Automate Kong API Mapping for Multiple Services with Different Ports hosts: localhost tasks:

    • name: Define a list of services and routes with different ports set_fact: services: – { name: service6, url: http://service6.com:8086, path: /service6 } – { name: service7, url: http://service7.com:8087, path: /service7 } – { name: service8, url: http://service8.com:8088, path: /service8 } – { name: service9, url: http://service9.com:8089, path: /service9 } – { name: service10, url: http://service10.com:8090, path: /service10 }

    • name: Create a Service in Kong for each service with different ports uri: url: http://localhost:8001/services method: POST body_format: json body: name: “{{ item.name }}” url: “{{ item.url }}” status_code: 201 with_items: “{{ services }}” register: service_creation

    • name: Create a Route for each Service uri: url: http://localhost:8001/routes method: POST body_format: json body: service: name: “{{ item.name }}” paths: – “{{ item.path }}” status_code: 201 with_items: “{{ services }}”

</code>

How to integrate mTLS Kong certificate with secrets management infrastructure

techhadoop , , , ,

To integrate mTLS (Mutual TLS) certificates used in Kong API Gateway with a secrets management infrastructure (such as HashiCorp Vault, AWS Secrets Manager, or other secret management tools), you can follow a systematic approach to store, retrieve, and rotate the certificates securely.

Key Steps:

  1. Generate mTLS certificates.
  2. Store certificates securely in the secrets management infrastructure.
  3. Configure Kong to retrieve and use certificates for mTLS.
  4. Automate certificate rotation for secure management.

Step 1: Generate mTLS Certificates

You need to generate the client and server certificates that Kong will use for mTLS.

  1. Generate a Certificate Authority (CA): First, generate a CA to sign the certificates.

openssl genrsa -out ca.key 2048

openssl req -x509 -new -nodes -key ca.key -sha256 -days 365 -out ca.crt \

-subj “/CN=Kong-CA”

  1. Generate the Server Certificate: Generate a private key and a certificate signing request (CSR) for Kong, and sign it with your CA.

openssl genrsa -out kong-server.key 2048

openssl req -new -key kong-server.key -out kong-server.csr -subj “/CN=kong-server”

openssl x509 -req -in kong-server.csr -CA ca.crt -CAkey ca.key -CAcreateserial \

-out kong-server.crt -days 365 -sha256

  1. Generate the Client Certificate: You also need a client certificate to authenticate incoming requests.

openssl genrsa -out kong-client.key 2048

openssl req -new -key kong-client.key -out kong-client.csr -subj “/CN=kong-client”

openssl x509 -req -in kong-client.csr -CA ca.crt -CAkey ca.key -CAcreateserial \

-out kong-client.crt -days 365 -sha256

Step 2: Store Certificates Securely in a Secrets Management Infrastructure

Use a secrets management service like HashiCorp Vault, AWS Secrets Manager, or another system to store the mTLS certificates securely.

Example: Store Certificates in HashiCorp Vault

  1. Start by enabling the secrets engine in Vault:

vault secrets enable -path=pki pki

vault secrets tune -max-lease-ttl=87600h pki

  1. Store the CA certificate in Vault:

vault write pki/config/ca pem_bundle=@ca.crt

  1. Store the server certificate and key in Vault:

vault kv put secret/kong/server cert=@kong-server.crt key=@kong-server.key

  1. Store the client certificate and key:

vault kv put secret/kong/client cert=@kong-client.crt key=@kong-client.key

Example: Store Certificates in AWS Secrets Manager

  1. Use AWS CLI to store the server certificate:

aws secretsmanager create-secret –name kong/server \

–secret-string file://kong-server.json

The kong-server.json file contains:

{

  “certificate”: “—–BEGIN CERTIFICATE—– …”,

  “private_key”: “—–BEGIN PRIVATE KEY—– …”

}

  1. Store the client certificate similarly in AWS Secrets Manager:

aws secretsmanager create-secret –name kong/client \

–secret-string file://kong-client.json

Step 3: Configure Kong to Retrieve and Use Certificates for mTLS

Once the certificates are securely stored, you need to configure Kong to retrieve and use these certificates from your secrets management infrastructure.

Option 1: Use HashiCorp Vault with Kong

  1. Install the Vault Plugin for Kong (or use an external script to retrieve certificates from Vault).
  2. Write a Lua script or custom plugin to dynamically retrieve the certificates from Vault.
    • Use the vault kv get API to retrieve certificates from Vault.
    • Load the certificates into Kong’s SSL configuration dynamically.
  3. Configure Kong to use the certificates:
    • Add the mTLS plugin to your service or route to enable mutual authentication using the retrieved certificates.

Example to configure the mTLS plugin:

curl -i -X POST http://localhost:8001/services/my-service/plugins \

–data “name=mtls-auth” \

–data “config.trusted_ca_ids=ca_id” \

–data “config.client_verify=true”

Option 2: Use AWS Secrets Manager with Kong

  1. Install the AWS SDK (or a script) on your Kong instances to fetch the certificates from Secrets Manager.
  2. Create a script or custom plugin to:
    • Retrieve the server and client certificates from AWS Secrets Manager using the AWS CLI or SDK:

aws secretsmanager get-secret-value –secret-id kong/server

aws secretsmanager get-secret-value –secret-id kong/client

  1. Dynamically load the certificates into Kong’s SSL configuration using Lua or custom logic.
  2. Configure mTLS in Kong:
    • Set up the mTLS plugin in Kong to verify client certificates:

curl -i -X POST http://localhost:8001/services/my-service/plugins \

–data “name=mtls-auth” \

–data “config.trusted_ca_ids=<ca-id>”

Step 4: Automate Certificate Rotation

To ensure secure and automated certificate management, integrate certificate rotation.

  1. Automated Certificate Renewal in Vault:
    • Configure Vault’s PKI secrets engine to automate certificate issuance and rotation.

Example:

vault write pki/roles/kong \

allowed_domains=example.com \

allow_subdomains=true \

max_ttl=72h

Use Vault’s pki/issue endpoint to automatically rotate certificates and replace them in Kong.

  1. Automate AWS Secrets Manager Rotation:
    • Set up AWS Secrets Manager’s built-in rotation feature for SSL certificates.
  2. Trigger Updates in Kong:
    • Use a periodic task (e.g., a cron job or Ansible playbook) to update the certificates in Kong without restarting the gateway:

kong reload

This ensures Kong always uses the latest mTLS certificates from the secrets manager.

Conclusion

To integrate Kong API Gateway’s mTLS certificates with a secrets management infrastructure, follow these steps:

  1. Generate the mTLS certificates and store them securely in a secrets management tool (e.g., HashiCorp Vault, AWS Secrets Manager).
  2. Configure Kong to retrieve the certificates dynamically from the secret manager.
  3. Implement automation for certificate renewal and rotation to ensure that Kong always uses up-to-date certificates without manual intervention.

This approach enhances security by managing sensitive SSL certificates in a centralized and automated manner.