Author: techhadoop

F5 / Kong 500 error

techhadoop

Yes — F5 (BIG-IP or similar load balancer) in front of Kong can definitely contribute to or cause a 500 Internal Server Error, depending on how it’s configured.

✅ Scenarios Where F5 Can Cause 500 Errors in Kong

ScenarioExplanationHow It Leads to 500
Improper header rewritesF5 strips or rewrites headers like Host, Authorization, X-Forwarded-*Kong plugins (e.g., OIDC, JWT) fail internally
SSL offloading with missing SNIF5 terminates TLS and doesn’t forward proper SNI or client cert infoKong mTLS/auth plugins crash
Incorrect HTTP method handlingF5 mishandles certain HTTP methods (e.g., PATCH, OPTIONS)Kong routes fail or misroute
Request body corruptionF5 changes chunked encoding or breaks body formatKong/Lua fails to parse body
Timeouts/retriesF5 retry logic sends malformed or duplicate requestsKong misinterprets retries, triggers internal logic error

🔍 What You Can Check

  1. Preserve Headers:
    • Ensure F5 forwards: Host, Authorization, X-Forwarded-For, X-Forwarded-Proto, etc.
  2. SSL Passthrough or Proper Re-Termination:
    • If you terminate SSL at F5, ensure Kong receives expected headers (X-Forwarded-Proto=https, etc.)
    • If Kong expects client certs (mTLS), make sure F5 passes them correctly.
  3. Check Kong Logs:
    Look for header-related Lua errors or nil value dereferencing: attempt to index a nil value
  4. F5 Request Logs:
    Check the exact format F5 sends the request in (method, headers, body).
  5. Run curl directly to Kong (bypass F5): curl -v http://<Kong-IP>:8000/<your-path> If it works directly but fails via F5, then F5 is the culprit.

✅ Fixes and Tips

  • In F5:
    • Enable X-Forwarded-For, X-Forwarded-Proto, and Host headers.
    • Disable chunking if Kong or upstream doesn’t support it.
    • Avoid rewriting or stripping headers unless needed.
  • In Kong:
    • Enable trusted_ips and configure real_ip_header if you’re using X-Forwarded-For.

If you show me:

  • A sample request through F5 (e.g., with curl -v)
  • Any Kong plugin you’re using (OIDC, JWT, Rate Limiting, etc.)
  • A redacted error log line

…I can help pinpoint whether F5 is definitely the cause.

cert issue

techhadoop

The error “This certificate cannot be verified” typically means that the client system cannot validate the certificate chain. Here are the most common causes and how to fix them:

🔍 Root Causes and Fixes

1. Missing Intermediate or Root CA

  • Cause: The certificate presented by the server is not trusted because the issuing authority (CA) is not in the client’s trusted store.
  • Fix:
    • Ensure the full certificate chain (leaf + intermediate + root) is provided.
    • In web servers or API gateways (like Kong), configure the full chain in the cert file: cat server.crt intermediate.crt > fullchain.crt And use fullchain.crt as your public certificate.

2. Self-Signed Certificate Not Trusted

  • Cause: If it’s a self-signed cert and the CA cert is not installed on the client machine.
  • Fix:
    • Manually install the root CA certificate on the client:
      • Windows: Double-click .crt > Install Certificate > Trusted Root Certification Authorities
      • Linux/macOS: Add to /etc/ssl/certs or Keychain Access respectively

3. Expired or Not Yet Valid Certificate

  • Cause: The system clock is incorrect or the certificate has expired/not yet valid.
  • Fix:
    • Check system date/time on both client and server.
    • Ensure certificate validity dates are current.

4. Incorrect Certificate Usage (e.g., wrong SAN or CN)

  • Cause: The certificate was issued for a different domain/IP.
  • Fix:
    • Check that the certificate’s Subject Alternative Names (SAN) includes the domain/IP being accessed.

5. Corrupted or Improperly Formatted Certificate

  • Cause: The .crt or .cer file is malformed or base64-encoded incorrectly.
  • Fix:
    • Open the cert file and check that it starts and ends with: -----BEGIN CERTIFICATE----- ... base64 ... -----END CERTIFICATE-----

🧪 How to Verify

From the client:

openssl s_client -connect your.domain.com:443 -showcerts
  • This will show the certificate chain; verify whether the chain is complete and trusted.

If you tell me what OS the client is on and whether you’re using a public or private CA, I can give you platform-specific install steps.

Kong – HIGH-LEVEL AUTH FLOW

techhadoop , , , ,

Here’s a clear breakdown of how authentication works in a Kong + Ping Identity + IAM setup, which is common in enterprise environments where Kong acts as an API gateway and Ping Identity (like PingFederate/PingOne) handles user authentication and token issuance.

HIGH-LEVEL AUTH FLOW

Scenario:

Client → Kong Gateway → Protected Service
Kong integrates with Ping Identity for OIDC authentication or JWT validation, often backed by a central IAM system.

COMPONENT ROLES

ComponentRole
Kong GatewayAPI gateway that enforces authentication & authorization plugins
Ping IdentityIdentity Provider (IdP) – handles login, token issuance, and federation
IAM (e.g., Ping IAM, LDAP, AD)Stores users, groups, permissions, policies

AUTHENTICATION FLOW (OIDC Plugin)

OpenID Connect (Authorization Code Flow):

  1. Client → Kong
    Tries to access a protected API.
  2. Kong (OIDC Plugin)
    Redirects client to Ping Identity (Authorization Endpoint).
  3. Ping Identity (PingFederate or PingOne)
    • Authenticates user (UI, MFA, etc.).
    • Issues authorization_code.
  4. Kong → Ping Identity (Token Endpoint)
    Exchanges code for access_token (and optionally id_token, refresh_token).
  5. Kong (OIDC Plugin)
    • Validates token.
    • Optionally maps user to Kong consumer.
    • Passes request to backend with enriched headers (e.g., X-Consumer-Username).
  6. Backend Service
    Receives authenticated request with headers.

JWT Token Validation (Alt. Flow)

If Ping Identity issues JWT tokens, you can use Kong’s JWT plugin to validate them without redirect:

  1. Client gets access_token from Ping (out of band or SPA).
  2. Client sends request with Authorization: Bearer <token>.
  3. Kong JWT plugin verifies:
    • Signature using Ping’s public key / JWKS.
    • Claims like iss, aud, exp, etc.
  4. If valid, forward to upstream.

Where IAM Comes In

IAM sits behind Ping Identity and handles:

  • User/group storage (via LDAP, AD, DB, etc.)
  • Role mapping (e.g., admin/user)
  • Authorization policies (PingAccess, policy processor)
  • MFA/SSO rules

Ping Identity federates those identities into SSO tokens and passes them to Kong.

Plugin Example: OIDC Config

curl -X POST http://<admin>:8001/services/my-service/plugins \
  --data "name=openid-connect" \
  --data "config.issuer=https://ping-idp.com" \
  --data "config.client_id=my-client" \
  --data "config.client_secret=abc123" \
  --data "config.redirect_uri=https://my-kong.com/callback" \
  --data "config.scopes=openid profile email"

Kong Headers Passed to Backend

After successful auth, Kong can pass:

  • X-Consumer-Username
  • X-Authenticated-Userid
  • Authorization: Bearer <token> (if configured)

Optional Enhancements

  • Use ACL plugin to enforce group-level access.
  • Use OIDC group claim mapping to Kong consumer groups.
  • Enable rate limiting per consumer.
  • Log authenticated user ID to Splunk or ELK.

KONG – TPS

techhadoop

Yes, 20 TPS (Transactions Per Second) is considered low for most production workloads — unless:

✅ When 20 TPS is reasonable:

  • You’re testing or doing a soft launch.
  • The API serves high-cost or sensitive operations (e.g., financial transactions).
  • You want to protect legacy backends that can’t handle high concurrency.
  • You’re applying it per client (per consumer) rather than globally.

❗ When 20 TPS is too low:

  • You’re limiting at the global or service level and expect hundreds or thousands of users.
  • Your application is public-facing, and users experience 429 errors (rate limit exceeded).
  • You’re throttling batch jobs, mobile apps, or systems that burst above 20 TPS.

📊 Typical API Rate Benchmarks:

Use CaseTypical Rate
Internal microservice100–10,000+ TPS
Public API (freemium)50–100 TPS per key
Critical API (payments, auth)5–50 TPS with burst handling
Global rate limit1,000+ TPS

🔁 Recommendation:

  • Set burst + sustained limits, e.g.,:
    • config.second=20
    • config.hour=72_000 (20 * 3600)
  • Use Redis policy for accurate global limits across nodes.
  • Add consumer-based limits if needed.

Would you like a rate-limiting strategy per user tier (e.g., free vs. premium)?

Here’s a practical rate-limiting strategy per user tier (e.g., Free, Standard, Premium) using Kong Gateway, suitable for API plans on AWS or anywhere Kong is deployed.

Here’s a practical rate-limiting strategy per user tier (e.g., Free, Standard, Premium) using Kong Gateway, suitable for API plans on AWS or anywhere Kong is deployed.

🎯 Goal: Per-Consumer Rate Limiting (e.g., Free = 20 TPS, Premium = 100 TPS)

✅ Assumptions:

  • You’re using Kong consumers for authentication (e.g., via API key, JWT, OIDC).
  • Rate limiting is applied per consumer.
  • You want to prevent abuse while allowing scalability by tier.

🏷️ Example Rate Limits by Plan

TierTPS (config.second)Daily Limit (config.day)Notes
Free20 TPS50,000 requests/dayStrict throttling
Standard60 TPS250,000 requests/dayMore flexible
Premium100 TPS1,000,000 requests/dayHigh throughput

⚙️ Step-by-Step in Kong

1. Create Consumers

curl -X POST http://<admin-api>:8001/consumers \
     --data "username=free_user"

curl -X POST http://<admin-api>:8001/consumers \
     --data "username=standard_user"

curl -X POST http://<admin-api>:8001/consumers \
     --data "username=premium_user"

2. Assign Auth (e.g., API key)

curl -X POST http://<admin-api>:8001/consumers/free_user/key-auth

3. Apply Rate Limit Plugin to Each Consumer

Free (20 TPS, 50k/day)

curl -X POST http://<admin-api>:8001/consumers/free_user/plugins \
  --data "name=rate-limiting" \
  --data "config.second=20" \
  --data "config.day=50000" \
  --data "config.policy=local" \
  --data "config.limit_by=consumer"

Standard (60 TPS)

curl -X POST http://<admin-api>:8001/consumers/standard_user/plugins \
  --data "name=rate-limiting" \
  --data "config.second=60" \
  --data "config.day=250000" \
  --data "config.policy=local" \
  --data "config.limit_by=consumer"

Premium (100 TPS)

curl -X POST http://<admin-api>:8001/consumers/premium_user/plugins \
  --data "name=rate-limiting" \
  --data "config.second=100" \
  --data "config.day=1000000" \
  --data "config.policy=local" \
  --data "config.limit_by=consumer"

🧠 Optional: Use Redis for Distributed Rate Limiting

In multi-node setups, replace policy=local with:

--data "config.policy=redis" \
--data "config.redis_host=<your_redis_host>" \
--data "config.redis_port=6379"

🛠️ Tip: Monitor for 429 Errors

Watch for HTTP 429 Too Many Requests in Kong logs, Splunk, or Prometheus dashboards.

📘 Want Automation?

Let me know if you’d like:

  • Ansible playbook
  • deck YAML for versioned config
  • Kong Ingress YAML (for EKS)
  • Terraform for Kong setup

Just tell me your setup.

Kong – error 500

techhadoop

The “upstream status 500” in Kong Gateway means that Kong successfully forwarded the client request to the upstream service (your backend/API), but the upstream service itself responded with a 500 Internal Server Error.

🔍 What this typically indicates:

  • The issue is with the upstream service, not Kong.
  • Kong acted as a proxy and did its job correctly; the 500 came from your backend application.

🔁 Common causes on the upstream (backend) side:

  1. Unhandled exceptions (e.g., null pointer, syntax errors).
  2. Backend server crashed or restarted.
  3. Database connection failures.
  4. Application logic bug.
  5. Timeouts or memory issues in the backend.

🧪 How to troubleshoot:

  1. Check upstream (backend) logs at the time of the 500.
  2. Use curl directly on the backend API to verify behavior: curl -i http://<upstream_host>:<port>/<path>
  3. Enable debug logging in Kong if you want to trace the request: kong reload exit --conf kong.conf --vv
  4. Look in Kong logs:
    • /usr/local/kong/logs/error.log
    • See if Kong shows errors related to the upstream or if it just reports the 500.

🧩 Helpful Kong log fields:

If using Splunk or another log aggregator:

  • status: final status returned to client
  • upstream_status: status from upstream (i.e. 500)
  • upstream_uri: target path Kong used to reach the service
  • latencies.proxy, latencies.kong, latencies.request: for performance analysis

Let me know if you’d like a diagram of the request flow or want help decoding a specific log line.

KONG 401 & 400 error

techhadoop , , ,

If you’re getting 401 and 400 intermittently from the upstream service, that strongly suggests authentication-related issues, often tied to token forwarding, expiration, or format mismatch.

🔁 Quick Summary of Key Differences

StatusMeaningCommon Cause
401UnauthorizedMissing/invalid/expired credentials
400Bad RequestMalformed or incomplete request (e.g. OIDC token request)

🧠 Intermittent 401 + 400: Common Root Causes

🔸 1. Expired or Reused Tokens

  • Kong gets a token once, caches it, and keeps using it—but upstream expects a fresh one.
  • Especially common with client credentials or authorization code flows.

Solution:

  • Set token caching to short duration or disable it in the OIDC plugin: config: cache_ttl: 0 # Or a very short TTL like 5

🔸 2. Multiple Consumers with Invalid Secrets

  • One client (consumer) is configured correctly, others are not.
  • You see 401/400 when the bad client makes a request.

Solution:

  • Enable verbose logging in Kong: export KONG_LOG_LEVEL=debug kong reload Then correlate consumer_id with the error.

🔸 3. Kong Not Forwarding Tokens Correctly

  • Kong authenticates but doesn’t forward Authorization header to the upstream.
  • Some plugins strip headers by default.

Solution:

  • Add request-transformer plugin to pass the token: curl -X POST http://localhost:8001/services/YOUR_SERVICE/plugins \ --data "name=request-transformer" \ --data "config.add.headers=Authorization:Bearer $(jwt_token)"

🔸 4. OIDC Plugin Misconfiguration

If you’re using the OpenID Connect plugin:

  • grant_type, client_id, or redirect_uri may be wrong or missing intermittently.
  • Kong might request a new token but fail to pass a correct one.

Check:

  • Kong OIDC plugin config
  • Errors like: error=invalid_request error_description="Unsupported client authentication method"

🧪 How to Debug Effectively

  1. Set KONG_LOG_LEVEL=debug, reload Kong, and tail the logs: tail -f /usr/local/kong/logs/error.log
  2. Inspect Upstream Request:
    • Look for what headers/body Kong is sending.
    • Especially Authorization, Content-Type, and request body if OIDC is involved.
  3. Track Errors to a Specific Consumer:
    • Use consumer_id in the access log to trace.
    • Maybe only some consumers are misconfigured.
  4. Try Curling the Upstream Directly with the exact payload Kong sends (use Postman or curl): curl -X POST https://upstream/token \ -H "Authorization: Bearer <your_token>" \ -H "Content-Type: application/x-www-form-urlencoded" \ -d "grant_type=client_credentials&client_id=...&client_secret=..."

to check

  • Kong plugin configs (especially OIDC/JWT)
  • A few lines from Kong’s debug logs showing the upstream request/response
  • Whether you’re using Ping Identity or a custom upstream

kong – 2 services same route

techhadoop , , , ,

In Kong Gateway, you cannot have two services bound to the exact same route — a route must be unique in terms of its combination of matching rules (such as paths, hosts, methods, etc.).

🚫 Why You Can’t Have Duplicate Routes

Kong uses the route’s matching criteria to determine which service should handle a request. If two routes have the same matching rules, Kong can’t disambiguate, which leads to a conflict error when trying to create the second route.

For example:

# Service A
curl -i -X POST http://localhost:8001/services/ --data name=service-a --data url=http://api-a
curl -i -X POST http://localhost:8001/routes --data name=route-a --data service.name=service-a --data paths[]=/api

# Service B
curl -i -X POST http://localhost:8001/services/ --data name=service-b --data url=http://api-b
curl -i -X POST http://localhost:8001/routes --data name=route-b --data service.name=service-b --data paths[]=/api

⛔ The second POST /routes will fail with:

HTTP 409 Conflict – duplicate entry

✅ Workarounds / Alternatives

GoalSolution
Blue/Green or Canary DeploymentsUse Kong Plugins (e.g., traffic-split, canary) or upstreams and targets instead of duplicate routes.
Different consumers/users hitting different backendsUse request transformers, ACLs, or Kong Enterprise Route-by-header/Route-by-consumer plugins.
Same path, different method or hostYou can differentiate routes by methods[], hosts[], or headers.

🧪 Example: Two Routes with Same Path, Different Hosts

# Route 1
curl -i -X POST http://localhost:8001/routes \
  --data name=api-v1 \
  --data paths[]=/api \
  --data hosts[]=v1.example.com \
  --data service.name=service-a

# Route 2
curl -i -X POST http://localhost:8001/routes \
  --data name=api-v2 \
  --data paths[]=/api \
  --data hosts[]=v2.example.com \
  --data service.name=service-b

These can coexist because their hosts[] fields are different.

🧠 Summary

Route Matching RuleMust Be Unique For
paths[]Same host/methods/headers
hosts[]If combined with same path
methods[], headersCan disambiguate routes with same path

If you’re trying to achieve load balancing, blue-green deployment, or AB testing between services at the same route — I can help you set that up using upstreams + targets or the traffic-split plugin.

Would you like an example?

kong – ssl client

techhadoop

Yes, when using Kong Konnect with an OIDC (OpenID Connect) client, SSL (TLS) is typically required in production environments for both security and functional reasons. Here’s a breakdown of why:

✅ Why SSL/TLS is Required for OIDC with Kong Konnect

1. OIDC Specification Requires HTTPS

  • The OIDC standard mandates that communication between clients and identity providers (IdPs) must use HTTPS to protect tokens and credentials.
  • For example, the redirect_uri used in the OIDC flow must be HTTPS, or most IdPs will reject it.

2. Kong Konnect’s OIDC Plugin Requires HTTPS for Secure Redirects

  • If you’re using the Kong OIDC plugin, especially in Authorization Code Flow, Kong must redirect users to the IdP and then receive the callback securely.
  • Without SSL, the redirect URI may be considered invalid or insecure by the IdP.

KONG – 504

techhadoop

Common Causes

  1. Upstream service is slow or down
  2. DNS resolution delay or failure
  3. Incorrect protocol (HTTP vs HTTPS)
  4. Upstream timeout too low in Kong config
  5. Load balancer or firewall in between is blocking/delaying

If you’re seeing upstream_status=504 in Kong logs, it means:

Kong sent the request to the upstream (your backend service), but did not receive a response within the configured timeout, so it returned a 504 Gateway Timeout to the client.

🔍 What to Check When You See upstream_status=504

✅ 1. Verify Upstream is Healthy

Try to access your upstream from the Kong host directly:

curl -v http://<upstream-host>:<port>/<endpoint>
  • If this is slow or hangs → your backend is the problem.
  • If it fails to connect → network or DNS issue.

✅ 2. Increase Kong Timeouts

Timeouts are per-service. You can increase them using the Admin API:

curl -X PATCH http://localhost:8001/services/{service_name} \
  --data "connect_timeout=10000" \
  --data "read_timeout=15000" \
  --data "write_timeout=10000"

All values are in milliseconds.

✅ 3. Check Protocol Mismatch (HTTP vs HTTPS)

If your upstream uses HTTPS but the service is configured with protocol = http, Kong cannot complete the TLS handshake and may hang.

Fix it with:

curl -X PATCH http://localhost:8001/services/{service_name} \
  --data "protocol=https"

✅ 4. Check Kong Error Log

Run:

tail -n 100 /usr/local/kong/logs/error.log | grep timeout

You may see:

[error] ... upstream timed out (110: Connection timed out) while reading response header from upstream

✅ 5. Check Health of Upstream Targets (If Using Upstreams)

curl http://localhost:8001/upstreams/{upstream_name}/health

If all targets are unhealthy, Kong will return 504s.

✅ 6. Review Load Balancer / Firewall Rules

If Kong goes through:

  • A corporate firewall
  • A cloud load balancer
  • An IPsec/VPN tunnel

Timeouts or dropped packets there can also lead to 504.

📘 Example Full Troubleshooting Flow

  1. Run from Kong host/container: curl -v http://your-upstream-host:port/health
  2. If that times out:
    • Check if service is running.
    • Check firewall rules.
  3. Increase Kong timeouts: curl -X PATCH http://localhost:8001/services/my-api \ --data "read_timeout=20000"
  4. Check logs: tail -f /usr/local/kong/logs/error.log

If you paste the exact Kong service config and a log snippet with upstream_status=504, I can give you a more specific fix.

A 504 Gateway Timeout from Kong means:

🔴 Kong Gateway sent a request to your upstream service, but didn’t get a response in time, so it gave up and returned a 504 to the client.

✅ QUICK FIX CHECKLIST

CheckAction
⏱ Timeout too short?Increase timeout values for the service
🔌 Backend down?Test the backend directly from Kong
❌ Wrong protocol?Ensure the protocol is set correctly (http vs https)
🔐 Firewall?Check network/firewall between Kong and upstream
📉 Backend overloaded?Review CPU/memory on backend service
🔁 Load balancer in the path?Check intermediate LB timeouts

🧪 1. Test the Backend Directly

From Kong host or container:

curl -v http://<upstream-host>:<port>/health

If this hangs or fails, the issue is upstream—not Kong.

🛠️ 2. Increase Kong Timeouts

Use the Admin API:

curl -X PATCH http://localhost:8001/services/{service_name} \
  --data "connect_timeout=10000" \
  --data "read_timeout=15000" \
  --data "write_timeout=10000"

(All values are in milliseconds.)

🔐 3. Check Protocol Mismatch

Is your backend using HTTPS but your service uses HTTP?

Fix:

curl -X PATCH http://localhost:8001/services/{service_name} \
  --data "protocol=https"

📄 4. Check Kong Logs

Look in:

/usr/local/kong/logs/error.log

Typical timeout error:

upstream timed out (110: Connection timed out) while reading response header from upstream

🧰 5. Enable Correlation ID for Tracing

Enable this to track requests:

curl -X POST http://localhost:8001/plugins \
  --data "name=correlation-id" \
  --data "config.header_name=X-Correlation-ID" \
  --data "config.generator=uuid" \
  --data "config.echo_downstream=true"

Now check logs using the X-Correlation-ID.

If you send your service config or a log example with upstream_status=504, I’ll pinpoint the issue faster.

The error “An invalid response from upstream server” in Kong typically means:

Kong was able to connect to the upstream service, but the response it got back was malformed, empty, or violated protocol expectations.

🔍 Common Root Causes

CauseDescription
❌ Empty responseUpstream sent no body or headers back
⚠️ Invalid HTTP formatResponse doesn’t follow HTTP spec
🔐 SSL/TLS mismatchHTTP used with HTTPS backend or vice versa
🧱 Protocol errorUpstream speaking gRPC, WebSocket, or another protocol unexpectedly
🔌 Upstream closed connectionBefore Kong finished reading the response

✅ How to Troubleshoot

1. Check Logs

Look for this in Kong’s error log:

tail -f /usr/local/kong/logs/error.log

Common message:

upstream prematurely closed connection while reading response header

2. Test Upstream Manually

Run from the Kong host or container:

curl -i http://<upstream-host>:<port>/<path>

If you get no response or a malformed one, the problem is the upstream.

3. Verify Protocol

Make sure you’re not calling an HTTPS backend as HTTP:

# Fix protocol
curl -X PATCH http://localhost:8001/services/{service_name} \
  --data "protocol=https"

4. Increase Read Timeout

Sometimes upstreams are just too slow:

curl -X PATCH http://localhost:8001/services/{service_name} \
  --data "read_timeout=15000"

5. Check Upstream for Non-Standard Response

If the upstream returns:

  • Non-HTTP data (e.g., binary, gRPC without HTTP/1.1 framing)
  • Incorrect headers
  • Incomplete response

It will break Kong’s HTTP parser.

🧪 Bonus: Use Request Termination for Testing

Temporarily override the upstream to test Kong behavior:

curl -X POST http://localhost:8001/services/{service_name}/plugins \
  --data "name=request-termination" \
  --data "config.status_code=200"

If this works, the issue is 100% upstream-related.

If you can share the log snippet or the upstream service config (or test curl output), I can give you a pinpointed fix.