curl -k https://:8088/services/collector \
-H “Authorization: Splunk ” \
-H “Content-Type: application/json” \
-d ‘{“event”: “Kong test log from curl”, “sourcetype”: “_json”}’

A 401 Unauthorized error in Kong Gateway means that the client request is missing valid authentication credentials required by Kong or the upstream service. Here’s how to troubleshoot and resolve it depending on your setup:

Common Causes & Fixes

1. No Auth Plugin Configured, but Upstream Requires Auth

• Cause: The upstream API needs an Authorization header, but it’s not provided.
• Fix: Either:
  • Add the required header at the client level.
  • Use Kong plugins (like JWT, Key Auth, OIDC) to inject it.

2. Auth Plugin Enabled, But Credentials Missing or Invalid

• Check for these plugins: key-auth, jwt, openid-connect, basic-auth, etc.
• Fix:
  • Ensure the request includes the proper header (apikey, Authorization: Bearer, etc.).
  • Example:
    • For JWT: Authorization: Bearer <token>
    • For Key-Auth: apikey: <your-api-key>
  • Confirm the credentials exist in Kong (consumer, credential objects).

3. Token Expired or Malformed

• Fix:
  • Verify the token is active and not expired.
  • Use tools like Postman or jwt.io to inspect token validity.

4. Consumer Not Mapped Correctly

• Fix:
  • Run curl http://<admin-host>:8001/consumers to verify the consumer is created.
  • Confirm credentials are linked to that consumer.

5. RBAC Token Error (For Admin API)

If you’re accessing the Admin API and get 401:

• Fix:
  • Use a valid RBAC token:
    curl -H "Authorization: Bearer <admin-token>" http://localhost:8001/routes
  • Ensure the user has the required roles.

Diagnostic Tips

Run with -v or --include to see full request/response headers:

curl -v -H "Authorization: Bearer <token>" http://<kong-host>:8000/<route>

Enable debug logging in Kong:

log_level = debug  # in kong.conf

F5

A 401 Unauthorized error in Kong can be caused by F5 if it’s modifying or stripping headers that Kong needs for authentication.

How F5 Can Cause 401 Errors in Kong

1. Missing Auth Header

F5 may be removing or rewriting headers like:

• Authorization
• apikey
• X-Consumer-*

Kong relies on these headers when using:

• key-auth
• jwt
• basic-auth
• openid-connect

Check in F5:

• Are Authorization headers allowed in HTTP requests?
• Are any “security policies” or “HTTP profile” stripping headers?

2. SSL Termination at F5

If F5 is terminating TLS, Kong may receive plain HTTP but your plugin expects HTTPS (especially with OIDC or JWT).

In Kong, try setting:

KONG_TRUSTED_IPS=0.0.0.0/0
KONG_REAL_IP_HEADER=X-Forwarded-For

Also set X-Forwarded-Proto and X-Forwarded-Host in F5 to preserve the request context.

3. Incorrect or Missing Host Header

Some Kong routes match based on Host. If F5 doesn’t forward it properly, Kong can reject the request.

✅ Ensure F5 passes:

Host: your-api.domain.com

How to Confirm

Run the same request directly to Kong (bypassing F5):

curl http://<Kong-IP>:8000/your-api-path \
  -H "Authorization: Bearer <token>" \
  -H "Host: your-api.domain.com"

If it works → F5 is the problem.

Fixes on F5

• Disable header stripping in HTTP profile or iRules
• Whitelist Authorization, apikey, X-* headers
• Preserve Host, X-Forwarded-For, X-Forwarded-Proto

Fixes on Kong

• Add KONG_TRUSTED_IPS and KONG_REAL_IP_HEADER
• If using RBAC: make sure the token is passed in full
• Use Kong debug logs to verify header presence

root@test:~# docker inspect 66ce28cab318 | grep IPAddress
            "SecondaryIPAddresses": null,
            "IPAddress": "",
                    "IPAddress": "172.21.0.2",

docker network inspect bridge
docker inspect <container_id> | grep IPAddress

F5 (BIG-IP or similar load balancer) in front of Kong can definitely contribute to or cause a 500 Internal Server Error, depending on how it’s configured.

✅ Scenarios Where F5 Can Cause 500 Errors in Kong

What You Can Check

1. Preserve Headers:
   • Ensure F5 forwards: Host, Authorization, X-Forwarded-For, X-Forwarded-Proto, etc.
2. SSL Passthrough or Proper Re-Termination:
   • If you terminate SSL at F5, ensure Kong receives expected headers (X-Forwarded-Proto=https, etc.)
   • If Kong expects client certs (mTLS), make sure F5 passes them correctly.
3. Check Kong Logs:
   Look for header-related Lua errors or nil value dereferencing: attempt to index a nil value
4. F5 Request Logs:
   Check the exact format F5 sends the request in (method, headers, body).
5. Run curl directly to Kong (bypass F5): curl -v http://<Kong-IP>:8000/<your-path> If it works directly but fails via F5, then F5 is the culprit.

Fixes and Tips

• In F5:
  • Enable X-Forwarded-For, X-Forwarded-Proto, and Host headers.
  • Disable chunking if Kong or upstream doesn’t support it.
  • Avoid rewriting or stripping headers unless needed.
• In Kong:
  • Enable trusted_ips and configure real_ip_header if you’re using X-Forwarded-For.

The error “This certificate cannot be verified” typically means that the client system cannot validate the certificate chain. Here are the most common causes and how to fix them:

Root Causes and Fixes

1. Missing Intermediate or Root CA

• Cause: The certificate presented by the server is not trusted because the issuing authority (CA) is not in the client’s trusted store.
• Fix:
  • Ensure the full certificate chain (leaf + intermediate + root) is provided.
  • In web servers or API gateways (like Kong), configure the full chain in the cert file: cat server.crt intermediate.crt > fullchain.crt And use fullchain.crt as your public certificate.

2. Self-Signed Certificate Not Trusted

• Cause: If it’s a self-signed cert and the CA cert is not installed on the client machine.
• Fix:
  • Manually install the root CA certificate on the client:
    • Windows: Double-click .crt > Install Certificate > Trusted Root Certification Authorities
    • Linux/macOS: Add to /etc/ssl/certs or Keychain Access respectively

3. Expired or Not Yet Valid Certificate

• Cause: The system clock is incorrect or the certificate has expired/not yet valid.
• Fix:
  • Check system date/time on both client and server.
  • Ensure certificate validity dates are current.

4. Incorrect Certificate Usage (e.g., wrong SAN or CN)

• Cause: The certificate was issued for a different domain/IP.
• Fix:
  • Check that the certificate’s Subject Alternative Names (SAN) includes the domain/IP being accessed.

5. Corrupted or Improperly Formatted Certificate

• Cause: The .crt or .cer file is malformed or base64-encoded incorrectly.
• Fix:
  • Open the cert file and check that it starts and ends with: -----BEGIN CERTIFICATE----- ... base64 ... -----END CERTIFICATE-----

How to Verify

From the client:

openssl s_client -connect your.domain.com:443 -showcerts

• This will show the certificate chain; verify whether the chain is complete and trusted.

Yes, 20 TPS (Transactions Per Second) is considered low for most production workloads — unless:

When 20 TPS is reasonable:

• You’re testing or doing a soft launch.
• The API serves high-cost or sensitive operations (e.g., financial transactions).
• You want to protect legacy backends that can’t handle high concurrency.
• You’re applying it per client (per consumer) rather than globally.

When 20 TPS is too low:

• You’re limiting at the global or service level and expect hundreds or thousands of users.
• Your application is public-facing, and users experience 429 errors (rate limit exceeded).
• You’re throttling batch jobs, mobile apps, or systems that burst above 20 TPS.

Typical API Rate Benchmarks:

Recommendation:

• Set burst + sustained limits, e.g.,:
  • config.second=20
  • config.hour=72_000 (20 * 3600)
• Use Redis policy for accurate global limits across nodes.
• Add consumer-based limits if needed.

Would you like a rate-limiting strategy per user tier (e.g., free vs. premium)?

Here’s a practical rate-limiting strategy per user tier (e.g., Free, Standard, Premium) using Kong Gateway, suitable for API plans on AWS or anywhere Kong is deployed.

Here’s a practical rate-limiting strategy per user tier (e.g., Free, Standard, Premium) using Kong Gateway, suitable for API plans on AWS or anywhere Kong is deployed.

Goal: Per-Consumer Rate Limiting (e.g., Free = 20 TPS, Premium = 100 TPS)

Assumptions:

• You’re using Kong consumers for authentication (e.g., via API key, JWT, OIDC).
• Rate limiting is applied per consumer.
• You want to prevent abuse while allowing scalability by tier.

Example Rate Limits by Plan

⚙️ Step-by-Step in Kong

1. Create Consumers

curl -X POST http://<admin-api>:8001/consumers \
     --data "username=free_user"

curl -X POST http://<admin-api>:8001/consumers \
     --data "username=standard_user"

curl -X POST http://<admin-api>:8001/consumers \
     --data "username=premium_user"

2. Assign Auth (e.g., API key)

curl -X POST http://<admin-api>:8001/consumers/free_user/key-auth

3. Apply Rate Limit Plugin to Each Consumer

Free (20 TPS, 50k/day)

curl -X POST http://<admin-api>:8001/consumers/free_user/plugins \
  --data "name=rate-limiting" \
  --data "config.second=20" \
  --data "config.day=50000" \
  --data "config.policy=local" \
  --data "config.limit_by=consumer"

Standard (60 TPS)

curl -X POST http://<admin-api>:8001/consumers/standard_user/plugins \
  --data "name=rate-limiting" \
  --data "config.second=60" \
  --data "config.day=250000" \
  --data "config.policy=local" \
  --data "config.limit_by=consumer"

Premium (100 TPS)

curl -X POST http://<admin-api>:8001/consumers/premium_user/plugins \
  --data "name=rate-limiting" \
  --data "config.second=100" \
  --data "config.day=1000000" \
  --data "config.policy=local" \
  --data "config.limit_by=consumer"

Optional: Use Redis for Distributed Rate Limiting

In multi-node setups, replace policy=local with:

--data "config.policy=redis" \
--data "config.redis_host=<your_redis_host>" \
--data "config.redis_port=6379"

Tip: Monitor for 429 Errors

Watch for HTTP 429 Too Many Requests in Kong logs, Splunk, or Prometheus dashboards.

The “upstream status 500” in Kong Gateway means that Kong successfully forwarded the client request to the upstream service (your backend/API), but the upstream service itself responded with a 500 Internal Server Error.

What this typically indicates:

• The issue is with the upstream service, not Kong.
• Kong acted as a proxy and did its job correctly; the 500 came from your backend application.

Common causes on the upstream (backend) side:

1. Unhandled exceptions (e.g., null pointer, syntax errors).
2. Backend server crashed or restarted.
3. Database connection failures.
4. Application logic bug.
5. Timeouts or memory issues in the backend.

How to troubleshoot:

1. Check upstream (backend) logs at the time of the 500.
2. Use curl directly on the backend API to verify behavior: curl -i http://<upstream_host>:<port>/<path>
3. Enable debug logging in Kong if you want to trace the request: kong reload exit --conf kong.conf --vv
4. Look in Kong logs:
   • /usr/local/kong/logs/error.log
   • See if Kong shows errors related to the upstream or if it just reports the 500.

Helpful Kong log fields:

If using Splunk or another log aggregator:

• status: final status returned to client
• upstream_status: status from upstream (i.e. 500)
• upstream_uri: target path Kong used to reach the service
• latencies.proxy, latencies.kong, latencies.request: for performance analysis