tail -f /usr/local/kong/logs/error.log | egrep -i ‘oidc|jwt|introspect|jwks|bearer|signature|aud|iss’
Typical log clues → fixes:

Got it—here’s a fast, practical playbook for “invalid bearer token” in a Kong + Ping Identity setup (with F5 in front).

Quick triage (60‑second checks)

1. Header format
   • Must be exactly: Authorization: Bearer <access_token> (case‑sensitive “Bearer”, single space, no quotes).
2. Right token type
   • Use the Access Token, not the ID token.
3. Header delivery
   • Ensure F5/LB isn’t stripping Authorization. Hit Kong directly to compare.

# Direct to Kong (bypass F5 if you can)
curl -si https://kong.example.com/api \
  -H "Authorization: Bearer $AT"

Validate the JWT (if your AT is JWT)

1) Is it a JWT?

A JWT has two dots: xxxxx.yyyyy.zzzzz. If not, it’s likely opaque → skip to “Opaque tokens”.

2) Decode & inspect claims

# header
cut -d. -f1 <<<"$AT" | base64 -d 2>/dev/null | jq
# payload
cut -d. -f2 <<<"$AT" | base64 -d 2>/dev/null | jq

Check:

• iss equals your Ping issuer in Kong config
• aud includes your API audience expected by Kong
• exp is in the future; nbf is not in the future (watch clock skew)
• scope / roles include what your route requires

3) Signature / key material

From the JWT header (kid, alg) confirm Kong can fetch the matching JWKS:

• Kong must reach Ping’s /.well-known/openid-configuration → jwks_uri
• No corporate proxy/DNS blocking
• alg matches what the IdP issues (e.g., RS256)

Validate with Kong

Turn on debug temporarily and reproduce once:

KONG_LOG_LEVEL=debug kong restart
tail -f /usr/local/kong/logs/error.log | egrep -i 'oidc|jwt|introspect|jwks|bearer|signature|aud|iss'

Typical log clues → fixes:

• signature verification failed → wrong JWKS, stale kid, or mismatched alg.
• token expired → exp passed; consider small clock_skew_leeway.
• audience not allowed / aud mismatch → adjust audience in IdP or audience_claim/allowed_audiences in plugin.
• issuer not allowed → fix issuer (exact string match).
• could not get jwks → networking/proxy or TLS trust to Ping.

Common F5/LB gotchas

• Missing Authorization header forwarding. On F5, ensure policies/iRules preserve it.
• Rewriting to lower/upper case header names is fine; removing isn’t.

Opaque tokens (non‑JWT access tokens)

If the access token looks random (no dots), enable introspection:

• Kong openid-connect plugin must have introspection_endpoint, client_id, client_secret.
• Kong needs network access to Ping’s introspection endpoint.
• Expect log lines: introspection active / token active:false if invalid or revoked.

Kong plugin configuration — key items to review

# For openid-connect plugin (per service/route)
config:
  issuer: "https://ping.example.com/as"
  discovery: "https://ping.example.com/as/.well-known/openid-configuration"
  # If using opaque tokens:
  introspection_endpoint: "https://ping.example.com/as/introspect.oauth2"
  client_id: "<kong-oidc-client>"
  client_secret: "<secret>"
  # Validation:
  verify_signature: true
  verify_claims: true
  allowed_audiences:
    - "api://my-api"
  scopes_required:
    - "api.read"
  # Optional:
  clock_skew_leeway: 60
  bearer_only: true   # if you don’t want redirects (API-only)

Upstream expectations (post‑auth)

Decide what the upstream needs:

• Forward original token: config.upstream_access_token_header = "Authorization" (default)
• Or forward derived identity headers (customize via config.upstream_headers_claims).
  Ensure your upstream checks the same audience/scope semantics (or trusts Kong’s verdict and uses headers only).

Quick curl matrix

# Good: Access token with proper prefix
curl -H "Authorization: Bearer $AT" https://api.example.com/orders

# Bad: ID token by mistake
curl -H "Authorization: Bearer $IDT" https://api.example.com/orders   # expect 401

# Test over F5 vs direct Kong
curl -si -H "Authorization: Bearer $AT" https://edge.example.com/orders
curl -si -H "Authorization: Bearer $AT" https://kong.example.com/orders

Frequent root causes & fixes

• Typo / missing “Bearer ” → fix header format.
• Using ID token → request AT (auth code or client credentials).
• Expired token → renew; verify system clocks (NTP) on F5, Kong, and upstream.
• Issuer/audience mismatch → align Ping client + API resources with Kong allowed_audiences and issuer.
• JWKS fetch failure → open egress, fix proxy/TLS trust, or pin jwks_uri.
• Opaque token with no introspection configured → enable introspection_endpoint + client creds.
• F5 stripped header → preserve Authorization.
• Scope missing → include required scopes in token request or relax scopes_required (if appropriate).

If you paste a redacted sample token header/payload (no signature needed) and your current openid-connect plugin snippet, I’ll pinpoint exactly which check is failing and provide the minimal config change or Ping policy tweak.

curl -k https://:8088/services/collector \
-H “Authorization: Splunk ” \
-H “Content-Type: application/json” \
-d ‘{“event”: “Kong test log from curl”, “sourcetype”: “_json”}’

Yes, F5 (BIG-IP) or any load balancer can cause a 401 Unauthorized error in Kong, depending on how it’s configured. Here’s how:

🔍 How F5 Can Cause 401 Errors in Kong

✅ How to Test if F5 is the Cause

1. Bypass F5 (send request directly to Kong): curl -i http://<kong-IP>:8000/<route-path> \ -H "Authorization: Bearer <your-token>"
   • If this works and through F5 it fails → F5 is modifying/blocking something.
2. Inspect headers from F5:
   • Add a logging plugin in Kong to dump all headers.
   • Use request-transformer or a custom plugin to inspect incoming headers.
3. Check F5 HTTP Profile:
   • Ensure Authorization headers are not removed.
     • F5 → Virtual Server → HTTP Profile → Request Header Erase should not include Authorization.
4. Enable debug logs in Kong:
   • Will show missing/malformed headers.

🛠 Fix Recommendations

• Preserve Authorization headers in F5 config.
• Ensure correct SSL termination and forwarding.
• Keep route paths intact when forwarding.
• Use F5 “passthrough” mode if possible for auth-related traffic.

If you share:

• A sample request/response flow through F5
• Kong route/plugin setup
• Any header manipulation in F5

…I can provide targeted help.

A 401 Unauthorized error in Kong Gateway means that the client request is missing valid authentication credentials required by Kong or the upstream service. Here’s how to troubleshoot and resolve it depending on your setup:

🔍 Common Causes & Fixes

1. No Auth Plugin Configured, but Upstream Requires Auth

• Cause: The upstream API needs an Authorization header, but it’s not provided.
• Fix: Either:
  • Add the required header at the client level.
  • Use Kong plugins (like JWT, Key Auth, OIDC) to inject it.

2. Auth Plugin Enabled, But Credentials Missing or Invalid

• Check for these plugins: key-auth, jwt, openid-connect, basic-auth, etc.
• Fix:
  • Ensure the request includes the proper header (apikey, Authorization: Bearer, etc.).
  • Example:
    • For JWT: Authorization: Bearer <token>
    • For Key-Auth: apikey: <your-api-key>
  • Confirm the credentials exist in Kong (consumer, credential objects).

3. Token Expired or Malformed

• Fix:
  • Verify the token is active and not expired.
  • Use tools like Postman or jwt.io to inspect token validity.

4. Consumer Not Mapped Correctly

• Fix:
  • Run curl http://<admin-host>:8001/consumers to verify the consumer is created.
  • Confirm credentials are linked to that consumer.

5. RBAC Token Error (For Admin API)

If you’re accessing the Admin API and get 401:

• Fix:
  • Use a valid RBAC token:
    curl -H "Authorization: Bearer <admin-token>" http://localhost:8001/routes
  • Ensure the user has the required roles.

🛠️ Diagnostic Tips

Run with -v or --include to see full request/response headers:

curl -v -H "Authorization: Bearer <token>" http://<kong-host>:8000/<route>

Enable debug logging in Kong:

log_level = debug  # in kong.conf

Want to Go Deeper?

If you share:

• The plugin you’re using (JWT, Key-Auth, OIDC, etc.)
• A sample request
• Kong logs or error message body
  … I can provide a more specific fix.

Let me know how you’re authenticating.

F5

Yes — a 401 Unauthorized error in Kong can be caused by F5 if it’s modifying or stripping headers that Kong needs for authentication.

🧱 How F5 Can Cause 401 Errors in Kong

🔴 1. Missing Auth Header

F5 may be removing or rewriting headers like:

• Authorization
• apikey
• X-Consumer-*

Kong relies on these headers when using:

• key-auth
• jwt
• basic-auth
• openid-connect

✅ Check in F5:

• Are Authorization headers allowed in HTTP requests?
• Are any “security policies” or “HTTP profile” stripping headers?

🔴 2. SSL Termination at F5

If F5 is terminating TLS, Kong may receive plain HTTP but your plugin expects HTTPS (especially with OIDC or JWT).

✅ In Kong, try setting:

KONG_TRUSTED_IPS=0.0.0.0/0
KONG_REAL_IP_HEADER=X-Forwarded-For

Also set X-Forwarded-Proto and X-Forwarded-Host in F5 to preserve the request context.

🔴 3. Incorrect or Missing Host Header

Some Kong routes match based on Host. If F5 doesn’t forward it properly, Kong can reject the request.

✅ Ensure F5 passes:

Host: your-api.domain.com

🧪 How to Confirm

Run the same request directly to Kong (bypassing F5):

curl http://<Kong-IP>:8000/your-api-path \
  -H "Authorization: Bearer <token>" \
  -H "Host: your-api.domain.com"

If it works → F5 is the problem.

✅ Fixes on F5

• Disable header stripping in HTTP profile or iRules
• Whitelist Authorization, apikey, X-* headers
• Preserve Host, X-Forwarded-For, X-Forwarded-Proto

✅ Fixes on Kong

• Add KONG_TRUSTED_IPS and KONG_REAL_IP_HEADER
• If using RBAC: make sure the token is passed in full
• Use Kong debug logs to verify header presence

If you’d like, I can help review your F5 virtual server config or Kong route/plugin setup. Just paste what you’re using.

root@test:~# docker inspect 66ce28cab318 | grep IPAddress
            "SecondaryIPAddresses": null,
            "IPAddress": "",
                    "IPAddress": "172.21.0.2",

docker network inspect bridge
docker inspect <container_id> | grep IPAddress

Yes — F5 (BIG-IP or similar load balancer) in front of Kong can definitely contribute to or cause a 500 Internal Server Error, depending on how it’s configured.

✅ Scenarios Where F5 Can Cause 500 Errors in Kong

🔍 What You Can Check

1. Preserve Headers:
   • Ensure F5 forwards: Host, Authorization, X-Forwarded-For, X-Forwarded-Proto, etc.
2. SSL Passthrough or Proper Re-Termination:
   • If you terminate SSL at F5, ensure Kong receives expected headers (X-Forwarded-Proto=https, etc.)
   • If Kong expects client certs (mTLS), make sure F5 passes them correctly.
3. Check Kong Logs:
   Look for header-related Lua errors or nil value dereferencing: attempt to index a nil value
4. F5 Request Logs:
   Check the exact format F5 sends the request in (method, headers, body).
5. Run curl directly to Kong (bypass F5): curl -v http://<Kong-IP>:8000/<your-path> If it works directly but fails via F5, then F5 is the culprit.

✅ Fixes and Tips

• In F5:
  • Enable X-Forwarded-For, X-Forwarded-Proto, and Host headers.
  • Disable chunking if Kong or upstream doesn’t support it.
  • Avoid rewriting or stripping headers unless needed.
• In Kong:
  • Enable trusted_ips and configure real_ip_header if you’re using X-Forwarded-For.

If you show me:

• A sample request through F5 (e.g., with curl -v)
• Any Kong plugin you’re using (OIDC, JWT, Rate Limiting, etc.)
• A redacted error log line

…I can help pinpoint whether F5 is definitely the cause.

The error “This certificate cannot be verified” typically means that the client system cannot validate the certificate chain. Here are the most common causes and how to fix them:

🔍 Root Causes and Fixes

1. Missing Intermediate or Root CA

• Cause: The certificate presented by the server is not trusted because the issuing authority (CA) is not in the client’s trusted store.
• Fix:
  • Ensure the full certificate chain (leaf + intermediate + root) is provided.
  • In web servers or API gateways (like Kong), configure the full chain in the cert file: cat server.crt intermediate.crt > fullchain.crt And use fullchain.crt as your public certificate.

2. Self-Signed Certificate Not Trusted

• Cause: If it’s a self-signed cert and the CA cert is not installed on the client machine.
• Fix:
  • Manually install the root CA certificate on the client:
    • Windows: Double-click .crt > Install Certificate > Trusted Root Certification Authorities
    • Linux/macOS: Add to /etc/ssl/certs or Keychain Access respectively

3. Expired or Not Yet Valid Certificate

• Cause: The system clock is incorrect or the certificate has expired/not yet valid.
• Fix:
  • Check system date/time on both client and server.
  • Ensure certificate validity dates are current.

4. Incorrect Certificate Usage (e.g., wrong SAN or CN)

• Cause: The certificate was issued for a different domain/IP.
• Fix:
  • Check that the certificate’s Subject Alternative Names (SAN) includes the domain/IP being accessed.

5. Corrupted or Improperly Formatted Certificate

• Cause: The .crt or .cer file is malformed or base64-encoded incorrectly.
• Fix:
  • Open the cert file and check that it starts and ends with: -----BEGIN CERTIFICATE----- ... base64 ... -----END CERTIFICATE-----

🧪 How to Verify

From the client:

openssl s_client -connect your.domain.com:443 -showcerts

• This will show the certificate chain; verify whether the chain is complete and trusted.

If you tell me what OS the client is on and whether you’re using a public or private CA, I can give you platform-specific install steps.