Good question 👍 — let’s break down upstream_status=500 in Kong.

🔹 What it means

• 500 = Internal Server Error returned by your upstream service, not Kong.
• Kong successfully forwarded the request to the backend, got a real HTTP response, and passed that status back.
• So unlike 502/503/504 (which usually mean Kong couldn’t reach the upstream), a 500 means Kong did reach it, but the app/server itself failed internally.

🔹 Common causes

• Unhandled exception in the app (backend crashed while processing request).
• Bad input from client → app logic throws error instead of handling gracefully.
• Dependency failure in upstream (e.g., DB connection error, null pointer, etc.).
• Misrouted traffic (Kong points to wrong host/port → some service returns 500).
• Authentication mismatch (upstream expected something different, failed internally).

🔹 How to verify it’s the upstream

1. Check Kong’s access log: status=500 upstream_status=500 → Both match = the error truly comes from upstream. status=401 upstream_status=200 → Kong overrode the upstream result (plugin decision).
2. If you’re unsure, temporarily point the route directly to a test echo server (httpbin or mockbin) and re-run → if 500 disappears, your real upstream is the culprit.

🔹 Troubleshooting steps

• Direct test upstream (bypass Kong): curl -i http://<upstream_host>:<port>/<path> If you still get 500 → it’s an upstream bug.
• Enable upstream error logging in Kong (in debug mode or use a logging plugin → Splunk/ELK).
• Check app logs on the backend service for stack traces.
• If you see status=502 upstream_status=- → that’s different (connection failed, no response).

✅ So, in short:
upstream_status=500 means your app/service returned a 500 → Kong didn’t block it, just proxied it.

Do you want me to also list the difference between 500 vs 502 vs 503 vs 504 in Kong upstream_status, so you can quickly tell where the problem is (Kong vs upstream)?

tail -f /usr/local/kong/logs/error.log | egrep -i ‘oidc|jwt|introspect|jwks|bearer|signature|aud|iss’
Typical log clues → fixes:

Got it—here’s a fast, practical playbook for “invalid bearer token” in a Kong + Ping Identity setup (with F5 in front).

Quick triage (60‑second checks)

1. Header format
   • Must be exactly: Authorization: Bearer <access_token> (case‑sensitive “Bearer”, single space, no quotes).
2. Right token type
   • Use the Access Token, not the ID token.
3. Header delivery
   • Ensure F5/LB isn’t stripping Authorization. Hit Kong directly to compare.

# Direct to Kong (bypass F5 if you can)
curl -si https://kong.example.com/api \
  -H "Authorization: Bearer $AT"

Validate the JWT (if your AT is JWT)

1) Is it a JWT?

A JWT has two dots: xxxxx.yyyyy.zzzzz. If not, it’s likely opaque → skip to “Opaque tokens”.

2) Decode & inspect claims

# header
cut -d. -f1 <<<"$AT" | base64 -d 2>/dev/null | jq
# payload
cut -d. -f2 <<<"$AT" | base64 -d 2>/dev/null | jq

Check:

• iss equals your Ping issuer in Kong config
• aud includes your API audience expected by Kong
• exp is in the future; nbf is not in the future (watch clock skew)
• scope / roles include what your route requires

3) Signature / key material

From the JWT header (kid, alg) confirm Kong can fetch the matching JWKS:

• Kong must reach Ping’s /.well-known/openid-configuration → jwks_uri
• No corporate proxy/DNS blocking
• alg matches what the IdP issues (e.g., RS256)

Validate with Kong

Turn on debug temporarily and reproduce once:

KONG_LOG_LEVEL=debug kong restart
tail -f /usr/local/kong/logs/error.log | egrep -i 'oidc|jwt|introspect|jwks|bearer|signature|aud|iss'

Typical log clues → fixes:

• signature verification failed → wrong JWKS, stale kid, or mismatched alg.
• token expired → exp passed; consider small clock_skew_leeway.
• audience not allowed / aud mismatch → adjust audience in IdP or audience_claim/allowed_audiences in plugin.
• issuer not allowed → fix issuer (exact string match).
• could not get jwks → networking/proxy or TLS trust to Ping.

Common F5/LB gotchas

• Missing Authorization header forwarding. On F5, ensure policies/iRules preserve it.
• Rewriting to lower/upper case header names is fine; removing isn’t.

Opaque tokens (non‑JWT access tokens)

If the access token looks random (no dots), enable introspection:

• Kong openid-connect plugin must have introspection_endpoint, client_id, client_secret.
• Kong needs network access to Ping’s introspection endpoint.
• Expect log lines: introspection active / token active:false if invalid or revoked.

Kong plugin configuration — key items to review

# For openid-connect plugin (per service/route)
config:
  issuer: "https://ping.example.com/as"
  discovery: "https://ping.example.com/as/.well-known/openid-configuration"
  # If using opaque tokens:
  introspection_endpoint: "https://ping.example.com/as/introspect.oauth2"
  client_id: "<kong-oidc-client>"
  client_secret: "<secret>"
  # Validation:
  verify_signature: true
  verify_claims: true
  allowed_audiences:
    - "api://my-api"
  scopes_required:
    - "api.read"
  # Optional:
  clock_skew_leeway: 60
  bearer_only: true   # if you don’t want redirects (API-only)

Upstream expectations (post‑auth)

Decide what the upstream needs:

• Forward original token: config.upstream_access_token_header = "Authorization" (default)
• Or forward derived identity headers (customize via config.upstream_headers_claims).
  Ensure your upstream checks the same audience/scope semantics (or trusts Kong’s verdict and uses headers only).

Quick curl matrix

# Good: Access token with proper prefix
curl -H "Authorization: Bearer $AT" https://api.example.com/orders

# Bad: ID token by mistake
curl -H "Authorization: Bearer $IDT" https://api.example.com/orders   # expect 401

# Test over F5 vs direct Kong
curl -si -H "Authorization: Bearer $AT" https://edge.example.com/orders
curl -si -H "Authorization: Bearer $AT" https://kong.example.com/orders

Frequent root causes & fixes

• Typo / missing “Bearer ” → fix header format.
• Using ID token → request AT (auth code or client credentials).
• Expired token → renew; verify system clocks (NTP) on F5, Kong, and upstream.
• Issuer/audience mismatch → align Ping client + API resources with Kong allowed_audiences and issuer.
• JWKS fetch failure → open egress, fix proxy/TLS trust, or pin jwks_uri.
• Opaque token with no introspection configured → enable introspection_endpoint + client creds.
• F5 stripped header → preserve Authorization.
• Scope missing → include required scopes in token request or relax scopes_required (if appropriate).

If you paste a redacted sample token header/payload (no signature needed) and your current openid-connect plugin snippet, I’ll pinpoint exactly which check is failing and provide the minimal config change or Ping policy tweak.

curl -k https://:8088/services/collector \
-H “Authorization: Splunk ” \
-H “Content-Type: application/json” \
-d ‘{“event”: “Kong test log from curl”, “sourcetype”: “_json”}’

Yes, F5 (BIG-IP) or any load balancer can cause a 401 Unauthorized error in Kong, depending on how it’s configured. Here’s how:

🔍 How F5 Can Cause 401 Errors in Kong

✅ How to Test if F5 is the Cause

1. Bypass F5 (send request directly to Kong): curl -i http://<kong-IP>:8000/<route-path> \ -H "Authorization: Bearer <your-token>"
   • If this works and through F5 it fails → F5 is modifying/blocking something.
2. Inspect headers from F5:
   • Add a logging plugin in Kong to dump all headers.
   • Use request-transformer or a custom plugin to inspect incoming headers.
3. Check F5 HTTP Profile:
   • Ensure Authorization headers are not removed.
     • F5 → Virtual Server → HTTP Profile → Request Header Erase should not include Authorization.
4. Enable debug logs in Kong:
   • Will show missing/malformed headers.

🛠 Fix Recommendations

• Preserve Authorization headers in F5 config.
• Ensure correct SSL termination and forwarding.
• Keep route paths intact when forwarding.
• Use F5 “passthrough” mode if possible for auth-related traffic.

If you share:

• A sample request/response flow through F5
• Kong route/plugin setup
• Any header manipulation in F5

…I can provide targeted help.

A 401 Unauthorized error in Kong Gateway means that the client request is missing valid authentication credentials required by Kong or the upstream service. Here’s how to troubleshoot and resolve it depending on your setup:

🔍 Common Causes & Fixes

1. No Auth Plugin Configured, but Upstream Requires Auth

• Cause: The upstream API needs an Authorization header, but it’s not provided.
• Fix: Either:
  • Add the required header at the client level.
  • Use Kong plugins (like JWT, Key Auth, OIDC) to inject it.

2. Auth Plugin Enabled, But Credentials Missing or Invalid

• Check for these plugins: key-auth, jwt, openid-connect, basic-auth, etc.
• Fix:
  • Ensure the request includes the proper header (apikey, Authorization: Bearer, etc.).
  • Example:
    • For JWT: Authorization: Bearer <token>
    • For Key-Auth: apikey: <your-api-key>
  • Confirm the credentials exist in Kong (consumer, credential objects).

3. Token Expired or Malformed

• Fix:
  • Verify the token is active and not expired.
  • Use tools like Postman or jwt.io to inspect token validity.

4. Consumer Not Mapped Correctly

• Fix:
  • Run curl http://<admin-host>:8001/consumers to verify the consumer is created.
  • Confirm credentials are linked to that consumer.

5. RBAC Token Error (For Admin API)

If you’re accessing the Admin API and get 401:

• Fix:
  • Use a valid RBAC token:
    curl -H "Authorization: Bearer <admin-token>" http://localhost:8001/routes
  • Ensure the user has the required roles.

🛠️ Diagnostic Tips

Run with -v or --include to see full request/response headers:

curl -v -H "Authorization: Bearer <token>" http://<kong-host>:8000/<route>

Enable debug logging in Kong:

log_level = debug  # in kong.conf

Want to Go Deeper?

If you share:

• The plugin you’re using (JWT, Key-Auth, OIDC, etc.)
• A sample request
• Kong logs or error message body
  … I can provide a more specific fix.

Let me know how you’re authenticating.

F5

Yes — a 401 Unauthorized error in Kong can be caused by F5 if it’s modifying or stripping headers that Kong needs for authentication.

🧱 How F5 Can Cause 401 Errors in Kong

🔴 1. Missing Auth Header

F5 may be removing or rewriting headers like:

• Authorization
• apikey
• X-Consumer-*

Kong relies on these headers when using:

• key-auth
• jwt
• basic-auth
• openid-connect

✅ Check in F5:

• Are Authorization headers allowed in HTTP requests?
• Are any “security policies” or “HTTP profile” stripping headers?

🔴 2. SSL Termination at F5

If F5 is terminating TLS, Kong may receive plain HTTP but your plugin expects HTTPS (especially with OIDC or JWT).

✅ In Kong, try setting:

KONG_TRUSTED_IPS=0.0.0.0/0
KONG_REAL_IP_HEADER=X-Forwarded-For

Also set X-Forwarded-Proto and X-Forwarded-Host in F5 to preserve the request context.

🔴 3. Incorrect or Missing Host Header

Some Kong routes match based on Host. If F5 doesn’t forward it properly, Kong can reject the request.

✅ Ensure F5 passes:

Host: your-api.domain.com

🧪 How to Confirm

Run the same request directly to Kong (bypassing F5):

curl http://<Kong-IP>:8000/your-api-path \
  -H "Authorization: Bearer <token>" \
  -H "Host: your-api.domain.com"

If it works → F5 is the problem.

✅ Fixes on F5

• Disable header stripping in HTTP profile or iRules
• Whitelist Authorization, apikey, X-* headers
• Preserve Host, X-Forwarded-For, X-Forwarded-Proto

✅ Fixes on Kong

• Add KONG_TRUSTED_IPS and KONG_REAL_IP_HEADER
• If using RBAC: make sure the token is passed in full
• Use Kong debug logs to verify header presence

If you’d like, I can help review your F5 virtual server config or Kong route/plugin setup. Just paste what you’re using.