ARCHITECTURE SUMMARY

Kong AI Gateway — Unified Architecture

Single Ingress Point for All AI Model and MCP Traffic

Mission

Every AI interaction in the enterprise — app-to-model, agent-to-agent, agent-to-tool — flows through one gateway. No exceptions, no direct provider calls, no shadow credentials. This document ties together the four pillars of the architecture.

1. Single Ingress Point

All apps, agents, and MCP clients hit one endpoint family on Kong, never a provider directly. This is the architectural premise everything else depends on: if traffic can bypass the gateway, none of the governance below actually holds.

2. Credential Governance

• Every provider key (Anthropic, Bedrock, OpenAI, internal APIs, MCP servers) lives in a vault, never in app code or Kong config.
• Agents authenticate to Kong with their own Consumer credential — an identity Kong owns and can revoke independently of the underlying provider key.
• Kong resolves the real credential at request time and injects it server-side. A leaked client-side key exposes nothing about the provider account behind it.
• One Consumer model covers both LLM and MCP auth (OAuth2 for MCP), so access governance and credential governance are the same system.

3. MCP Registry & Gateway

• The MCP Registry (Konnect Catalog, tech preview) catalogs approved MCP servers — owner, purpose, status — so agents discover tools in one place instead of hardcoding server URLs.
• The AI MCP Proxy plugin fronts internal servers (passthrough), converts REST APIs into tools (conversion), or aggregates multiple servers into one endpoint (listener).
• Tool-level ACLs gate both discovery and invocation, so a Consumer’s permissions extend down to individual tools, not just “can call MCP or not.”

4. Multi-Provider Failover

• AI Proxy Advanced’s priority algorithm ranks providers into tiers (e.g. Anthropic → Bedrock → OpenAI). The top tier takes all traffic while healthy; the balancer only spills to the next tier on failure.
• Active and passive health checks on the Upstream detect a degraded provider and flip targets automatically — no manual intervention during an outage.
• Each tier still uses its own vault-backed credential, since Bedrock’s SigV4 auth, Anthropic’s API key, and OpenAI’s bearer token are structurally different.

5. Observability — The Thread Through All Three

• Metrics, audit logs, and cost data are captured centrally, keyed to Consumer identity, not to whichever provider or MCP server actually served the request.
• Failover events, tool invocations, and credential resolutions all land in the same logging/alerting pipeline, so “who called what, through which path, at what cost” is answerable from one place.

The Four Pillars at a Glance

Why This Design Holds Together

Take any pillar away and the “single ingress point” claim weakens:

• No credential governance → the gateway is a pass-through, not a security boundary.
• No MCP registry/gateway → agents discover and call tools outside Kong’s visibility.
• No failover → a single provider outage becomes a company-wide AI outage.
• No shared observability → incidents can’t be traced back to an owner or a root cause.

Together, they turn “Kong is where AI traffic goes” into “Kong is where AI traffic is governed” — which is the actual job of the architect role.

I would implement it from scratch in a controlled POC first, using Kong Gateway Enterprise 3.16 + PostgreSQL + Ollama, so the whole AI path can run on-prem without needing an external LLM API. Once that works, replace/augment Ollama with Azure OpenAI, OpenAI, vLLM, etc.

Kong’s current self-managed AI Gateway model uses standard Services + Routes + AI plugins; for multiple providers/models, the main plugin is ai-proxy-advanced. That plugin requires an AI license. Kong Docs

1. Target architecture

Start with this:

                 Client / Application
                         |
                         | HTTPS
                         v
                 +----------------+
                 |      Kong      |
                 |   AI Gateway   |
                 |                |
                 | :8000 Proxy    |
                 | :8001 Admin    |
                 +-------+--------+
                         |
                AI Proxy Advanced
                         |
                         v
                 +---------------+
                 |    Ollama     |
                 |    :11434     |
                 +-------+-------+
                         |
                         v
                     Qwen/Llama

                 +---------------+
                 |  PostgreSQL   |
                 |     :5432     |
                 +---------------+

For a lab, I’d use three Ubuntu 24.04 VMs:

kong01       4 vCPU   8 GB RAM
postgres01   2 vCPU   4 GB RAM
ollama01     8+ vCPU  16+ GB RAM

GPU on ollama01 is preferable but isn’t required for basic testing.

2. Install PostgreSQL

On postgres01:

sudo apt update
sudo apt install postgresql postgresql-contrib -y

Create Kong DB/user:

sudo -u postgres psql

Then:

CREATE USER kong WITH PASSWORD 'StrongKongPassword';
CREATE DATABASE kong OWNER kong;
GRANT ALL PRIVILEGES ON DATABASE kong TO kong;
\q

Configure PostgreSQL to accept your Kong server.

Edit:

sudo vi /etc/postgresql/*/main/postgresql.conf

Set:

listen_addresses = '*'

Then:

sudo vi /etc/postgresql/*/main/pg_hba.conf

For example:

host    kong    kong    192.168.10.0/24    scram-sha-256

Restart:

sudo systemctl restart postgresql
sudo systemctl enable postgresql

Test from Kong VM:

psql -h postgres01 -U kong -d kong

3. Install Kong Gateway

Kong currently documents self-managed Gateway 3.16.0.0 packages for Ubuntu. Kong Docs

On kong01:

sudo apt update
sudo apt install -y curl gnupg

Add Kong repository:

curl -1sLf \
"https://packages.konghq.com/public/gateway-316/gpg.998DFF461A62FF7C.key" \
| gpg --dearmor \
| sudo tee /usr/share/keyrings/kong-gateway-316-archive-keyring.gpg \
> /dev/null

For Ubuntu 24.04/Noble:

curl -1sLf \
"https://packages.konghq.com/public/gateway-316/config.deb.txt?distro=ubuntu&codename=noble" \
| sudo tee /etc/apt/sources.list.d/kong-gateway-316.list

Then:

sudo apt update
sudo apt install kong-enterprise-edition=3.16.0.0

Verify:

kong version

Expected roughly:

3.16.0.0

4. Configure Kong

Edit:

sudo vi /etc/kong/kong.conf

Basic lab configuration:

database = postgres
pg_host = postgres01
pg_port = 5432
pg_user = kong
pg_password = StrongKongPassword
pg_database = kong
proxy_listen = 0.0.0.0:8000
proxy_listen_ssl = 0.0.0.0:8443 ssl
admin_listen = 0.0.0.0:8001
admin_listen_ssl = 0.0.0.0:8444 ssl
status_listen = 0.0.0.0:8100
admin_gui_listen = 0.0.0.0:8002
admin_gui_ssl_listen = 0.0.0.0:8445

Do not expose 8001/8002 externally in production.

5. Add the Kong Enterprise / AI license

AI Proxy Advanced is an AI-license-required plugin. Kong Docs

Your license normally gets configured securely rather than embedded directly into scripts.

Confirm Kong sees the relevant plugins:

kong config parse /etc/kong/kong.conf

Later you can also check:

curl http://localhost:8001/plugins/enabled

Look for:

ai-proxy
ai-proxy-advanced
ai-prompt-guard
ai-rate-limiting-advanced
...

6. Bootstrap the Kong database

Run:

sudo -u kong kong migrations bootstrap \
  -c /etc/kong/kong.conf

Then:

sudo kong start -c /etc/kong/kong.conf

Check:

curl http://localhost:8001/status

And:

curl http://localhost:8001

7. Install Ollama

Now create the local AI backend.

On ollama01, install Ollama using its supported installation method for your OS.

After installation verify:

systemctl status ollama

It normally listens on:

11434

You need it reachable from Kong:

kong01
   |
   | TCP 11434
   v
ollama01

Test connectivity from Kong:

curl http://ollama01:11434/api/tags

You should receive JSON.

8. Pull an LLM

For the current Kong example, Kong documents a local Qwen 3 model using Ollama. Kong Docs

On Ollama:

ollama pull qwen3

Check:

ollama list

You should see:

NAME
qwen3

Test Ollama directly first.

For example:

curl http://ollama01:11434/api/chat \
  -H "Content-Type: application/json" \
  -d '{
        "model":"qwen3",
        "messages":[
          {
            "role":"user",
            "content":"What is Kong Gateway?"
          }
        ]
      }'

This is important: make Ollama work before introducing Kong.

9. Create the Kong AI Service

Create a Service:

curl -X POST http://localhost:8001/services \
  --data name=ai-service \
  --data url=http://ollama01:11434

Verify:

curl http://localhost:8001/services/ai-service

10. Create the AI Route

Create:

curl -X POST \
  http://localhost:8001/services/ai-service/routes \
  --data name=ai-chat-route \
  --data 'paths[]=/ai/chat'

Check:

curl http://localhost:8001/routes

You now have:

Client
   |
   v
http://kong01:8000/ai/chat
   |
   v
ai-service

But we haven’t added AI translation yet.

11. Enable AI Proxy Advanced

This is the important part.

Attach ai-proxy-advanced to the route:

curl -X POST \
http://localhost:8001/routes/ai-chat-route/plugins \
-H "Content-Type: application/json" \
-d '{
  "name": "ai-proxy-advanced",
  "config": {
    "targets": [
      {
        "route_type": "llm/v1/chat",
        "model": {
          "provider": "ollama",
          "name": "qwen3",
          "options": {
            "upstream_url": "http://ollama01:11434"
          }
        }
      }
    ]
  }
}'

This configuration matches Kong’s documented Ollama/Qwen AI Proxy Advanced pattern. Kong Docs

Check it:

curl \
http://localhost:8001/routes/ai-chat-route/plugins

12. Test through Kong

Now the important test.

Instead of calling:

Ollama directly

you call:

Kong

Example:

curl -X POST \
http://kong01:8000/ai/chat \
-H "Content-Type: application/json" \
-d '{
  "messages": [
    {
      "role": "user",
      "content": "Explain what an API gateway does."
    }
  ]
}'

Flow:

curl
 |
 v
Kong :8000
 |
Route
 |
AI Proxy Advanced
 |
provider = ollama
 |
model = qwen3
 |
 v
Ollama :11434
 |
 v
Qwen

At this point you have built your first working Kong AI Gateway.

13. Add API authentication

Don’t leave:

/ai/chat

public.

For the initial lab you can use key-auth:

curl -X POST \
http://localhost:8001/routes/ai-chat-route/plugins \
--data name=key-auth

Create Consumer:

curl -X POST \
http://localhost:8001/consumers \
--data username=ai-application

Create key:

curl -X POST \
http://localhost:8001/consumers/ai-application/key-auth \
--data key=my-secret-ai-key

Test without key:

curl http://kong01:8000/ai/chat

Should return:

401 Unauthorized

Use:

curl -X POST \
http://kong01:8000/ai/chat \
-H "apikey: my-secret-ai-key" \
-H "Content-Type: application/json" \
-d '{
 "messages":[
   {
     "role":"user",
     "content":"Explain Kong AI Gateway."
   }
 ]
}'

For production, given your existing environment, I’d replace key-auth with Ping Identity + OIDC.

14. Enable Prometheus AI metrics

Kong’s Prometheus plugin supports AI-specific metrics including requests, token consumption, provider/model information, LLM latency and cost information where available. Set:

ai_metrics = true

Kong documents AI metrics from Gateway 3.8+, including consumer attribution in newer releases. Kong Docs

Enable globally:

curl -X POST \
http://localhost:8001/plugins \
--data name=prometheus \
--data config.status_code_metrics=true \
--data config.latency_metrics=true \
--data config.ai_metrics=true

Check:

curl http://kong01:8100/metrics

Useful metrics will include Kong latency plus AI-specific measurements.

Kong recommends the Status API /metrics endpoint when available rather than exposing the Admin API to Prometheus. Kong Docs

15. Prometheus configuration

On Prometheus:

scrape_configs:
  - job_name: "kong-ai-gateway"
    scrape_interval: 15s
    static_configs:
      - targets:
          - "kong01:8100"

Restart/reload Prometheus.

Check:

Status → Targets

Kong should show:

UP

16. Grafana dashboard

I would build panels for:

AI Gateway Requests/sec
Input Tokens/sec
Output Tokens/sec
Kong latency:
 p50
 p95
 p99
LLM latency:
 p50
 p95
 p99
HTTP:
 2xx
 4xx
 429
 5xx
AI provider
AI model
Consumer
Cache hit/miss

For Kong p95:

histogram_quantile(
  0.95,
  sum by (le) (
    rate(kong_kong_latency_ms_bucket[5m])
  )
)

And p99:

histogram_quantile(
  0.99,
  sum by (le) (
    rate(kong_kong_latency_ms_bucket[5m])
  )
)

17. Add Azure OpenAI later

Once Ollama works, adding Azure OpenAI is easy.

The architecture becomes:

                       Kong AI Gateway
                              |
             +----------------+--------------+
             |                               |
             v                               v
          Ollama                       Azure OpenAI
          Qwen3                         GPT model
        On-premise                       Azure

Kong’s documented Azure configuration uses:

provider: azure
auth:
  header_name: api-key
model:
  name: gpt-4o
options:
  azure_instance: ...
  azure_deployment_id: ...
  azure_api_version: ...
``` :chatgpt-content-reference{index="7"}
---
# 18. Then introduce AI security controls
Your eventual plugin chain should look like:
```text
Client
  |
  v
F5
  |
  v
Kong
  |
  +-- OIDC / Ping Identity
  |
  +-- ACL / Consumer Group
  |
  +-- AI Prompt Guard
  |
  +-- AI Rate Limiting Advanced
  |
  +-- AI Semantic Cache
  |
  +-- AI Proxy Advanced
  |
  +-- Prometheus
  |
  +-- HTTP Log
  |
  v
AI Provider

Don’t add all of those on day one.

Use this order:

1  Kong
2  PostgreSQL
3  Ollama
4  AI Proxy Advanced
5  Validate AI request
6  Authentication
7  Prometheus/Grafana
8  Logging/ELK
9  Prompt Guard
10 AI Rate Limiting
11 Redis
12 Semantic Cache
13 Multiple LLM providers
14 MCP

19. Production architecture

After your single-node POC works, I would move to this:

                         Applications
                              |
                              |
                         +----v----+
                         |   F5    |
                         |   VIP   |
                         +----+----+
                              |
                    +---------+---------+
                    |                   |
                +---v---+           +---v---+
                | DP-01 |           | DP-02 |
                | Kong  |           | Kong  |
                +---+---+           +---+---+
                    |                   |
                    +---------+---------+
                              |
                 AI Providers / Models
                   /        |       \
                  /         |        \
             Ollama       vLLM     Azure OpenAI


          Control plane – NOT in request path

             +-----------+     +-----------+
             | Kong CP01 |     | Kong CP02 |
             +-----+-----+     +-----+-----+
                   \                 /
                    \               /
                     +------v------+
                     | PostgreSQL  |
                     +-------------+


                     +-------------+
                     |    Redis    |
                     +-------------+
                       /         \
                Rate limit    AI cache


DP01/DP02
   |
   +------ Prometheus ------ Grafana
   |
   +------ Filebeat -------- Logstash
                              |
                              v
                         Elasticsearch
                              |
                            Kibana

Kong hybrid mode explicitly separates the roles: the Control Plane uses PostgreSQL and distributes configuration, while the Data Planes run DB-less and proxy traffic. Kong Docs

I would target this configuration for production deployment:

2 x Kong Control Plane
4 vCPU
8–16 GB RAM
2 x Kong Data Plane
8 vCPU
16 GB RAM
1 x PostgreSQL HA service/cluster
1–3 x Redis nodes depending on HA requirements
2 x F5 pool members initially
Prometheus/Grafana
ELK
Ollama/vLLM GPU nodes if running models internally

The important design principle is:

Kong should not perform inference. Kong handles authentication, routing, governance, rate limits, caching, security, metrics and provider abstraction; Ollama/vLLM/Azure/OpenAI performs the actual model inference.

For your environment, I would therefore build the first working POC as:

Ubuntu 24.04 → PostgreSQL → Kong 3.16 → Ollama/Qwen → ai-proxy-advanced → key-auth → Prometheus/Grafana, and only after that convert it to F5 → 2 Kong DP → Ollama/Azure OpenAI with 2 Kong CP + PostgreSQL. Kong Docs

Kong’s AI capabilities ship as a set of plugins on top of the core Kong Gateway/Konnect data plane, rather than a separate product. The first six AI plugins shipped in Kong Gateway 3.6 (Feb 2024) as open source; more advanced governance plugins were added later and require Kong Enterprise or Konnect licensing.

Traffic proxying & routing

Load balancing algorithms (inside AI Proxy Advanced)

Configured via the Upstream entity, not separate plugins:

• Round-robin (weighted) — proportional traffic split by weight.
• Consistent-hashing — sticky sessions via a hashed header (default X-Kong-LLM-Request-ID).
• Lowest-latency — routes to whichever target is currently fastest.
• Lowest-usage — routes based on token/cost usage, including a cost option added to the usage strategy.
• Priority — tiered failover (what we configured for Anthropic → Bedrock → OpenAI).
• Semantic — routes prompts to domain-specialized models based on embedding similarity, with distance_metric and threshold controlling match sensitivity.
• 3.13+ added health checks and a circuit breaker for the load balancer itself, plus failover/circuit-breaking for semantic routing specifically.

Governance & safety

Cost & performance

Observability

• Prometheus plugin was extended with AI-specific metrics (token counts, cost, latency, MCP call volume) — this is what feeds the shared observability layer from the architecture summary.

Licensing note

Per Kong’s own docs and third-party comparisons: AI Proxy, AI Proxy Advanced, AI Prompt Guard, and AI Prompt Decorator are open source. AI Rate Limiting Advanced, AI Semantic Cache, AI Semantic Prompt Guard, and AI MCP Proxy sit behind Kong Enterprise/Konnect licensing. Confirm current tier gating against Kong’s plugin hub for your specific version, since this has shifted release to release.

Good — pulled the current plugin hub listing directly. Here’s the accurate, up-to-date table, tagged by license tier and mapped to the four pillars from the architecture doc.

AI Plugin Tier Table (Kong Plugin Hub, current)

Free / open source (no AI license required)

AI License Required (Enterprise/Konnect AI Gateway tier)

Third-party (separate vendor licensing)
CrowdStrike Falcon AIDR (Request/Response/MCP), Harness AI Security, Noma Runtime Protection, Prisma AIRS, Skyflow De-identify, Straiker (+ Coding Agent variants), Impart Security, VeriKnox.

Mapped to the four architecture pillars

One correction from earlier: I’d said AI Prompt Guard and AI Prompt Decorator might be gated — they’re actually free/open source. AI Proxy Advanced, AI Rate Limiting Advanced, AI Semantic Cache, AI Semantic Prompt Guard, AI MCP Proxy and AI MCP OAuth2 are the ones that need the AI license.

I’d also flag that Guardrails has grown into its own real pillar — with cloud-native integrations (AWS/Azure/GCP) plus third-party options (Lakera, CrowdStrike, Straiker) — separate from what I’d folded into “credential governance” earlier.