Understanding Ingress Limitations and Gateway API Benefits

techhadoop kubernetes

Ingress vs Gateway API

The Problem Ingress Has

Ingress was designed in the early days of Kubernetes and was never meant to handle the complexity of modern traffic management. Over time its limitations became painful.

Ingress limitations:
├── No native traffic splitting (canary needs annotations)
├── No TCP/UDP routing
├── Annotations are controller-specific (NGINX ≠ Traefik ≠ AGIC)
├── No role separation (devs and ops share same resource)
├── No header manipulation without annotations
└── Everything crammed into one resource

Gateway API was built to fix all of this.

Side-by-Side Architecture

INGRESS                              GATEWAY API
─────────────────────────────────    ─────────────────────────────────
                                     
┌─────────────────────────┐          ┌──────────────┐
│      Ingress            │          │GatewayClass  │ (cluster admin)
│  - host routing         │          │ which        │ defines controller
│  - path routing         │          │ controller   │
│  - TLS                  │          └──────┬───────┘
│  - annotations for      │                 │
│    everything else      │          ┌──────▼───────┐
│                         │          │   Gateway    │ (infra team)
│  Everything in          │          │ - listeners  │ defines ports,
│  ONE resource           │          │ - TLS        │ TLS, addresses
└─────────────────────────┘          └──────┬───────┘
                                            │
                                     ┌──────▼───────┐
                                     │  HTTPRoute   │ (app developer)
                                     │  TCPRoute    │ defines routing
                                     │  GRPCRoute   │ rules per service
                                     └──────────────┘

Role Separation (Biggest Difference)

Gateway API introduces a clear separation of concerns between personas:

INGRESS — everyone edits the same resource:
Cluster Admin ─┐
Infra Team    ─┼──▶ Ingress resource ◀─┬─ App Team A
Platform Team ─┘                        └─ App Team B
                   (conflicts, no boundaries)
GATEWAY API — each role owns their layer:
Cluster Admin ──▶ GatewayClass  (which controller to use)
                        │
Infra / Platform ──▶ Gateway    (ports, TLS, addresses)
                        │
App Team A ──▶ HTTPRoute        (routing rules for service A)
App Team B ──▶ HTTPRoute        (routing rules for service B)
# Cluster admin owns this — once
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: nginx
spec:
  controllerName: k8s.io/ingress-nginx
---
# Infra team owns this — defines the entry point
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: main-gateway
  namespace: infra
spec:
  gatewayClassName: nginx
  listeners:
  - name: https
    port: 443
    protocol: HTTPS
    tls:
      mode: Terminate
      certificateRefs:
      - name: wildcard-tls
    allowedRoutes:
      namespaces:
        from: Selector
        selector:
          matchLabels:
            gateway-access: "true"   # ← only these namespaces can attach
---
# App team owns this — in their own namespace
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-route
  namespace: team-a          # ← app team's namespace
spec:
  parentRefs:
  - name: main-gateway
    namespace: infra         # ← attaches to infra gateway
  hostnames:
  - api.acme.com
  rules:
  - backendRefs:
    - name: api-service
      port: 8080

Traffic Splitting

# INGRESS — needs controller-specific annotations
# NGINX example (won't work on Traefik or AGIC):
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"
spec:
  rules:
  - host: api.acme.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-v2          # canary target
            port:
              number: 80
# GATEWAY API — native, works on any conformant controller
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-canary
spec:
  parentRefs:
  - name: main-gateway
  hostnames:
  - api.acme.com
  rules:
  - backendRefs:
    - name: api-v1
      port: 8080
      weight: 90          # ← native weight, no annotations
    - name: api-v2
      port: 8080
      weight: 10

Header Manipulation

# INGRESS — annotation-based, controller-specific
metadata:
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      add_header X-Version "v2";
      proxy_set_header X-Real-IP $remote_addr;
# GATEWAY API — native, portable
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
spec:
  rules:
  - filters:
    - type: RequestHeaderModifier
      requestHeaderModifier:
        add:
        - name: X-Real-IP
          value: "client-ip"
        set:
        - name: X-Version
          value: "v2"
        remove:
        - X-Internal-Token
    - type: ResponseHeaderModifier
      responseHeaderModifier:
        add:
        - name: Strict-Transport-Security
          value: "max-age=31536000"
    backendRefs:
    - name: api-service
      port: 8080

URL Rewriting & Redirects

# INGRESS
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  rules:
  - host: acme.com
    http:
      paths:
      - path: /api(/|$)(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: api-service
            port:
              number: 80
# GATEWAY API — clean and readable
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
spec:
  rules:
  # Rewrite /api/v1/users → /users
  - matches:
    - path:
        type: PathPrefix
        value: /api/v1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          type: ReplacePrefixMatch
          replacePrefixMatch: /
    backendRefs:
    - name: api-service
      port: 8080
  # Redirect HTTP → HTTPS
  - matches:
    - path:
        type: PathPrefix
        value: /
    filters:
    - type: RequestRedirect
      requestRedirect:
        scheme: https
        statusCode: 301

TCP and gRPC Routing

# INGRESS — cannot do this natively
# GATEWAY API — TCPRoute (L4)
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TCPRoute
metadata:
  name: postgres-route
spec:
  parentRefs:
  - name: main-gateway
    sectionName: postgres-listener   # port 5432
  rules:
  - backendRefs:
    - name: postgres-service
      port: 5432
---
# GATEWAY API — GRPCRoute
apiVersion: gateway.networking.k8s.io/v1
kind: GRPCRoute
metadata:
  name: grpc-route
spec:
  parentRefs:
  - name: main-gateway
  hostnames:
  - grpc.acme.com
  rules:
  - matches:
    - method:
        service: acme.UserService
        method: GetUser
    backendRefs:
    - name: user-grpc-service
      port: 50051

Advanced Matching

# GATEWAY API — rich matching out of the box
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
spec:
  rules:
  # Match by header
  - matches:
    - headers:
      - name: X-User-Group
        value: beta
    backendRefs:
    - name: api-v2
      port: 8080
  # Match by query param
  - matches:
    - queryParams:
      - name: version
        value: "2"
    backendRefs:
    - name: api-v2
      port: 8080
  # Match by HTTP method
  - matches:
    - method: POST
      path:
        type: PathPrefix
        value: /api/events
    backendRefs:
    - name: events-service
      port: 8080
  # Default route
  - backendRefs:
    - name: api-v1
      port: 8080

TLS Configuration

# INGRESS
spec:
  tls:
  - hosts:
    - api.acme.com
    secretName: api-tls-secret
  rules:
  - host: api.acme.com
    ...
# GATEWAY API — TLS lives in Gateway (infra concern)
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
spec:
  listeners:
  # HTTPS with TLS termination
  - name: https
    port: 443
    protocol: HTTPS
    tls:
      mode: Terminate
      certificateRefs:
      - name: wildcard-tls
        namespace: infra
  # TLS passthrough — app handles TLS
  - name: tls-passthrough
    port: 8443
    protocol: TLS
    tls:
      mode: Passthrough     # ← forwards raw TLS to backend
  # mTLS
  - name: mtls
    port: 9443
    protocol: HTTPS
    tls:
      mode: Terminate
      options:
        gateway.nginx.org/client-cert-verification: "on"

Cross-Namespace Routing

# INGRESS — cannot natively route across namespaces
# GATEWAY API — ReferenceGrant allows cross-namespace routing
# Team B grants permission for Gateway in infra namespace to
# route to their service
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: allow-infra-gateway
  namespace: team-b           # ← in the target namespace
spec:
  from:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    namespace: infra           # ← allow routes from this namespace
  to:
  - group: ""
    kind: Service              # ← to reach Services in team-b
---
# Now team-b's service can be referenced cross-namespace
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: infra
spec:
  rules:
  - backendRefs:
    - name: team-b-service
      namespace: team-b        # ← cross-namespace reference
      port: 8080

Full Feature Comparison

FeatureIngressGateway API
HTTP routing✅ Basic✅ Advanced
HTTPS / TLS✅✅
TCP routing❌✅ TCPRoute
UDP routing❌✅ UDPRoute
gRPC routing❌✅ GRPCRoute
Traffic splitting⚠️ Annotations✅ Native weights
Header manipulation⚠️ Annotations✅ Native filters
URL rewrite⚠️ Annotations✅ Native
Redirect⚠️ Annotations✅ Native
Role separation❌✅ 3-tier model
Cross-namespace❌✅ ReferenceGrant
TLS passthrough⚠️ Controller-specific✅ Native
mTLS⚠️ Annotations✅ Native
Portability❌ Annotations vary✅ Standardized
Extensibility❌✅ Policy attachment

Policy Attachment (Gateway API Extension)

Gateway API supports attaching policies to any resource — a clean extensibility model:

# Timeout policy on a route
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-route
spec:
  rules:
  - backendRefs:
    - name: api-service
      port: 8080
    timeouts:
      request: 10s            # ← native timeout, no annotation
      backendRequest: 5s
---
# Rate limit policy (controller-specific extension)
apiVersion: gateway.nginx.org/v1alpha1
kind: ObservabilityPolicy
metadata:
  name: api-observability
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: api-route
  tracing:
    strategy: ratio
    ratio: 10

Migration Path: Ingress → Gateway API

Step 1: Install Gateway API CRDs
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/
        releases/download/v1.1.0/standard-install.yaml
Step 2: Create GatewayClass (once per cluster)
        Maps to your existing NGINX/Traefik controller
Step 3: Create Gateway (replaces controller Service config)
        Moves TLS config here
Step 4: Convert Ingress rules → HTTPRoutes
        One HTTPRoute per service/team
Step 5: Test HTTPRoutes alongside existing Ingress
        Both can run simultaneously
Step 6: Remove Ingress resources when validated
# Helpful tool — ingress2gateway converter
go install sigs.k8s.io/ingress2gateway@latest
ingress2gateway print \
  --input-file ingress.yaml \
  --providers nginx
# Outputs equivalent Gateway API resources

When to Use Each

Use Ingress if:
✅ Simple HTTP/HTTPS routing
✅ Single team manages everything
✅ Already heavily invested in Ingress annotations
✅ Your controller doesn't support Gateway API yet
✅ K8s version < 1.24
Use Gateway API if:
✅ Multiple teams share the cluster
✅ Need TCP / UDP / gRPC routing
✅ Need native traffic splitting (canary)
✅ Want controller-portable config
✅ Building new clusters from scratch
✅ K8s 1.24+ (stable support)
✅ Need fine-grained TLS control
✅ Enterprise / production-grade setup

Summary

INGRESS                         GATEWAY API
───────────────────────         ───────────────────────
One resource does everything    Three-tier role model
Annotations for everything      Native filters + rules
Controller-specific behaviour   Portable across controllers
HTTP only                       HTTP, TCP, UDP, gRPC
No role separation              GatewayClass → Gateway → Route
Stable, widely supported        Stable from K8s 1.28
Good for simple setups          Built for enterprise scale

Gateway API is the future of Kubernetes traffic management — Ingress is in maintenance mode. For new clusters and teams, Gateway API is the clear choice. For existing Ingress setups, migration is straightforward and the ingress2gateway tool makes it even easier.

Leave a Reply