cert-manager in Kubernetes

techhadoop , , , ,

cert-manager in Kubernetes is an add-on that automates TLS certificate management.

It helps you:

  • request certificates
  • renew them before they expire
  • store them as Kubernetes Secrets
  • connect them to Ingresses, gateways, and apps

Common uses:

  • Let’s Encrypt certificates for public apps
  • internal PKI certificates for cluster services
  • webhook, mTLS, and service cert automation

Core pieces:

  • Issuer / ClusterIssuer: tells cert-manager where certs come from
    • Issuer = namespaced
    • ClusterIssuer = cluster-wide
  • Certificate: declares the cert you want
  • Secret: where the cert and private key get stored
  • Challenge / Order: used mainly with ACME providers like Let’s Encrypt

Typical flow:

  1. You install cert-manager.
  2. Create an Issuer or ClusterIssuer.
  3. Create a Certificate resource or annotate an Ingress.
  4. cert-manager gets the cert and saves it into a Secret.
  5. Your Ingress/controller uses that Secret for HTTPS.

Very simple example with Let’s Encrypt:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: you@example.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: nginx

Then a certificate:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: myapp-cert
  namespace: default
spec:
  secretName: myapp-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
  - example.com

Why people use it:

  • avoids manual certificate renewal
  • reduces downtime from expired certs
  • works natively with Kubernetes resources
  • supports public and private certificate authorities

Alright — let’s walk through cert-manager in Kubernetes step by step, focusing on the three things you asked for:

1️⃣ How to install cert-manager

The easiest and most reliable way is using Helm.

Add the Helm repo

helm repo add jetstack https://charts.jetstack.io
helm repo update

Install cert-manager (with CRDs)

helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set installCRDs=true

Verify installation

kubectl get pods -n cert-manager

You should see pods like:

  • cert-manager
  • cert-manager-webhook
  • cert-manager-cainjector

2️⃣ How cert-manager works with Ingress

This is the most common real-world setup.

🔁 Flow (important concept)

  1. You create an Ingress
  2. Add a special annotation → tells cert-manager to issue a cert
  3. cert-manager:
    • requests cert from Let’s Encrypt
    • completes challenge (HTTP-01 or DNS-01)
    • stores cert in a Secret
  4. Ingress uses that Secret → HTTPS works automatically

🧩 Example Ingress with cert-manager

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
  - hosts:
    - example.com
    secretName: myapp-tls
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: myapp-service
            port:
              number: 80

👉 Key things:

  • annotation → tells cert-manager which issuer to use
  • tls.secretName → where the cert will be stored
  • cert-manager fills that Secret automatically

3️⃣ Issuer vs ClusterIssuer vs Certificate

This is where people usually get confused — here’s the clean breakdown:

Issuer

  • Namespace-scoped
  • Only works inside one namespace
kind: Issuer
metadata:
  name: my-issuer
  namespace: default

Use when:

  • small apps
  • strict namespace isolation

ClusterIssuer

  • Cluster-wide
  • Can be used from any namespace
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod

Use when:

  • shared infrastructure
  • most production setups ✅

🟣 Certificate

  • Declares what cert you want
  • cert-manager turns this into a real cert
kind: Certificate
metadata:
  name: my-cert
spec:
  secretName: my-tls
  dnsNames:
  - example.com
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer

How they connect together

Think of it like this:

  • Issuer / ClusterIssuer → “Who gives me the cert?”
  • Certificate → “I want a cert for this domain”
  • Secret → “Here’s your actual cert + key”
  • Ingress → “Use this cert for HTTPS”

Typical production setup

Most teams use:

  • ✅ ClusterIssuer (Let’s Encrypt)
  • ✅ Ingress annotation (auto certs)
  • ❌ No manual Certificate objects (optional)

Bonus: HTTP-01 vs DNS-01

HTTP-01 (most common)

  • cert-manager creates a temporary endpoint
  • Let’s Encrypt verifies via HTTP
  • works with Ingress

DNS-01

  • adds a DNS record to prove ownership
  • needed for:
    • wildcard certs (*.example.com)
    • internal services

Great question — this is where cert-manager becomes really powerful.

At a high level:

👉 cert-manager = certificate lifecycle automation
👉 Service mesh (Istio / Linkerd) = uses certificates for mTLS between services

So cert-manager can act as the certificate authority (or CA manager) for your mesh.

🧠 Big picture: how they fit together

cert-manager → issues certificates
        ↓
service mesh → uses them for mTLS
        ↓
secure pod-to-pod communication

🔐 What mTLS in a service mesh actually means

In both Istio and Linkerd:

  • Every pod gets a certificate + private key
  • Pods authenticate each other using certs
  • Traffic is:
    • encrypted ✅
    • authenticated ✅
    • tamper-proof ✅

⚙️ Option 1: Built-in CA (default behavior)

Istio / Linkerd by default:

  • run their own internal CA
  • automatically issue certs to pods
  • rotate certs

👉 This works out-of-the-box and is easiest.

🧩 Option 2: Using cert-manager as the CA

This is where integration happens.

Instead of mesh managing certs itself:

👉 cert-manager becomes the source of truth for certificates

🧱 Architecture with cert-manager

cert-manager
   ↓
(Issuer / ClusterIssuer)
   ↓
Mesh control plane (Istio / Linkerd)
   ↓
Sidecars / proxies in pods

🔵 Istio + cert-manager

Default Istio:

  • uses istiod as CA

With cert-manager:

  • you replace Istio’s CA with:
    • cert-manager + external CA (Vault, Let’s Encrypt, internal PKI)

Common approach: Istio + cert-manager + external CA

cert-manager:

  • manages root/intermediate certs

Istio:

  • requests workload certs from that CA

Why do this?

  • centralized certificate management
  • enterprise PKI integration (e.g. HashiCorp Vault)
  • compliance requirements

Linkerd + cert-manager

Linkerd has cleaner native integration.

👉 Linkerd actually recommends using cert-manager.

How it works:

  • cert-manager issues:
    • trust anchor (root cert)
    • issuer cert
  • Linkerd uses those to:
    • issue certs to proxies
    • rotate automatically

Example flow:

  1. Create a ClusterIssuer (e.g. self-signed or Vault)
  2. cert-manager generates:
    • root cert
    • intermediate cert
  3. Linkerd control plane uses them
  4. Sidecars get short-lived certs

🔁 Certificate lifecycle in mesh (with cert-manager)

  1. cert-manager creates CA certs
  2. mesh control plane uses them
  3. sidecars request short-lived certs
  4. certs rotate automatically

When to use cert-manager with a mesh

✅ Use cert-manager if:

  • you need custom CA / PKI
  • you want centralized certificate control
  • you’re integrating with:
    • Vault
    • enterprise PKI
  • compliance/security requirements

❌ Skip it if:

  • you just want simple mTLS
  • default mesh CA is enough

Important distinction

👉 cert-manager does NOT handle:

  • traffic encryption itself
  • service-to-service routing

👉 service mesh does NOT handle:

  • external certificate issuance (well)
  • complex PKI integrations (alone)

Simple mental model

  • cert-manager = certificate factory
  • Istio / Linkerd = security + traffic engine

Interview-style summary

If you need a sharp answer:

“cert-manager integrates with service meshes by acting as an external certificate authority. While Istio and Linkerd can issue certificates internally, cert-manager enables centralized PKI management, supports external CAs like Vault, and provides automated rotation, making it useful for production-grade mTLS setups.”

Here’s a real-world debugging checklist for cert-manager + service mesh / mTLS, organized in the order that usually finds the issue fastest.

1. Start with the symptom, not the YAML

First sort the failure into one of these buckets:

  • Certificate issuance problem: Secrets are missing, Certificate is not Ready, ACME challenges fail, or issuer/webhook errors appear. cert-manager’s troubleshooting flow centers on the Certificate, CertificateRequest, Order, and Challenge resources. (cert-manager)
  • Mesh identity / mTLS problem: certificates exist, but workloads still fail handshakes, sidecars can’t get identities, or mesh health checks fail. Istio and Linkerd both separate certificate management from runtime identity distribution. (Istio)

That split matters because cert-manager can be healthy while the mesh is broken, and vice versa. (cert-manager)

2. Confirm the control planes are healthy

Check the obvious first:

kubectl get pods -n cert-manager
kubectl get pods -n istio-system
kubectl get pods -n linkerd

For cert-manager, the important core components are the controller, webhook, and cainjector; webhook issues are a documented source of certificate failures. (cert-manager)

For Linkerd, run:

linkerd check

Linkerd’s official troubleshooting starts with linkerd check, and many identity and certificate problems show up there directly. (Linkerd)

For Istio, check control-plane health and then inspect config relevant to CA integration if you are using istio-csr or another external CA path. Istio’s cert-manager integration for workload certificates requires specific CA-server changes. (cert-manager)

3. Check the certificate objects before the Secrets

If cert-manager is involved, do this before anything else:

kubectl get certificate -A
kubectl describe certificate <name> -n <ns>
kubectl get certificaterequest -A
kubectl describe certificaterequest <name> -n <ns>

cert-manager’s own troubleshooting guidance points to these resources first because they expose the reason issuance or renewal failed. (cert-manager)

What you’re looking for:

  • Ready=False
  • issuer not found
  • permission denied
  • webhook validation errors
  • failed renewals
  • pending requests that never progress

If you’re using ACME, continue with:

kubectl get order,challenge -A
kubectl describe order <name> -n <ns>
kubectl describe challenge <name> -n <ns>

ACME failures are usually visible at the Order / Challenge level. (cert-manager)

4. Verify the issuer chain and secret contents

Typical failure pattern: the Secret exists, but it is the wrong Secret, wrong namespace, missing keys, or signed by the wrong CA.

Check:

kubectl get issuer,clusterissuer -A
kubectl describe issuer <name> -n <ns>
kubectl describe clusterissuer <name>
kubectl get secret <secret-name> -n <ns> -o yaml

For mesh-related certs, validate:

  • the Secret name matches what the mesh expects
  • the Secret is in the namespace the mesh component actually reads
  • the chain is correct
  • the certificate has not expired
  • the issuer/trust anchor relationship is the intended one

In Linkerd specifically, the trust anchor and issuer certificate are distinct, and Linkerd documents that workload certs rotate automatically but the control-plane issuer/trust-anchor credentials do not unless you set up rotation. (Linkerd)

5. Check expiration and rotation next

A lot of “random” mesh outages are just expired identity material.

For Linkerd, verify:

  • trust anchor validity
  • issuer certificate validity
  • whether rotation was automated or done manually

Linkerd’s docs are explicit that proxy workload certs rotate automatically, but issuer and trust anchor rotation require separate handling; expired root or issuer certs are a known failure mode. (Linkerd)

For Istio, if using a custom CA or Kubernetes CSR integration, verify the configured CA path and signing certs are still valid and match the active mesh configuration. (cert-manager)

6. If this is Istio, verify whether the mesh is using its built-in CA or an external one

This is a very common confusion point.

If you use cert-manager with Istio workloads, you are typically not just “adding cert-manager”; you are replacing or redirecting the CA flow, often through istio-csr or Kubernetes CSR integration. cert-manager’s Istio integration docs call out changes like disabling the built-in CA server and setting the CA address. (cert-manager)

So check:

  • Is istiod acting as CA, or is an external CA path configured?
  • Is caAddress pointing to the expected service?
  • If istio-csr is used, is it healthy and reachable?
  • Are workload cert requests actually reaching the intended signer?

If that split-brain exists, pods may get no certs or certs from the wrong signer. That is an inference from how Istio’s custom CA flow is wired. (cert-manager)

7. If this is Linkerd, run the identity checks early

For Linkerd, do not guess. Run:

linkerd check
linkerd check --proxy

The Linkerd troubleshooting docs center on linkerd check, and certificate / identity issues often surface there more quickly than raw Kubernetes inspection. (Linkerd)

Then look for:

  • identity component failures
  • issuer/trust-anchor mismatch
  • certificate expiration warnings
  • injected proxies missing identity

If linkerd check mentions expired identity material, go straight to issuer/trust-anchor rotation docs. (Linkerd)

8. Verify sidecar or proxy injection happened

If the pod is not meshed, mTLS debugging is a distraction.

Check:

kubectl get pod <pod> -n <ns> -o yaml

Look for the expected sidecar/proxy containers and mesh annotations. If they are absent, the issue is injection or policy, not certificate issuance. Istio and Linkerd both rely on the dataplane proxy to actually use workload identities for mTLS. (Istio)

9. Check policy mismatches after identities are confirmed

Once certificates and proxies look correct, inspect whether the traffic policy demands mTLS where the peer does not support it.

For Istio, check authentication policy objects such as PeerAuthentication and any destination-side expectations. Istio’s authentication docs cover how mTLS policy is applied. (Istio)

Classic symptom:

  • one side is strict mTLS
  • the other side is plaintext, outside mesh, or not injected

That usually produces handshake/reset errors even when cert-manager is completely fine. This is an inference from Istio’s mTLS policy model. (Istio)

10. Read the logs in this order

When the issue is still unclear, the best signal usually comes from logs in this order:

  1. cert-manager controller
  2. cert-manager webhook
  3. mesh identity/CA component (istiod, istio-csr, or Linkerd identity)
  4. the source and destination proxy containers

Use:

kubectl logs -n cert-manager deploy/cert-manager
kubectl logs -n cert-manager deploy/cert-manager-webhook
kubectl logs -n istio-system deploy/istiod
kubectl logs -n <istio-csr-namespace> deploy/istio-csr
kubectl logs -n linkerd deploy/linkerd-identity
kubectl logs <pod> -n <ns> -c <proxy-container>

cert-manager specifically documents webhook and issuance troubleshooting as core paths. Linkerd and Istio docs likewise center on their identity components for mesh cert issues. (cert-manager)

11. For ingress or gateway TLS, separate north-south from east-west

A lot of teams mix up:

  • ingress/gateway TLS
  • service-to-service mTLS

With Istio, cert-manager integration for gateways is straightforward and separate from workload identity. Istio’s docs show cert-manager managing gateway TLS credentials, while workload certificate management is handled through different CA mechanisms. (Istio)

So ask:

  • Is the failure only at ingress/gateway?
  • Or only pod-to-pod?
  • Or both?

If only ingress is broken, inspect the gateway Secret and gateway config, not mesh identity. (Istio)

12. Fast triage map

Use this shortcut:

  • Certificate not Ready → inspect CertificateRequest, Order, Challenge, issuer, webhook. (cert-manager)
  • Secret exists but mesh still fails → inspect trust chain, expiry, namespace, and mesh CA configuration. (cert-manager)
  • Linkerd only → run linkerd check, then inspect issuer/trust anchor status. (Linkerd)
  • Istio + cert-manager for workloads → verify external CA wiring, especially CA server disablement and caAddress. (cert-manager)
  • Handshake failures with healthy certs → inspect mesh policy and whether both endpoints are actually meshed. (Istio)

13. The three most common root causes

In practice, the big ones are:

  1. Expired or non-rotated issuer / trust anchor, especially in Linkerd. (Linkerd)
  2. Istio external CA miswiring, especially when using cert-manager for workloads rather than just gateway TLS. (cert-manager)
  3. Policy/injection mismatch, where strict mTLS is enabled but one side is not part of the mesh. (Istio)

14. Minimal command pack to keep handy

kubectl get certificate,certificaterequest,issuer,clusterissuer -A
kubectl describe certificate <name> -n <ns>
kubectl get order,challenge -A
kubectl logs -n cert-manager deploy/cert-manager
kubectl logs -n cert-manager deploy/cert-manager-webhook
linkerd check
linkerd check --proxy
kubectl logs -n istio-system deploy/istiod
kubectl get pods -A -o wide
kubectl get secret -A

Leave a comment