Debugging ClusterIP Issues in OVN Kubernetes

Great—let’s go through another very common real-world issue and include a simple visual to make it click.


Scenario

Service works via pod IP, but fails via ClusterIP (service name/IP)

Environment:

  • frontend → calling backend
  • Direct call works:curl http://10.128.2.15:8080 ✅
  • Service call fails:curl http://backend-service ❌

What this means (important insight)

If pod IP works but service fails, then:

Pod networking (OVN routing) is working
Problem is in service load-balancing layer inside OVN-Kubernetes


Mental model (diagram)

Image

Interpretation:

  • Pod → Pod = direct routing (works)
  • Pod → Service = goes through OVN load balancer (broken here)

Step-by-step debugging

Step 1: Confirm endpoints exist

oc get endpoints backend-service

If EMPTY:

Root cause = wrong labels

Example:

# Service selector
selector:
app: backend

But pod has:

labels:
app: api ❌ mismatch

Fix labels → service starts working instantly


Step 2: Verify service definition

oc get svc backend-service -o yaml

Check:

  • correct port
  • correct targetPort

Common mistake:

port: 80
targetPort: 8080 ✅ must match container port

Step 3: Test ClusterIP directly

curl <ClusterIP>:<port>

Results:

  • ❌ fails → OVN load balancer issue
  • ✅ works → DNS issue instead

Step 4: Check DNS (don’t skip this)

From pod:

nslookup backend-service
If fails:

→ Not OVN
→ Check:

oc get pods -n openshift-dns

Step 5: Inspect OVN load balancer

On a node:

oc debug node/<node>
chroot /host

Then:

ovn-nbctl lb-list

You should see something like:

VIP: 172.30.0.10:80 → 10.128.2.15:8080

If missing:

OVN didn’t program the service


Step 6: Check OVN logs

oc logs -n openshift-ovn-kubernetes <ovnkube-master>

Look for:

  • load balancer sync errors
  • endpoint update failures

Step 7: Check kube-proxy replacement

In OpenShift Container Platform, OVN replaces kube-proxy.

So if service routing is broken:
It’s handled by OVN, not iptables


Real root causes (from production)

1. Label mismatch (MOST COMMON)

  • Service selector doesn’t match pod
    → no endpoints → service dead

2. Wrong port/targetPort

  • Service pointing to wrong container port
    → connection refused

3. OVN load balancer not programmed

  • OVN DB out of sync
    → ClusterIP has no backend mapping

4. NetworkPolicy blocking service traffic

  • Pod allows direct IP but blocks service path
    (less common but happens)

5. DNS issue (misdiagnosed often)

  • Service name fails, ClusterIP works

Fast debugging logic (this is gold)

When pod IP works but service fails:

  1. Endpoints exist?
    • ❌ → labels problem
  2. ClusterIP works?
    • ❌ → OVN load balancing
  3. DNS works?
    • ❌ → DNS issue

Pro tip (what experts do fast)

From a debug pod:

oc run debug --image=busybox -it --rm -- sh

Run:

nslookup backend-service
curl <ClusterIP>
curl <pod-IP>

This instantly isolates:

  • DNS
  • service
  • networking

Key takeaway

  • Pod IP = routing layer (OVN switching)
  • Service IP = OVN load balancer layer
  • If one works and the other doesn’t → you know exactly where to look

Troubleshooting Node-Specific Pod Traffic Failures

Scenario

Traffic works for pods on node A, but fails for pods on node B.

That usually points to a node-local OVN/OVS problem, not an app problem.

Example:

  • frontend on worker-1 can reach backend
  • same app on worker-2 cannot

That pattern is a huge clue.


How to debug it

1. Prove it’s node-specific

List pods and nodes:

oc get pods -A -o wide

Run the same network test from a pod on each node:

oc exec -it <good-pod> -- curl http://<target-pod-ip>:<port>
oc exec -it <bad-pod> -- curl http://<target-pod-ip>:<port>

If one node always works and another always fails, focus on the bad node.


2. Check the OVN pod on the bad node

Find the ovnkube-node pod for that worker:

oc get pods -n openshift-ovn-kubernetes -o wide

Look for the pod scheduled on the failing node.

Then inspect:

oc describe pod -n openshift-ovn-kubernetes <ovnkube-node-pod>
oc logs -n openshift-ovn-kubernetes <ovnkube-node-pod>

Things that matter:

  • restarts
  • readiness failures
  • DB connection errors
  • OVS/flow programming errors

If ovnkube-node is unhealthy there, that is often the root cause.


3. Check node readiness and basic health

oc get node
oc describe node <bad-node>

Look for:

  • NotReady
  • memory/disk pressure
  • network-related events

Sometimes OVN is fine and the node itself is degraded.


4. Inspect OVS on the bad node

Open a debug shell:

oc debug node/<bad-node>
chroot /host

Then:

ovs-vsctl show

You want to see expected bridges such as br-int.

Also useful:

ovs-ofctl dump-ports br-int
ovs-appctl bond/show

Red flags:

  • missing br-int
  • interfaces missing
  • counters not increasing on expected ports

If OVS is broken on that node, pod traffic there will fail even while the rest of the cluster looks fine.


5. Check the node’s host networking

Still on the node:

ip addr
ip route
ip link

Look for:

  • missing routes
  • down interfaces
  • wrong MTU

A node can have OVN running, but if the host interface or route is wrong, encapsulated traffic will still fail.


6. Compare MTU with a working node

MTU mismatches are sneaky.

On both a good node and bad node:

ip link

Look at the main NIC and OVN-related interfaces.

Symptoms of MTU trouble:

  • DNS works sometimes
  • small pings work
  • larger curls/higher-volume traffic fail or hang

A quick test from a pod can help:

ping -M do -s 1400 <target-ip>

If smaller packets work and larger ones fail, suspect MTU.


7. Check if pod wiring exists on the bad node

From the failing node’s ovnkube-node logs, check whether the affected pod sandbox/interface got programmed correctly.

Also inspect pods on that node:

oc get pods -A -o wide | grep <bad-node>

If all pods on that node fail, it is likely node OVN/OVS or host network.
If only one pod fails, it may be a pod-specific attachment/setup issue.


8. Test service vs direct pod IP

From a failing pod:

curl http://<target-pod-ip>:<port>
curl http://<service-cluster-ip>:<port>

Interpretation:

  • both fail → node/local OVN path likely broken
  • pod IP works, service fails → service/load-balancer programming problem
  • DNS name fails, ClusterIP works → DNS problem

This helps avoid blaming OVN for the wrong layer.


9. Check for node-local firewall or host changes

On the bad node, inspect whether something changed outside OpenShift:

iptables -S
nft list ruleset
systemctl status ovs-vswitchd
systemctl status ovn-controller

A manual host change, bad firewall rule, or failed service can break just one node.


10. Restart scope carefully

If evidence points clearly to the bad node’s OVN stack, a targeted recovery step is safer than broad cluster changes.

Typical sequence:

  • cordon/drain the bad node if workloads are impacted
  • restart or recover the bad node’s OVN/OVS components
  • verify traffic before uncordoning

Avoid random restarts cluster-wide unless you’ve ruled out a local issue.


What this usually turns out to be

Most common causes:

  • ovnkube-node unhealthy on one node
  • broken or stale OVS state on that node
  • host NIC / route / MTU mismatch
  • node-specific firewall or kernel/network issue
  • the node recently rebooted or partially lost connectivity to OVN DB

Fast triage checklist

When traffic fails only on one node, I’d do this in order:

oc get pods -A -o wide
oc get pods -n openshift-ovn-kubernetes -o wide
oc logs -n openshift-ovn-kubernetes <ovnkube-node-on-bad-node>
oc debug node/<bad-node>
chroot /host
ovs-vsctl show
ip route
ip link
systemctl status ovs-vswitchd
systemctl status ovn-controller

That usually gets you very close.


Mental model

When only one node is broken:

  • cluster-wide policy is less likely
  • app config is less likely
  • service config is less likely
  • node-local data plane is most likely

So think:
bad node → ovnkube-node → OVS → host NIC/route/MTU


Here’s a realistic example:

  • pods on worker-2 cannot reach anything off-node
  • pods on worker-1 are fine
  • ovnkube-node on worker-2 shows repeated connection/programming errors
  • ovs-vsctl show on worker-2 is missing expected state

That strongly suggests the fix is on worker-2, not in the app or service definitions.

Linux network

Linux network issues can come from a lot of places—misconfiguration, hardware problems, DNS failures, firewall rules, or even upstream network outages. The trick is to diagnose step-by-step instead of guessing. Here’s a practical workflow you can follow.


1. Check Basic Connectivity

Start simple—make sure the system is even connected.

ip a
  • Look for an IP address (not just 127.0.0.1)
  • Check if the interface is UP
ip link
  • Verify your interface (e.g., eth0, wlan0) is not DOWN

If it’s down:

sudo ip link set eth0 up

2. Test Network Reachability

Try pinging known addresses:

ping -c 4 8.8.8.8
  • If this fails → no internet or routing issue

Then test DNS:

ping -c 4 google.com
  • If IP ping works but domain fails → DNS problem

3. Check Routing Table

ip route

Look for a default route:

default via 192.168.1.1 dev eth0

If missing, add it:

sudo ip route add default via <gateway-ip>

4. Verify DNS Configuration

Check:

cat /etc/resolv.conf

You should see something like:

nameserver 8.8.8.8

If not, temporarily add:

sudo nano /etc/resolv.conf

Or if using systemd:

systemd-resolve --status

5. Check Network Manager / Services

Depending on your system:

Using NetworkManager

nmcli device status
nmcli connection show

Restart:

sudo systemctl restart NetworkManager

Using systemd-networkd

networkctl status
sudo systemctl restart systemd-networkd

6. Inspect Firewall Rules

Firewall can silently block traffic.

sudo iptables -L -n

Or if using UFW:

sudo ufw status

Temporarily disable (for testing):

sudo ufw disable

7. Check Open Ports

ss -tulnp

This shows:

  • Listening services
  • Ports in use

8. Test with traceroute

See where packets stop:

traceroute 8.8.8.8

If it stops early → local network issue
If it goes far but fails → ISP or remote issue


9. Check Logs

Logs often reveal the real problem:

journalctl -u NetworkManager
journalctl -xe
dmesg | grep -i network

10. Wireless-Specific Checks

If using Wi-Fi:

iwconfig

Scan networks:

iw dev wlan0 scan

Reconnect:

nmcli device wifi connect "SSID" password "PASSWORD"

11. Hardware / Driver Issues

Check drivers:

lspci | grep -i network
lsmod | grep <driver>

If missing drivers, you may need to install firmware packages.


Common Problems & Quick Fixes

  • No IP address → DHCP issue → restart network service
  • Can ping IP but not domain → DNS misconfigured
  • Interface down → enable with ip link set
  • Connected but no internet → gateway missing
  • Wi-Fi connected but slow → interference or driver issue

On Ubuntu, networking is usually managed by NetworkManager, so troubleshooting is a bit more standardized. Let’s narrow this down quickly with a focused Ubuntu workflow.


1. Check if NetworkManager is running

sudo systemctl status NetworkManager

If it’s not active:

sudo systemctl restart NetworkManager

2. See connection status

nmcli device status

You’ll see something like:

  • connected → good
  • disconnected → problem
  • unmanaged → config issue

To reconnect:

nmcli connection up <connection-name>

3. Check IP address

ip a

Look for something like:

inet 192.168.x.x

❌ No IP? Try renewing DHCP:

sudo dhclient -v

4. Test connectivity

ping -c 4 8.8.8.8
  • ❌ fails → network/gateway issue
  • ✅ works → move to DNS test:
ping -c 4 google.com
  • ❌ fails → DNS problem

5. Check default gateway

ip route

You should see:

default via 192.168.x.1

If missing:

sudo ip route add default via 192.168.1.1

6. Fix DNS (very common on Ubuntu)

Check:

cat /etc/resolv.conf

If it’s broken or empty, try:

sudo nano /etc/systemd/resolved.conf

Set:

DNS=8.8.8.8 1.1.1.1

Then restart:

sudo systemctl restart systemd-resolved

7. Wi-Fi specific (if applicable)

Check:

nmcli device wifi list

Reconnect:

nmcli device wifi connect "SSID" password "PASSWORD"

If Wi-Fi adapter missing:

lspci | grep -i network

8. Firewall check

Ubuntu often uses UFW:

sudo ufw status

Temporarily disable:

sudo ufw disable

9. Look at logs (very useful)

journalctl -u NetworkManager --no-pager | tail -50

Fast “reset everything” fix (works surprisingly often)

sudo systemctl restart NetworkManager
sudo systemctl restart systemd-resolved
sudo dhclient

Docker Image Optimization: Best Practices & Tips

Here’s the practical best-practice checklist for building Docker images today:

Use a small, trusted base image and pin versions instead of relying on latest. Docker recommends choosing the right base image, keeping it small, and pinning base image versions for better security and repeatability. (Docker Documentation)

Use multi-stage builds so build tools never end up in the final runtime image. This is one of Docker’s main recommendations for producing smaller, cleaner, more secure images. (Docker Documentation)

Keep the build context small with a .dockerignore file. Excluding node_modules, .git, test artifacts, local env files, and temp files speeds builds and reduces accidental leakage into the image. Docker explicitly recommends using .dockerignore. (Docker Documentation)

Design your Dockerfile to maximize cache reuse. Copy dependency files first, install dependencies, then copy the rest of the app. Since Docker images are layer-based, ordering instructions well can make rebuilds much faster. (Docker Documentation)

Do not install unnecessary packages. Keep the image focused on one service, and remove build-only tools from the final stage. Docker also recommends creating ephemeral containers and decoupling applications where possible. (Docker Documentation)

Run the app as a non-root user whenever possible. Docker’s learning materials call out that a production-ready Dockerfile should improve security by running as non-root. (Docker Documentation)

Rebuild images regularly and use fresh base layers, especially for security patches. Docker recommends rebuilding often and using flags like --pull and, when needed, --no-cache for clean rebuilds. Also build and test images in CI. (Docker Documentation)

A solid production pattern looks like this:

# syntax=docker/dockerfile:1
FROM node:22-alpine AS deps
WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
FROM node:22-alpine AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build
FROM node:22-alpine
WORKDIR /app
ENV NODE_ENV=production
COPY --from=deps /app/node_modules ./node_modules
COPY --from=build /app/dist ./dist
COPY package*.json ./
USER node
EXPOSE 3000
CMD ["node", "dist/server.js"]

A matching .dockerignore should usually include:

node_modules
npm-debug.log
.git
.gitignore
Dockerfile*
docker-compose*
.env
coverage
dist
tmp

For most teams, the simplest rule set is:

  1. Small pinned base image
  2. Multi-stage build
  3. .dockerignore
  4. Cache-friendly Dockerfile order
  5. Non-root runtime
  6. Rebuild in CI and scan often (Docker Documentation)

Absolutely — here’s a production-ready Docker image pattern you can reuse for most apps.

Good Dockerfile pattern

# syntax=docker/dockerfile:1
# 1) Install dependencies in a separate stage
FROM node:22-alpine AS deps
WORKDIR /app
COPY package*.json ./
RUN npm ci
# 2) Build the app
FROM node:22-alpine AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build
# 3) Runtime image
FROM node:22-alpine AS runtime
WORKDIR /app
ENV NODE_ENV=production
# Create/use non-root runtime
USER node
# Copy only what is needed at runtime
COPY --chown=node:node --from=deps /app/node_modules ./node_modules
COPY --chown=node:node --from=build /app/dist ./dist
COPY --chown=node:node package*.json ./
EXPOSE 3000
CMD ["node", "dist/server.js"]

Matching .dockerignore

node_modules
npm-debug.log
.git
.gitignore
Dockerfile*
docker-compose*
.env
.env.*
coverage
dist
tmp
.vscode
.idea

Why this is a strong default

Docker’s current guidance recommends:

  • multi-stage builds to keep the final image smaller and cleaner (Docker Documentation)
  • using a .dockerignore file to keep the build context small and avoid sending unnecessary files to the builder (Docker Documentation)
  • structuring the Dockerfile for better cache reuse, like copying dependency manifests before app source (Docker Documentation)
  • running the app as a non-root user in production images (Docker Documentation)
  • avoiding secrets in ARG or ENV; Docker recommends using secret mounts instead because build args and env vars can be exposed in image metadata or the final image (Docker Documentation)

Even better build command

docker build --pull -t myapp:latest .

--pull helps refresh the base image layers so you don’t keep building on stale images, which aligns with Docker’s recommendation to rebuild often and keep base layers fresh. (Docker Documentation)

7 rules to follow every time

  1. Pin the base image
FROM node:22.14-alpine
  1. Do not use latest in production
  2. Copy dependency files first
COPY package*.json ./
RUN npm ci
COPY . .
  1. Only copy runtime artifacts into the final stage
  2. Run as non-root
  3. Keep secrets out of the Dockerfile
  4. Keep one main responsibility per container when possible (Docker Documentation)

Common mistakes

Bad:

COPY . .
RUN npm install

Better:

COPY package*.json ./
RUN npm ci
COPY . .

Bad:

FROM node:latest

Better:

FROM node:22-alpine

Bad:

ENV API_KEY=secret123

Better: pass secrets at runtime or use Docker build secrets. (Docker Documentation)

If your app does not need Node at runtime

For frontend apps like React/Vite/Angular/Vue, it is often better to build in Node and serve with Nginx in the final stage, which Docker’s current framework guides also demonstrate for modern frontend apps. (Docker Documentation)

Best-practice summary

Use:

  • small pinned base image
  • multi-stage build
  • .dockerignore
  • cache-friendly layer order
  • non-root runtime
  • no secrets in ARG or ENV
  • regular rebuilds with fresh base layers (Docker Documentation)

to build a project (code + config) – production ready

Here’s the production version of the starter project: real domain, automatic HTTPS, HTTP→HTTPS redirect, and a secured Traefik dashboard.

This uses Traefik’s Docker provider with labels for routing, a Let’s Encrypt certificate resolver for TLS, and the dashboard in secure mode rather than api.insecure=true. Traefik’s docs recommend securing the dashboard and show Docker Compose setups for HTTPS with ACME. (Traefik Docs)

Before you start

You need:

  • a Linux server with Docker and Docker Compose
  • a domain or subdomain pointing to that server
  • ports 80 and 443 open to the internet

For the HTTP-01 challenge, Traefik’s ACME guide requires the app to be reachable publicly and the domain to point to the Traefik instance. (Traefik Docs)


Recommended structure

devops-starter/
├── app/
│ ├── package.json
│ └── server.js
├── letsencrypt/
│ └── acme.json
├── .github/
│ └── workflows/
│ └── publish.yml
├── .env
├── Dockerfile
└── compose.yml

1) app/package.json

{
  "name": "devops-starter",
  "version": "1.0.0",
  "description": "Node app behind Traefik with HTTPS",
  "main": "server.js",
  "scripts": {
    "start": "node server.js"
  },
  "license": "MIT"
}

2) app/server.js

const http = require("http");
const PORT = process.env.PORT || 3000;
const server = http.createServer((req, res) => {
if (req.url === "/healthz") {
res.writeHead(200, { "Content-Type": "application/json" });
return res.end(JSON.stringify({ ok: true }));
}
const body = {
ok: true,
message: "Hello from production",
method: req.method,
url: req.url,
hostname: req.headers.host,
time: new Date().toISOString()
};
res.writeHead(200, { "Content-Type": "application/json" });
res.end(JSON.stringify(body, null, 2));
});
server.listen(PORT, () => {
console.log(`Server running on port ${PORT}`);
});

3) Dockerfile

FROM node:20-alpine

WORKDIR /app

COPY app/package.json ./
RUN npm install --omit=dev

COPY app/server.js ./

ENV PORT=3000
EXPOSE 3000

CMD ["npm", "start"]


4) .env

Replace these with your real values:

DOMAIN=app.yourdomain.com
TRAEFIK_DASHBOARD_HOST=traefik.yourdomain.com
LETSENCRYPT_EMAIL=you@example.com

# Generate this with: htpasswd -nb admin 'your-strong-password'
# Then double the $ signs when putting it here for docker labels
TRAEFIK_BASIC_AUTH=admin:$$apr1$$replace$$with-real-hash

Traefik’s BasicAuth middleware supports htpasswd-style hashes, and its docs note that when using Docker labels, dollar signs need escaping. (Traefik Docs)


5) Create the certificate storage file

Run this once on the server:

mkdir -p letsencrypt
touch letsencrypt/acme.json
chmod 600 letsencrypt/acme.json

Traefik’s ACME examples store certificates in acme.json, and the file should be writable by Traefik while remaining protected. (Traefik Docs)


6) compose.yml

services:
  traefik:
    image: traefik:v3.4
    restart: unless-stopped
    command:
      - "--api.dashboard=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"

      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"

      - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"

      - "--certificatesresolvers.le.acme.email=${LETSENCRYPT_EMAIL}"
      - "--certificatesresolvers.le.acme.storage=/letsencrypt/acme.json"
      - "--certificatesresolvers.le.acme.httpchallenge=true"
      - "--certificatesresolvers.le.acme.httpchallenge.entrypoint=web"

      - "--accesslog=true"
      - "--log.level=INFO"

    ports:
      - "80:80"
      - "443:443"

    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "./letsencrypt:/letsencrypt"

    labels:
      - "traefik.enable=true"

      # Secure dashboard
      - "traefik.http.routers.dashboard.rule=Host(`${TRAEFIK_DASHBOARD_HOST}`)"
      - "traefik.http.routers.dashboard.entrypoints=websecure"
      - "traefik.http.routers.dashboard.tls=true"
      - "traefik.http.routers.dashboard.tls.certresolver=le"
      - "traefik.http.routers.dashboard.service=api@internal"
      - "traefik.http.routers.dashboard.middlewares=dashboard-auth"
      - "traefik.http.middlewares.dashboard-auth.basicauth.users=${TRAEFIK_BASIC_AUTH}"

  app:
    build:
      context: .
      dockerfile: Dockerfile
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.app.rule=Host(`${DOMAIN}`)"
      - "traefik.http.routers.app.entrypoints=websecure"
      - "traefik.http.routers.app.tls=true"
      - "traefik.http.routers.app.tls.certresolver=le"

      # Tell Traefik which internal port the app listens on
      - "traefik.http.services.app.loadbalancer.server.port=3000"

Why these labels and flags matter:

  • Traefik uses Docker labels as dynamic config when Docker is the provider. (Traefik Docs)
  • entrypoints.web and entrypoints.websecure define listeners on ports 80 and 443. (Traefik Docs)
  • The web entrypoint redirects all traffic to websecure, which is the standard Traefik redirect pattern. (Traefik Docs)
  • tls.certresolver=le tells the router to request and renew certificates through the Let’s Encrypt resolver you defined. (Traefik Docs)
  • The dashboard can be exposed securely through api@internal and protected with BasicAuth instead of insecure mode. (Traefik Docs)

7) DNS records

Create DNS records like:

  • A app.yourdomain.com -> your_server_ip
  • A traefik.yourdomain.com -> your_server_ip

If you use IPv6, add AAAA records too. The names used in your router Host(...) rules must resolve to the server running Traefik for ACME issuance to work. (Traefik Docs)


8) First deploy

From the project folder on your server:

docker compose up -d --build

Then open:

  • https://app.yourdomain.com
  • https://traefik.yourdomain.com

On first startup, Traefik should obtain certificates automatically via Let’s Encrypt as requests arrive for matching routers using the resolver. (Traefik Docs)

Useful commands:

docker compose logs -f traefik
docker compose logs -f app
docker compose ps


9) Generate the dashboard password hash

If htpasswd is installed:

htpasswd -nb admin 'your-strong-password'

Put the result in .env as TRAEFIK_BASIC_AUTH=..., but replace every $ with $$ so Docker Compose does not treat them as variable substitutions. Traefik’s BasicAuth docs explicitly mention escaping dollar signs in Docker label contexts. (Traefik Docs)


10) Publish the image from GitHub Actions

If you want Actions to build and push your app image to GHCR, use this workflow.

.github/workflows/publish.yml

name: Build and publish image
on:
push:
branches: ["main"]
env:
IMAGE_NAME: ghcr.io/${{ github.repository_owner }}/devops-starter
jobs:
publish:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- name: Check out repository
uses: actions/checkout@v4
- name: Log in to GHCR
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Set up Buildx
uses: docker/setup-buildx-action@v3
- name: Extract metadata
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.IMAGE_NAME }}
tags: |
type=raw,value=latest
type=sha
- name: Build and push
uses: docker/build-push-action@v6
with:
context: .
file: ./Dockerfile
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}

GitHub’s docs recommend docker/build-push-action for building and publishing images, and GHCR uses the Container registry at ghcr.io. Workflows can authenticate with GITHUB_TOKEN when package permissions are configured appropriately. (GitHub Docs)

If you switch to pulling the published image on the server, replace the app service in Compose with:

  app:
    image: ghcr.io/YOUR_GITHUB_USERNAME/devops-starter:latest
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.app.rule=Host(`${DOMAIN}`)"
      - "traefik.http.routers.app.entrypoints=websecure"
      - "traefik.http.routers.app.tls=true"
      - "traefik.http.routers.app.tls.certresolver=le"
      - "traefik.http.services.app.loadbalancer.server.port=3000"


11) What makes this “production enough” for a first real project

This version is much closer to a real deployment because it has:

  • automatic HTTPS
  • secure dashboard access
  • HTTP→HTTPS redirect
  • restart policy
  • access logs
  • a health endpoint
  • optional CI image publishing

Those pieces line up with Traefik’s Docker standalone guidance and dashboard/ACME docs. (Traefik Docs)


12) Common failure points

If it does not work, the usual causes are:

  • DNS not pointing at the server
  • ports 80/443 blocked by firewall or cloud security group
  • invalid TRAEFIK_BASIC_AUTH hash formatting
  • acme.json missing or wrong permissions
  • router hostnames not matching the browser request

Those are the most common things that prevent Traefik from issuing certs or matching routers in Docker setups. (Traefik Docs)


13) Resume line for this project

Deployed a containerized Node.js service behind Traefik with automatic Let’s Encrypt TLS, secure reverse-proxy routing, and GitHub Actions image publishing to GHCR.

That is solid, real DevOps experience.

step-by-step to build a project (code + config)

here’s a copyable starter project you can build end to end.

It gives you:

  • a tiny Node app in Docker
  • Traefik in front of it
  • hostname-based routing on localhost
  • a GitHub Actions workflow that builds the image
  • a path to deploy the same stack to a server later

This matches Traefik’s current Docker provider pattern, where Traefik watches Docker and picks up routing config from container labels. (Traefik Labs Documentation)


Project structure

devops-starter/
├── app/
│ ├── package.json
│ └── server.js
├── .github/
│ └── workflows/
│ └── docker.yml
├── .dockerignore
├── Dockerfile
└── compose.yml

1) app/package.json

{
"name": "devops-starter",
"version": "1.0.0",
"description": "Simple Node app behind Traefik",
"main": "server.js",
"scripts": {
"start": "node server.js"
},
"license": "MIT"
}

2) app/server.js

const http = require("http");
const PORT = process.env.PORT || 3000;
const server = http.createServer((req, res) => {
const body = {
ok: true,
message: "Hello from the app behind Traefik",
method: req.method,
url: req.url,
hostname: req.headers.host,
time: new Date().toISOString()
};
res.writeHead(200, { "Content-Type": "application/json" });
res.end(JSON.stringify(body, null, 2));
});
server.listen(PORT, () => {
console.log(`Server running on port ${PORT}`);
});

3) Dockerfile

This uses an official Node image and a fixed major version tag, which is in line with GitHub’s Dockerfile guidance. (GitHub Docs)

FROM node:20-alpine
WORKDIR /app
COPY app/package.json ./
RUN npm install --omit=dev
COPY app/server.js ./
ENV PORT=3000
EXPOSE 3000
CMD ["npm", "start"]

4) .dockerignore

node_modules
npm-debug.log
.git
.github
Dockerfile
compose.yml

5) compose.yml

This follows the same core idea as Traefik’s Docker Compose examples: enable the Docker provider, disable exposing containers by default, define an HTTP entrypoint, and add labels to the app container so Traefik creates the router automatically. (Traefik Labs Documentation)

services:
traefik:
image: traefik:v3.0
command:
- "--api.insecure=true"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--entryPoints.web.address=:80"
ports:
- "80:80"
- "8080:8080"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
app:
build:
context: .
dockerfile: Dockerfile
labels:
- "traefik.enable=true"
- "traefik.http.routers.app.rule=Host(`app.localhost`)"
- "traefik.http.routers.app.entrypoints=web"

A couple of notes:

  • api.insecure=true is fine for learning locally, but not for a public server. Traefik’s dashboard docs treat this as something to secure for real deployments. (Traefik Labs Documentation)
  • Because both services are in the same Compose stack, Docker networking handles connectivity between Traefik and the app. That is the same pattern used in Docker and Traefik quick-start examples. (Traefik Labs Documentation)

6) .github/workflows/docker.yml

GitHub’s docs show Docker builds in Actions using actions/checkout and docker/build-push-action. This workflow keeps it simple: it builds on every push to main, and you can later extend it to push to Docker Hub or GHCR. (GitHub Docs)

name: Build Docker image
on:
push:
branches: ["main"]
pull_request:
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Check out repo
uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build image
uses: docker/build-push-action@v6
with:
context: .
file: ./Dockerfile
push: false
tags: devops-starter:latest

7) Run it locally

From the project root:

docker compose up -d --build

Then open:

  • http://app.localhost
  • http://localhost:8080 for the Traefik dashboard

Traefik’s Docker quick-start uses the same localhost-style host rule pattern, and the dashboard is commonly exposed on port 8080 in the getting-started setup. (Traefik Labs Documentation)

To stop:

docker compose down

To view logs:

docker compose logs -f

8) What’s happening

When you visit http://app.localhost:

  1. your browser sends a request to port 80
  2. Traefik receives it
  3. Traefik checks Docker-discovered labels
  4. the router rule Host(\app.localhost`)` matches
  5. Traefik forwards the request to the app container

That “dynamic config from Docker labels” model is a central part of Traefik’s configuration overview and Docker provider docs. (Traefik Labs Documentation)


9) Make it feel more real

Add a second app to prove routing works.

Update compose.yml like this:

services:
traefik:
image: traefik:v3.0
command:
- "--api.insecure=true"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--entryPoints.web.address=:80"
ports:
- "80:80"
- "8080:8080"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
app:
build:
context: .
dockerfile: Dockerfile
labels:
- "traefik.enable=true"
- "traefik.http.routers.app.rule=Host(`app.localhost`)"
- "traefik.http.routers.app.entrypoints=web"
whoami:
image: traefik/whoami
labels:
- "traefik.enable=true"
- "traefik.http.routers.whoami.rule=Host(`whoami.localhost`)"
- "traefik.http.routers.whoami.entrypoints=web"

Then:

  • http://app.localhost → your app
  • http://whoami.localhost → sample Traefik test service

That mirrors Traefik’s own examples for exposing services with Docker labels. (Traefik Labs Documentation)


10) How to deploy this later

For a simple first deployment:

  • get a Linux VM
  • install Docker and Docker Compose
  • copy this project to the server
  • point a domain at the server IP
  • swap localhost routing for your real domain
  • add HTTPS with Traefik + Let’s Encrypt

Traefik documents Docker standalone setup, HTTPS entrypoints, and ACME/Let’s Encrypt support as part of its normal production path. (Traefik Labs Documentation)

Your production router label would look more like:

- "traefik.http.routers.app.rule=Host(`app.yourdomain.com`)"

11) Resume-worthy version of this project

Once this is live, you can honestly describe it like this:

Built and deployed a containerized Node.js service using Docker and Traefik with hostname-based routing and automated image builds via GitHub Actions.

That is a real DevOps project, not tutorial-only practice.


12) Best next upgrades

After this works, do these in order:

  1. add /healthz endpoint
  2. add a test job to GitHub Actions
  3. push built images to GHCR or Docker Hub
  4. deploy on a small cloud VM
  5. add HTTPS with Let’s Encrypt
  6. add Prometheus/Grafana later

GitHub’s Actions docs already provide the build-and-publish direction if you want to turn your build-only workflow into a registry-pushing workflow. (GitHub Docs)


13) The shortest possible checklist

Create files → run:

docker compose up -d --build

Visit:

http://app.localhost
http://localhost:8080

Push to GitHub → Actions builds the image definition automatically.


Traefik

Traefik is an open-source reverse proxy and load balancer designed for modern cloud-native applications—especially those running in containers.


What that actually means

Think of Traefik as a smart traffic controller sitting in front of your apps:

  • It receives incoming requests (like someone visiting your website)
  • Then routes them to the correct service (e.g., your API, frontend, or another container)
  • It can also balance traffic across multiple instances of the same service

Key features

Automatic service discovery

Image
Image

Traefik integrates directly with tools like:

  • Docker
  • Kubernetes

It automatically detects new containers/services and routes traffic to them—no manual config needed.


Built-in HTTPS (SSL/TLS)

  • Automatically generates and renews certificates using Let’s Encrypt
  • Handles HTTPS setup for you (no manual certificate management)

Load balancing

  • Distributes requests across multiple instances
  • Helps keep your app fast and available

Dynamic configuration

  • Updates routes in real time when services start/stop
  • No restarts required

Dashboard & monitoring

  • Web UI shows routes, services, and traffic
  • Useful for debugging and observability

Simple example (Docker)

If you run a container with labels like:

labels:
- "traefik.http.routers.myapp.rule=Host(`myapp.local`)"

Traefik will:

  • Detect the container
  • Create a route for myapp.local
  • Start sending traffic there automatically

When people use Traefik

  • Microservices architectures
  • Docker or Kubernetes setups
  • Hosting multiple apps on one server
  • Replacing tools like:
    • NGINX
    • HAProxy

In one sentence

Traefik is a modern, automatic reverse proxy that makes routing traffic to containerized apps simple and dynamic.


Here’s a small working Docker Compose example you can run locally. It follows Traefik’s current Docker quick-start pattern: Traefik listens to Docker, exposes port 80 for app traffic and 8080 for the dashboard, and routes a sample whoami container using labels. (Traefik Docs)

version: "3.9"
services:
traefik:
image: traefik:v3.0
container_name: traefik
command:
- "--api.insecure=true"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--entryPoints.web.address=:80"
ports:
- "80:80"
- "8080:8080"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
whoami:
image: traefik/whoami
container_name: whoami
labels:
- "traefik.enable=true"
- "traefik.http.routers.whoami.rule=Host(`whoami.localhost`)"
- "traefik.http.routers.whoami.entrypoints=web"

Run it with:

docker compose up -d

Then open:

  • http://whoami.localhost → sample app
  • http://localhost:8080 → Traefik dashboard

That hostname rule is the key idea: Traefik reads the Docker labels and creates a router so requests for whoami.localhost go to the whoami container. Traefik hot-reloads this dynamic routing config from Docker without restarting. (Traefik Docs)

How to read the important lines:

  • --providers.docker=true tells Traefik to watch Docker for containers/services. (Traefik Docs)
  • --providers.docker.exposedbydefault=false means only containers with traefik.enable=true get exposed. (Traefik Docs)
  • --entryPoints.web.address=:80 creates an HTTP entrypoint on port 80. (Traefik Docs)
  • traefik.http.routers.whoami.rule=Host(\whoami.localhost`)` matches incoming requests by hostname. (Traefik Docs)

A more realistic example is routing two apps:

version: "3.9"
services:
traefik:
image: traefik:v3.0
command:
- "--api.insecure=true"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--entryPoints.web.address=:80"
ports:
- "80:80"
- "8080:8080"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
app1:
image: traefik/whoami
labels:
- "traefik.enable=true"
- "traefik.http.routers.app1.rule=Host(`app1.localhost`)"
- "traefik.http.routers.app1.entrypoints=web"
app2:
image: traefik/whoami
labels:
- "traefik.enable=true"
- "traefik.http.routers.app2.rule=Host(`app2.localhost`)"
- "traefik.http.routers.app2.entrypoints=web"

Then:

  • http://app1.localhost → app1
  • http://app2.localhost → app2

That is basically the Traefik workflow: define a service, add labels, and Traefik discovers it automatically. The official docs also note that when both containers are in the same Compose file, Docker’s default network is enough for Traefik to reach them. (Traefik Docs)

A couple of useful notes:

  • The dashboard setting shown here is insecure and meant for local learning, not production. (Traefik Docs)
  • For production, people usually add TLS/HTTPS and often Let’s Encrypt certificate automation through Traefik’s config. The routing and certificates are part of Traefik’s dynamic config model. (Traefik Docs)

Want me to show the same thing with HTTPS + Let’s Encrypt, or a Kubernetes Ingress example?

Here’s a real HTTPS + Let’s Encrypt Docker Compose setup for Traefik.

It uses the HTTP-01 challenge, which means your server must be publicly reachable on ports 80 and 443, and your domain’s DNS must point at that server. Traefik’s docs also note that certificate resolvers are defined in static config, the router must have TLS enabled, and the router references the resolver by name. (Traefik Docs)

version: "3.9"
services:
traefik:
image: traefik:v3.4
container_name: traefik
command:
- "--api.dashboard=true"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
# entrypoints
- "--entrypoints.web.address=:80"
- "--entrypoints.websecure.address=:443"
# redirect http -> https
- "--entrypoints.web.http.redirections.entrypoint.to=websecure"
- "--entrypoints.web.http.redirections.entrypoint.scheme=https"
# lets encrypt
- "--certificatesresolvers.le.acme.email=you@example.com"
- "--certificatesresolvers.le.acme.storage=/letsencrypt/acme.json"
- "--certificatesresolvers.le.acme.httpchallenge=true"
- "--certificatesresolvers.le.acme.httpchallenge.entrypoint=web"
ports:
- "80:80"
- "443:443"
- "8080:8080" # optional dashboard
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
- "./letsencrypt:/letsencrypt"
app:
image: traefik/whoami
container_name: whoami
labels:
- "traefik.enable=true"
# router
- "traefik.http.routers.app.rule=Host(`whoami.yourdomain.com`)"
- "traefik.http.routers.app.entrypoints=websecure"
- "traefik.http.routers.app.tls=true"
- "traefik.http.routers.app.tls.certresolver=le"

Then run:

mkdir -p letsencrypt
touch letsencrypt/acme.json
chmod 600 letsencrypt/acme.json
docker compose up -d

Then open:

  • https://whoami.yourdomain.com
  • http://whoami.yourdomain.com will redirect to HTTPS

Why this works:

  • providers.docker=true makes Traefik watch Docker containers and labels. (Traefik Docs)
  • entrypoints.web and entrypoints.websecure create listeners on ports 80 and 443. (Traefik Docs)
  • certificatesresolvers.le.acme.* configures Let’s Encrypt and stores certs in acme.json. (Traefik Docs)
  • tls.certresolver=le tells that router to request and renew a cert through the le resolver. (Traefik Docs)

A couple of important gotchas:

  • Replace you@example.com with your real email.
  • Replace whoami.yourdomain.com with a real domain/subdomain you control.
  • That hostname needs an A or AAAA record pointing to your server. Traefik’s docs say ACME domains must point to Traefik. (Traefik Docs)
  • Port 8080 is just for the dashboard. Don’t leave it openly exposed on the internet unless you secure it. (Traefik Docs)

For Kubernetes, the usual Traefik-native route is an IngressRoute or standard Ingress; Traefik also documents a working Kubernetes + Let’s Encrypt example with CRDs. (Traefik Docs)

cert-manager in Kubernetes

cert-manager in Kubernetes is an add-on that automates TLS certificate management.

It helps you:

  • request certificates
  • renew them before they expire
  • store them as Kubernetes Secrets
  • connect them to Ingresses, gateways, and apps

Common uses:

  • Let’s Encrypt certificates for public apps
  • internal PKI certificates for cluster services
  • webhook, mTLS, and service cert automation

Core pieces:

  • Issuer / ClusterIssuer: tells cert-manager where certs come from
    • Issuer = namespaced
    • ClusterIssuer = cluster-wide
  • Certificate: declares the cert you want
  • Secret: where the cert and private key get stored
  • Challenge / Order: used mainly with ACME providers like Let’s Encrypt

Typical flow:

  1. You install cert-manager.
  2. Create an Issuer or ClusterIssuer.
  3. Create a Certificate resource or annotate an Ingress.
  4. cert-manager gets the cert and saves it into a Secret.
  5. Your Ingress/controller uses that Secret for HTTPS.

Very simple example with Let’s Encrypt:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
email: you@example.com
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- http01:
ingress:
class: nginx

Then a certificate:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: myapp-cert
namespace: default
spec:
secretName: myapp-tls
issuerRef:
name: letsencrypt-prod
kind: ClusterIssuer
dnsNames:
- example.com

Why people use it:

  • avoids manual certificate renewal
  • reduces downtime from expired certs
  • works natively with Kubernetes resources
  • supports public and private certificate authorities

Alright — let’s walk through cert-manager in Kubernetes step by step, focusing on the three things you asked for:


1. How to install cert-manager

The easiest and most reliable way is using Helm.

Add the Helm repo

helm repo add jetstack https://charts.jetstack.io
helm repo update

Install cert-manager (with CRDs)

helm install cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--set installCRDs=true

Verify installation

kubectl get pods -n cert-manager

You should see pods like:

  • cert-manager
  • cert-manager-webhook
  • cert-manager-cainjector

2. How cert-manager works with Ingress

This is the most common real-world setup.

Flow (important concept)

  1. You create an Ingress
  2. Add a special annotation → tells cert-manager to issue a cert
  3. cert-manager:
    • requests cert from Let’s Encrypt
    • completes challenge (HTTP-01 or DNS-01)
    • stores cert in a Secret
  4. Ingress uses that Secret → HTTPS works automatically

Example Ingress with cert-manager

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: myapp
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
tls:
- hosts:
- example.com
secretName: myapp-tls
rules:
- host: example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: myapp-service
port:
number: 80

Key things:

  • annotation → tells cert-manager which issuer to use
  • tls.secretName → where the cert will be stored
  • cert-manager fills that Secret automatically

3. Issuer vs ClusterIssuer vs Certificate

This is where people usually get confused — here’s the clean breakdown:


Issuer

  • Namespace-scoped
  • Only works inside one namespace
kind: Issuer
metadata:
name: my-issuer
namespace: default

Use when:

  • small apps
  • strict namespace isolation

ClusterIssuer

  • Cluster-wide
  • Can be used from any namespace
kind: ClusterIssuer
metadata:
name: letsencrypt-prod

Use when:

  • shared infrastructure
  • most production setups ✅

Certificate

  • Declares what cert you want
  • cert-manager turns this into a real cert
kind: Certificate
metadata:
name: my-cert
spec:
secretName: my-tls
dnsNames:
- example.com
issuerRef:
name: letsencrypt-prod
kind: ClusterIssuer

How they connect together

Think of it like this:

  • Issuer / ClusterIssuer → “Who gives me the cert?”
  • Certificate → “I want a cert for this domain”
  • Secret → “Here’s your actual cert + key”
  • Ingress → “Use this cert for HTTPS”

Typical production setup

Most teams use:

  • ClusterIssuer (Let’s Encrypt)
  • ✅ Ingress annotation (auto certs)
  • ❌ No manual Certificate objects (optional)

Bonus: HTTP-01 vs DNS-01

HTTP-01 (most common)

  • cert-manager creates a temporary endpoint
  • Let’s Encrypt verifies via HTTP
  • works with Ingress

DNS-01

  • adds a DNS record to prove ownership
  • needed for:
    • wildcard certs (*.example.com)
    • internal services

Great question — this is where cert-manager becomes really powerful.

At a high level:

👉 cert-manager = certificate lifecycle automation
👉 Service mesh (Istio / Linkerd) = uses certificates for mTLS between services

So cert-manager can act as the certificate authority (or CA manager) for your mesh.


Big picture: how they fit together

cert-manager → issues certificates
service mesh → uses them for mTLS
secure pod-to-pod communication

What mTLS in a service mesh actually means

In both Istio and Linkerd:

  • Every pod gets a certificate + private key
  • Pods authenticate each other using certs
  • Traffic is:
    • encrypted ✅
    • authenticated ✅
    • tamper-proof ✅

Option 1: Built-in CA (default behavior)

Istio / Linkerd by default:

  • run their own internal CA
  • automatically issue certs to pods
  • rotate certs

👉 This works out-of-the-box and is easiest.


Option 2: Using cert-manager as the CA

This is where integration happens.

Instead of mesh managing certs itself:

👉 cert-manager becomes the source of truth for certificates


Architecture with cert-manager

cert-manager
(Issuer / ClusterIssuer)
Mesh control plane (Istio / Linkerd)
Sidecars / proxies in pods

Istio + cert-manager

Default Istio:

  • uses istiod as CA

With cert-manager:

  • you replace Istio’s CA with:
    • cert-manager + external CA (Vault, Let’s Encrypt, internal PKI)

Common approach: Istio + cert-manager + external CA

cert-manager:

  • manages root/intermediate certs

Istio:

  • requests workload certs from that CA

Why do this?

  • centralized certificate management
  • enterprise PKI integration (e.g. HashiCorp Vault)
  • compliance requirements

Linkerd + cert-manager

Linkerd has cleaner native integration.

Linkerd actually recommends using cert-manager.


How it works:

  • cert-manager issues:
    • trust anchor (root cert)
    • issuer cert
  • Linkerd uses those to:
    • issue certs to proxies
    • rotate automatically

Example flow:

  1. Create a ClusterIssuer (e.g. self-signed or Vault)
  2. cert-manager generates:
    • root cert
    • intermediate cert
  3. Linkerd control plane uses them
  4. Sidecars get short-lived certs

Certificate lifecycle in mesh (with cert-manager)

  1. cert-manager creates CA certs
  2. mesh control plane uses them
  3. sidecars request short-lived certs
  4. certs rotate automatically

When to use cert-manager with a mesh

✅ Use cert-manager if:

  • you need custom CA / PKI
  • you want centralized certificate control
  • you’re integrating with:
    • Vault
    • enterprise PKI
  • compliance/security requirements

❌ Skip it if:

  • you just want simple mTLS
  • default mesh CA is enough

Important distinction

👉 cert-manager does NOT handle:

  • traffic encryption itself
  • service-to-service routing

👉 service mesh does NOT handle:

  • external certificate issuance (well)
  • complex PKI integrations (alone)

Simple mental model

  • cert-manager = certificate factory
  • Istio / Linkerd = security + traffic engine

Interview-style summary

If you need a sharp answer:

“cert-manager integrates with service meshes by acting as an external certificate authority. While Istio and Linkerd can issue certificates internally, cert-manager enables centralized PKI management, supports external CAs like Vault, and provides automated rotation, making it useful for production-grade mTLS setups.”


Here’s a real-world debugging checklist for cert-manager + service mesh / mTLS, organized in the order that usually finds the issue fastest.

1. Start with the symptom, not the YAML

First sort the failure into one of these buckets:

  • Certificate issuance problem: Secrets are missing, Certificate is not Ready, ACME challenges fail, or issuer/webhook errors appear. cert-manager’s troubleshooting flow centers on the Certificate, CertificateRequest, Order, and Challenge resources. (cert-manager)
  • Mesh identity / mTLS problem: certificates exist, but workloads still fail handshakes, sidecars can’t get identities, or mesh health checks fail. Istio and Linkerd both separate certificate management from runtime identity distribution. (Istio)

That split matters because cert-manager can be healthy while the mesh is broken, and vice versa. (cert-manager)

2. Confirm the control planes are healthy

Check the obvious first:

kubectl get pods -n cert-manager
kubectl get pods -n istio-system
kubectl get pods -n linkerd

For cert-manager, the important core components are the controller, webhook, and cainjector; webhook issues are a documented source of certificate failures. (cert-manager)

For Linkerd, run:

linkerd check

Linkerd’s official troubleshooting starts with linkerd check, and many identity and certificate problems show up there directly. (Linkerd)

For Istio, check control-plane health and then inspect config relevant to CA integration if you are using istio-csr or another external CA path. Istio’s cert-manager integration for workload certificates requires specific CA-server changes. (cert-manager)

3. Check the certificate objects before the Secrets

If cert-manager is involved, do this before anything else:

kubectl get certificate -A
kubectl describe certificate <name> -n <ns>
kubectl get certificaterequest -A
kubectl describe certificaterequest <name> -n <ns>

cert-manager’s own troubleshooting guidance points to these resources first because they expose the reason issuance or renewal failed. (cert-manager)

What you’re looking for:

  • Ready=False
  • issuer not found
  • permission denied
  • webhook validation errors
  • failed renewals
  • pending requests that never progress

If you’re using ACME, continue with:

kubectl get order,challenge -A
kubectl describe order <name> -n <ns>
kubectl describe challenge <name> -n <ns>

ACME failures are usually visible at the Order / Challenge level. (cert-manager)

4. Verify the issuer chain and secret contents

Typical failure pattern: the Secret exists, but it is the wrong Secret, wrong namespace, missing keys, or signed by the wrong CA.

Check:

kubectl get issuer,clusterissuer -A
kubectl describe issuer <name> -n <ns>
kubectl describe clusterissuer <name>
kubectl get secret <secret-name> -n <ns> -o yaml

For mesh-related certs, validate:

  • the Secret name matches what the mesh expects
  • the Secret is in the namespace the mesh component actually reads
  • the chain is correct
  • the certificate has not expired
  • the issuer/trust anchor relationship is the intended one

In Linkerd specifically, the trust anchor and issuer certificate are distinct, and Linkerd documents that workload certs rotate automatically but the control-plane issuer/trust-anchor credentials do not unless you set up rotation. (Linkerd)

5. Check expiration and rotation next

A lot of “random” mesh outages are just expired identity material.

For Linkerd, verify:

  • trust anchor validity
  • issuer certificate validity
  • whether rotation was automated or done manually

Linkerd’s docs are explicit that proxy workload certs rotate automatically, but issuer and trust anchor rotation require separate handling; expired root or issuer certs are a known failure mode. (Linkerd)

For Istio, if using a custom CA or Kubernetes CSR integration, verify the configured CA path and signing certs are still valid and match the active mesh configuration. (cert-manager)

6. If this is Istio, verify whether the mesh is using its built-in CA or an external one

This is a very common confusion point.

If you use cert-manager with Istio workloads, you are typically not just “adding cert-manager”; you are replacing or redirecting the CA flow, often through istio-csr or Kubernetes CSR integration. cert-manager’s Istio integration docs call out changes like disabling the built-in CA server and setting the CA address. (cert-manager)

So check:

  • Is istiod acting as CA, or is an external CA path configured?
  • Is caAddress pointing to the expected service?
  • If istio-csr is used, is it healthy and reachable?
  • Are workload cert requests actually reaching the intended signer?

If that split-brain exists, pods may get no certs or certs from the wrong signer. That is an inference from how Istio’s custom CA flow is wired. (cert-manager)

7. If this is Linkerd, run the identity checks early

For Linkerd, do not guess. Run:

linkerd check
linkerd check --proxy

The Linkerd troubleshooting docs center on linkerd check, and certificate / identity issues often surface there more quickly than raw Kubernetes inspection. (Linkerd)

Then look for:

  • identity component failures
  • issuer/trust-anchor mismatch
  • certificate expiration warnings
  • injected proxies missing identity

If linkerd check mentions expired identity material, go straight to issuer/trust-anchor rotation docs. (Linkerd)

8. Verify sidecar or proxy injection happened

If the pod is not meshed, mTLS debugging is a distraction.

Check:

kubectl get pod <pod> -n <ns> -o yaml

Look for the expected sidecar/proxy containers and mesh annotations. If they are absent, the issue is injection or policy, not certificate issuance. Istio and Linkerd both rely on the dataplane proxy to actually use workload identities for mTLS. (Istio)

9. Check policy mismatches after identities are confirmed

Once certificates and proxies look correct, inspect whether the traffic policy demands mTLS where the peer does not support it.

For Istio, check authentication policy objects such as PeerAuthentication and any destination-side expectations. Istio’s authentication docs cover how mTLS policy is applied. (Istio)

Classic symptom:

  • one side is strict mTLS
  • the other side is plaintext, outside mesh, or not injected

That usually produces handshake/reset errors even when cert-manager is completely fine. This is an inference from Istio’s mTLS policy model. (Istio)

10. Read the logs in this order

When the issue is still unclear, the best signal usually comes from logs in this order:

  1. cert-manager controller
  2. cert-manager webhook
  3. mesh identity/CA component (istiod, istio-csr, or Linkerd identity)
  4. the source and destination proxy containers

Use:

kubectl logs -n cert-manager deploy/cert-manager
kubectl logs -n cert-manager deploy/cert-manager-webhook
kubectl logs -n istio-system deploy/istiod
kubectl logs -n <istio-csr-namespace> deploy/istio-csr
kubectl logs -n linkerd deploy/linkerd-identity
kubectl logs <pod> -n <ns> -c <proxy-container>

cert-manager specifically documents webhook and issuance troubleshooting as core paths. Linkerd and Istio docs likewise center on their identity components for mesh cert issues. (cert-manager)

11. For ingress or gateway TLS, separate north-south from east-west

A lot of teams mix up:

  • ingress/gateway TLS
  • service-to-service mTLS

With Istio, cert-manager integration for gateways is straightforward and separate from workload identity. Istio’s docs show cert-manager managing gateway TLS credentials, while workload certificate management is handled through different CA mechanisms. (Istio)

So ask:

  • Is the failure only at ingress/gateway?
  • Or only pod-to-pod?
  • Or both?

If only ingress is broken, inspect the gateway Secret and gateway config, not mesh identity. (Istio)

12. Fast triage map

Use this shortcut:

  • Certificate not Ready → inspect CertificateRequest, Order, Challenge, issuer, webhook. (cert-manager)
  • Secret exists but mesh still fails → inspect trust chain, expiry, namespace, and mesh CA configuration. (cert-manager)
  • Linkerd only → run linkerd check, then inspect issuer/trust anchor status. (Linkerd)
  • Istio + cert-manager for workloads → verify external CA wiring, especially CA server disablement and caAddress. (cert-manager)
  • Handshake failures with healthy certs → inspect mesh policy and whether both endpoints are actually meshed. (Istio)

13. The three most common root causes

In practice, the big ones are:

  1. Expired or non-rotated issuer / trust anchor, especially in Linkerd. (Linkerd)
  2. Istio external CA miswiring, especially when using cert-manager for workloads rather than just gateway TLS. (cert-manager)
  3. Policy/injection mismatch, where strict mTLS is enabled but one side is not part of the mesh. (Istio)

14. Minimal command pack to keep handy

kubectl get certificate,certificaterequest,issuer,clusterissuer -A
kubectl describe certificate <name> -n <ns>
kubectl get order,challenge -A
kubectl logs -n cert-manager deploy/cert-manager
kubectl logs -n cert-manager deploy/cert-manager-webhook
linkerd check
linkerd check --proxy
kubectl logs -n istio-system deploy/istiod
kubectl get pods -A -o wide
kubectl get secret -A

Kong HA

Setting up Kong Gateway with high availability (HA) on-premise on bare metal servers involves several steps. Below is a comprehensive guide to achieve this setup:

Prerequisites

  1. Bare Metal Servers: Ensure you have multiple physical servers available.
  2. Network Configuration: Ensure all servers are on the same network and can communicate with each other.
  3. Data Store: Kong Gateway requires a shared data store like PostgreSQL or Cassandra. Ensure you have a highly available setup for your data store.
  4. Load Balancer: A hardware or software load balancer to distribute traffic across Kong Gateway nodes.

Step-by-Step Guide

1. Install PostgreSQL for the Shared Data Store

  1. Install PostgreSQL:

sudo apt-get update

sudo apt-get install -y postgresql postgresql-contrib

  1. Configure PostgreSQL for High Availability:
    • Set up replication between multiple PostgreSQL instances.
    • Ensure that the primary and standby instances are configured correctly.
  2. Create a Kong Database:

sudo -u postgres psql

CREATE DATABASE kong;

CREATE USER kong WITH PASSWORD ‘yourpassword’;

GRANT ALL PRIVILEGES ON DATABASE kong TO kong;

\q

2. Install Kong Gateway on Each Server

  1. Install Kong Gateway:

sudo apt-get update

sudo apt-get install -y apt-transport-https

curl -s https://packages.konghq.com/keys/kong.key | sudo apt-key add –

echo “deb https://packages.konghq.com/debian/ $(lsb_release -sc) main” | sudo tee -a /etc/apt/sources.list

sudo apt-get update

sudo apt-get install -y kong

  1. Configure Kong Gateway:
    • Create a kong.conf file on each server with the following configuration:

database = postgres

pg_host = <primary_postgresql_host>

pg_port = 5432

pg_user = kong

pg_password = yourpassword

pg_database = kong

  1. Start Kong Gateway:

kong migrations bootstrap

kong start

3. Configure Load Balancer

  1. Set Up a Load Balancer:
    • Configure your load balancer to distribute traffic across the Kong Gateway nodes.
    • Ensure the load balancer is set up for high availability (e.g., using a failover IP or DNS).
  2. Configure Health Checks:
    • Configure health checks on the load balancer to monitor the health of each Kong Gateway node.
    • Ensure that traffic is only sent to healthy nodes.

4. Set Up Failover Mechanism

  1. Database Failover:
    • Ensure your PostgreSQL setup has a failover mechanism in place (e.g., using Patroni or pgpool-II).
  2. Kong Gateway Failover:
    • Ensure that the load balancer can detect when a Kong Gateway node is down and redirect traffic to other nodes.

5. Implement Monitoring and Alerts

  1. Set Up Monitoring:
    • Use tools like Prometheus and Grafana to monitor the health and performance of your Kong Gateway nodes and PostgreSQL database.
  2. Set Up Alerts:
    • Configure alerts to notify you of any issues with the Kong Gateway nodes or the PostgreSQL database.

Example Configuration Files

PostgreSQL Configuration (pg_hba.conf):

# TYPE  DATABASE        USER            ADDRESS                 METHOD

host    kong            kong            192.168.1.0/24          md5

Kong Gateway Configuration (kong.conf):

database = postgres

pg_host = 192.168.1.10

pg_port = 5432

pg_user = kong

pg_password = yourpassword

pg_database = kong

Summary

By following these steps, you can set up a highly available Kong Gateway on bare metal servers. This setup ensures that your API gateway remains reliable and performs well under various conditions. Make sure to thoroughly test your setup to ensure that failover and load balancing work as expected.

initramfs

What is initramfs ?

initramfs stands for initial RAM filesystem. It plays a crucial role in the Linux boot process by providing a temporary root filesystem that is loaded into memory. This temporary root filesystem contains the necessary drivers, tools, and scripts needed to mount the real root filesystem and continue the boot process.

In simpler terms, it acts as a bridge between the bootloader and the main operating system, ensuring that the system has everything it needs to boot successfully.

Key Concepts of initramfs:

FeatureDescription
Temporary FilesystemIt’s loaded into memory as a temporary root filesystem.
Kernel ModulesContains drivers (kernel modules) required to access disks, filesystems, and other hardware.
ScriptsContains initialization scripts to prepare the system for booting the real root filesystem.
Critical FilesIncludes essential tools like mount, udev, bash, and libraries.

Key Functions of initramfs:

  1. Kernel Initialization: During the boot process, the Linux kernel loads the initramfs into memory.
  2. Loading Drivers: initramfs includes essential drivers needed to access hardware components, such as storage devices and filesystems.
  3. Mounting Root Filesystem: The primary function of initramfs is to mount the real root filesystem from a storage device (e.g., hard drive, SSD).
  4. Transitioning to Real Root: Once the real root filesystem is mounted, the initramfs transitions control to the system’s main init process, allowing the boot process to continue.

How initramfs Works:

  1. Bootloader Stage: The bootloader (e.g., GRUB) loads the Linux kernel and initramfs into memory.
  2. Kernel Stage: The kernel initializes and mounts the initramfs as the root filesystem.
  3. Init Stage: The init script or program within initramfs runs, performing tasks such as loading additional drivers, mounting filesystems, and locating the real root filesystem.
  4. Switch Root: The initramfs mounts the real root filesystem and switches control to it, allowing the system to boot normally.

Customizing initramfs:

You can customize the initramfs by including specific drivers, tools, and scripts. This is useful for scenarios where the default initramfs does not include the necessary components for your system.

Tools for Managing initramfs:

  • mkinitramfs: A tool to create initramfs images.
  • update-initramfs: A tool to update existing initramfs images.

Difference Between initramfs and initrd

Featureinitramfsinitrd
Formatcpio archive (compressed)Disk image (block device)
MountingExtracted directly into RAM as a rootfsMounted as a loop device
FlexibilityMore flexible and fasterLess flexible, older technology

Location of initramfs

On most Linux distributions, the initramfs file is located in the /boot directory:

ls /boot/initramfs-*.img

How to Rebuild initramfs

If you’ve made changes to the kernel, /etc/fstab, or storage configuration (e.g., LUKS, LVM), you may need to rebuild the initramfs.

Rebuild initramfs on RHEL/CentOS:

sudo dracut -f

Rebuild initramfs on Ubuntu/Debian:

sudo update-initramfs -u

Common Issues Related to initramfs

IssueCauseSolution
Dropped into initramfs shellKernel can’t find the root filesystemCheck /etc/fstab, rebuild initramfs, or fix missing drivers.
Boot failure after kernel updateMissing or corrupt initramfsRebuild initramfs.
Filesystem not mountingIncorrect or missing drivers in initramfsEnsure necessary drivers are included and rebuild.

By understanding how initramfs works, you can better appreciate its role in the Linux boot process and customize it to suit your needs.