devops

Kong – full mini project folder

techhadoop kong, Uncategorized , , , ,

Here’s a full mini project folder for Kong that you can copy as-is.

It uses Kong Gateway in DB-less mode, so all config lives in one declarative kong.yml file. That mode is a good fit for CI/CD and Git-managed config, but the Admin API is effectively read-only for config changes in this setup. (Kong Docs)

Folder structure

kong-mini-project/
├── app/
│   ├── package.json
│   └── server.js
├── kong/
│   └── kong.yml
├── .dockerignore
├── Dockerfile
└── compose.yml

1) app/package.json

{
  "name": "kong-mini-project",
  "version": "1.0.0",
  "description": "Node app behind Kong Gateway",
  "main": "server.js",
  "scripts": {
    "start": "node server.js"
  },
  "license": "MIT"
}

2) app/server.js

const http = require("http");
const PORT = process.env.PORT || 3000;
const server = http.createServer((req, res) => {
  if (req.url === "/healthz") {
    res.writeHead(200, { "Content-Type": "application/json" });
    return res.end(JSON.stringify({ ok: true }));
  }
  const body = {
    ok: true,
    message: "Hello from app behind Kong",
    method: req.method,
    url: req.url,
    host: req.headers.host,
    time: new Date().toISOString()
  };
  res.writeHead(200, { "Content-Type": "application/json" });
  res.end(JSON.stringify(body, null, 2));
});
server.listen(PORT, () => {
  console.log(`Server listening on ${PORT}`);
});

3) Dockerfile

FROM node:20-alpine
WORKDIR /app
COPY app/package.json ./
RUN npm install --omit=dev
COPY app/server.js ./
ENV PORT=3000
EXPOSE 3000
CMD ["npm", "start"]

4) .dockerignore

node_modules
npm-debug.log
.git
.github

5) kong/kong.yml

This is the heart of the project. It defines:

  • one upstream Service
  • one public Route
  • a key-auth plugin
  • a rate-limiting plugin
  • one Consumer with an API key

Kong’s declarative config format supports entities like Services, Routes, Consumers, and Plugins in DB-less mode. The Key Auth plugin can require API keys, and the Rate Limiting plugin can throttle requests by time window such as per minute. When authentication is present, rate limiting uses the authenticated Consumer identity. (Kong Docs)

_format_version: "3.0"
services:
  - name: app-service
    url: http://app:3000
    routes:
      - name: app-route
        paths:
          - /api
        protocols:
          - http
          - https
plugins:
  - name: key-auth
    service: app-service
    config:
      key_names:
        - apikey
  - name: rate-limiting
    service: app-service
    config:
      minute: 5
      policy: local
consumers:
  - username: demo-client
    keyauth_credentials:
      - key: super-secret-demo-key

A note on policy: local: that works well for a single local node, but Kong notes that plugins needing shared database state do not fully function in DB-less mode, so this is best for learning or single-node setups rather than clustered distributed quotas. (Kong Docs)

6) compose.yml

Kong’s Docker docs support running Kong with Docker Compose, and the read-only Docker Compose guide for DB-less mode uses KONG_DATABASE=off plus KONG_DECLARATIVE_CONFIG pointing to the config file. (Kong Docs)

services:
  kong:
    image: kong:3.10
    environment:
      KONG_DATABASE: "off"
      KONG_DECLARATIVE_CONFIG: /kong/declarative/kong.yml
      KONG_PROXY_ACCESS_LOG: /dev/stdout
      KONG_ADMIN_ACCESS_LOG: /dev/stdout
      KONG_PROXY_ERROR_LOG: /dev/stderr
      KONG_ADMIN_ERROR_LOG: /dev/stderr
      KONG_ADMIN_LISTEN: 0.0.0.0:8001
    ports:
      - "8000:8000"   # public proxy
      - "8001:8001"   # admin api (read-only for config in DB-less mode)
    volumes:
      - ./kong/kong.yml:/kong/declarative/kong.yml:ro
  app:
    build:
      context: .
      dockerfile: Dockerfile

7) Run it

docker compose up -d --build

Then test it.

Without an API key, access should fail because the route is protected by the Key Auth plugin. (Kong Docs)

curl -i http://localhost:8000/api

With the API key in a header, it should succeed. Kong’s Key Auth plugin supports reading keys from headers, query parameters, or request body, depending on config. (Kong Docs)

curl -i \
  -H "apikey: super-secret-demo-key" \
  http://localhost:8000/api

You can also use a query string:

curl -i "http://localhost:8000/api?apikey=super-secret-demo-key"

8) Test rate limiting

The plugin is set to 5 requests per minute, so the sixth quick request should return 429. Kong’s rate-limiting plugin supports time windows including seconds, minutes, hours, days, months, and years. (Kong Docs)

for i in {1..6}; do
  curl -s -o /dev/null -w "%{http_code}\n" \
    -H "apikey: super-secret-demo-key" \
    http://localhost:8000/api
done

9) Useful checks

See running containers:

docker compose ps

Follow Kong logs:

docker compose logs -f kong

Follow app logs:

docker compose logs -f app

Read the service list from the Admin API:

curl http://localhost:8001/services

In DB-less mode, that Admin API is useful for inspection, but Kong’s docs say you cannot use it for normal write-based configuration management because the declarative file is the source of truth. (Kong Docs)

10) What makes this different from Traefik

With Traefik, the main workflow was “discover containers and route traffic to them.” With Kong, the model is “define Services and Routes, then attach policy plugins like auth and rate limiting.” Kong’s docs emphasize entities such as Services, Routes, Consumers, Upstreams, and Plugins as the core gateway model. (Kong Docs)

So in practice:

  • Traefik is great for app routing and reverse proxying.
  • Kong is better when you want API-specific control like identity, quotas, and policy.

11) Resume line

Built a containerized API behind Kong Gateway in DB-less mode using declarative configuration, API key authentication, and per-consumer rate limiting.

12) Best next upgrade

The strongest next step is to add JWT auth or request transformation, because those show off Kong as an API gateway rather than just a reverse proxy. Kong’s plugin ecosystem is one of its main strengths. (Kong Docs)

to build a project (code + config) – production ready

techhadoop Uncategorized , , , ,

Here’s the production version of the starter project: real domain, automatic HTTPS, HTTP→HTTPS redirect, and a secured Traefik dashboard.

This uses Traefik’s Docker provider with labels for routing, a Let’s Encrypt certificate resolver for TLS, and the dashboard in secure mode rather than api.insecure=true. Traefik’s docs recommend securing the dashboard and show Docker Compose setups for HTTPS with ACME. (Traefik Docs)

Before you start

You need:

  • a Linux server with Docker and Docker Compose
  • a domain or subdomain pointing to that server
  • ports 80 and 443 open to the internet

For the HTTP-01 challenge, Traefik’s ACME guide requires the app to be reachable publicly and the domain to point to the Traefik instance. (Traefik Docs)

Recommended structure

devops-starter/
├── app/
│   ├── package.json
│   └── server.js
├── letsencrypt/
│   └── acme.json
├── .github/
│   └── workflows/
│       └── publish.yml
├── .env
├── Dockerfile
└── compose.yml

1) app/package.json

{
  "name": "devops-starter",
  "version": "1.0.0",
  "description": "Node app behind Traefik with HTTPS",
  "main": "server.js",
  "scripts": {
    "start": "node server.js"
  },
  "license": "MIT"
}

2) app/server.js

const http = require("http");
const PORT = process.env.PORT || 3000;
const server = http.createServer((req, res) => {
  if (req.url === "/healthz") {
    res.writeHead(200, { "Content-Type": "application/json" });
    return res.end(JSON.stringify({ ok: true }));
  }
  const body = {
    ok: true,
    message: "Hello from production",
    method: req.method,
    url: req.url,
    hostname: req.headers.host,
    time: new Date().toISOString()
  };
  res.writeHead(200, { "Content-Type": "application/json" });
  res.end(JSON.stringify(body, null, 2));
});
server.listen(PORT, () => {
  console.log(`Server running on port ${PORT}`);
});

3) Dockerfile

FROM node:20-alpine

WORKDIR /app

COPY app/package.json ./
RUN npm install --omit=dev

COPY app/server.js ./

ENV PORT=3000
EXPOSE 3000

CMD ["npm", "start"]

4) .env

Replace these with your real values:

DOMAIN=app.yourdomain.com
TRAEFIK_DASHBOARD_HOST=traefik.yourdomain.com
LETSENCRYPT_EMAIL=you@example.com

# Generate this with: htpasswd -nb admin 'your-strong-password'
# Then double the $ signs when putting it here for docker labels
TRAEFIK_BASIC_AUTH=admin:$$apr1$$replace$$with-real-hash

Traefik’s BasicAuth middleware supports htpasswd-style hashes, and its docs note that when using Docker labels, dollar signs need escaping. (Traefik Docs)

5) Create the certificate storage file

Run this once on the server:

mkdir -p letsencrypt
touch letsencrypt/acme.json
chmod 600 letsencrypt/acme.json

Traefik’s ACME examples store certificates in acme.json, and the file should be writable by Traefik while remaining protected. (Traefik Docs)

6) compose.yml

services:
  traefik:
    image: traefik:v3.4
    restart: unless-stopped
    command:
      - "--api.dashboard=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"

      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"

      - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"

      - "--certificatesresolvers.le.acme.email=${LETSENCRYPT_EMAIL}"
      - "--certificatesresolvers.le.acme.storage=/letsencrypt/acme.json"
      - "--certificatesresolvers.le.acme.httpchallenge=true"
      - "--certificatesresolvers.le.acme.httpchallenge.entrypoint=web"

      - "--accesslog=true"
      - "--log.level=INFO"

    ports:
      - "80:80"
      - "443:443"

    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "./letsencrypt:/letsencrypt"

    labels:
      - "traefik.enable=true"

      # Secure dashboard
      - "traefik.http.routers.dashboard.rule=Host(`${TRAEFIK_DASHBOARD_HOST}`)"
      - "traefik.http.routers.dashboard.entrypoints=websecure"
      - "traefik.http.routers.dashboard.tls=true"
      - "traefik.http.routers.dashboard.tls.certresolver=le"
      - "traefik.http.routers.dashboard.service=api@internal"
      - "traefik.http.routers.dashboard.middlewares=dashboard-auth"
      - "traefik.http.middlewares.dashboard-auth.basicauth.users=${TRAEFIK_BASIC_AUTH}"

  app:
    build:
      context: .
      dockerfile: Dockerfile
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.app.rule=Host(`${DOMAIN}`)"
      - "traefik.http.routers.app.entrypoints=websecure"
      - "traefik.http.routers.app.tls=true"
      - "traefik.http.routers.app.tls.certresolver=le"

      # Tell Traefik which internal port the app listens on
      - "traefik.http.services.app.loadbalancer.server.port=3000"

Why these labels and flags matter:

  • Traefik uses Docker labels as dynamic config when Docker is the provider. (Traefik Docs)
  • entrypoints.web and entrypoints.websecure define listeners on ports 80 and 443. (Traefik Docs)
  • The web entrypoint redirects all traffic to websecure, which is the standard Traefik redirect pattern. (Traefik Docs)
  • tls.certresolver=le tells the router to request and renew certificates through the Let’s Encrypt resolver you defined. (Traefik Docs)
  • The dashboard can be exposed securely through api@internal and protected with BasicAuth instead of insecure mode. (Traefik Docs)

7) DNS records

Create DNS records like:

  • A app.yourdomain.com -> your_server_ip
  • A traefik.yourdomain.com -> your_server_ip

If you use IPv6, add AAAA records too. The names used in your router Host(...) rules must resolve to the server running Traefik for ACME issuance to work. (Traefik Docs)

8) First deploy

From the project folder on your server:

docker compose up -d --build

Then open:

  • https://app.yourdomain.com
  • https://traefik.yourdomain.com

On first startup, Traefik should obtain certificates automatically via Let’s Encrypt as requests arrive for matching routers using the resolver. (Traefik Docs)

Useful commands:

docker compose logs -f traefik
docker compose logs -f app
docker compose ps

9) Generate the dashboard password hash

If htpasswd is installed:

htpasswd -nb admin 'your-strong-password'

Put the result in .env as TRAEFIK_BASIC_AUTH=..., but replace every $ with $$ so Docker Compose does not treat them as variable substitutions. Traefik’s BasicAuth docs explicitly mention escaping dollar signs in Docker label contexts. (Traefik Docs)

10) Publish the image from GitHub Actions

If you want Actions to build and push your app image to GHCR, use this workflow.

.github/workflows/publish.yml

name: Build and publish image
on:
  push:
    branches: ["main"]
env:
  IMAGE_NAME: ghcr.io/${{ github.repository_owner }}/devops-starter
jobs:
  publish:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    steps:
      - name: Check out repository
        uses: actions/checkout@v4
      - name: Log in to GHCR
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Set up Buildx
        uses: docker/setup-buildx-action@v3
      - name: Extract metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.IMAGE_NAME }}
          tags: |
            type=raw,value=latest
            type=sha
      - name: Build and push
        uses: docker/build-push-action@v6
        with:
          context: .
          file: ./Dockerfile
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}

GitHub’s docs recommend docker/build-push-action for building and publishing images, and GHCR uses the Container registry at ghcr.io. Workflows can authenticate with GITHUB_TOKEN when package permissions are configured appropriately. (GitHub Docs)

If you switch to pulling the published image on the server, replace the app service in Compose with:

  app:
    image: ghcr.io/YOUR_GITHUB_USERNAME/devops-starter:latest
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.app.rule=Host(`${DOMAIN}`)"
      - "traefik.http.routers.app.entrypoints=websecure"
      - "traefik.http.routers.app.tls=true"
      - "traefik.http.routers.app.tls.certresolver=le"
      - "traefik.http.services.app.loadbalancer.server.port=3000"

11) What makes this “production enough” for a first real project

This version is much closer to a real deployment because it has:

  • automatic HTTPS
  • secure dashboard access
  • HTTP→HTTPS redirect
  • restart policy
  • access logs
  • a health endpoint
  • optional CI image publishing

Those pieces line up with Traefik’s Docker standalone guidance and dashboard/ACME docs. (Traefik Docs)

12) Common failure points

If it does not work, the usual causes are:

  • DNS not pointing at the server
  • ports 80/443 blocked by firewall or cloud security group
  • invalid TRAEFIK_BASIC_AUTH hash formatting
  • acme.json missing or wrong permissions
  • router hostnames not matching the browser request

Those are the most common things that prevent Traefik from issuing certs or matching routers in Docker setups. (Traefik Docs)

13) Resume line for this project

Deployed a containerized Node.js service behind Traefik with automatic Let’s Encrypt TLS, secure reverse-proxy routing, and GitHub Actions image publishing to GHCR.

That is solid, real DevOps experience.

step-by-step to build a project (code + config)

techhadoop Uncategorized , , , ,

here’s a copyable starter project you can build end to end.

It gives you:

  • a tiny Node app in Docker
  • Traefik in front of it
  • hostname-based routing on localhost
  • a GitHub Actions workflow that builds the image
  • a path to deploy the same stack to a server later

This matches Traefik’s current Docker provider pattern, where Traefik watches Docker and picks up routing config from container labels. (Traefik Labs Documentation)

Project structure

devops-starter/
├── app/
│   ├── package.json
│   └── server.js
├── .github/
│   └── workflows/
│       └── docker.yml
├── .dockerignore
├── Dockerfile
└── compose.yml

1) app/package.json

{
  "name": "devops-starter",
  "version": "1.0.0",
  "description": "Simple Node app behind Traefik",
  "main": "server.js",
  "scripts": {
    "start": "node server.js"
  },
  "license": "MIT"
}

2) app/server.js

const http = require("http");
const PORT = process.env.PORT || 3000;
const server = http.createServer((req, res) => {
  const body = {
    ok: true,
    message: "Hello from the app behind Traefik",
    method: req.method,
    url: req.url,
    hostname: req.headers.host,
    time: new Date().toISOString()
  };
  res.writeHead(200, { "Content-Type": "application/json" });
  res.end(JSON.stringify(body, null, 2));
});
server.listen(PORT, () => {
  console.log(`Server running on port ${PORT}`);
});

3) Dockerfile

This uses an official Node image and a fixed major version tag, which is in line with GitHub’s Dockerfile guidance. (GitHub Docs)

FROM node:20-alpine
WORKDIR /app
COPY app/package.json ./
RUN npm install --omit=dev
COPY app/server.js ./
ENV PORT=3000
EXPOSE 3000
CMD ["npm", "start"]

4) .dockerignore

node_modules
npm-debug.log
.git
.github
Dockerfile
compose.yml

5) compose.yml

This follows the same core idea as Traefik’s Docker Compose examples: enable the Docker provider, disable exposing containers by default, define an HTTP entrypoint, and add labels to the app container so Traefik creates the router automatically. (Traefik Labs Documentation)

services:
  traefik:
    image: traefik:v3.0
    command:
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entryPoints.web.address=:80"
    ports:
      - "80:80"
      - "8080:8080"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
  app:
    build:
      context: .
      dockerfile: Dockerfile
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.app.rule=Host(`app.localhost`)"
      - "traefik.http.routers.app.entrypoints=web"

A couple of notes:

  • api.insecure=true is fine for learning locally, but not for a public server. Traefik’s dashboard docs treat this as something to secure for real deployments. (Traefik Labs Documentation)
  • Because both services are in the same Compose stack, Docker networking handles connectivity between Traefik and the app. That is the same pattern used in Docker and Traefik quick-start examples. (Traefik Labs Documentation)

6) .github/workflows/docker.yml

GitHub’s docs show Docker builds in Actions using actions/checkout and docker/build-push-action. This workflow keeps it simple: it builds on every push to main, and you can later extend it to push to Docker Hub or GHCR. (GitHub Docs)

name: Build Docker image
on:
  push:
    branches: ["main"]
  pull_request:
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Check out repo
        uses: actions/checkout@v4
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      - name: Build image
        uses: docker/build-push-action@v6
        with:
          context: .
          file: ./Dockerfile
          push: false
          tags: devops-starter:latest

7) Run it locally

From the project root:

docker compose up -d --build

Then open:

  • http://app.localhost
  • http://localhost:8080 for the Traefik dashboard

Traefik’s Docker quick-start uses the same localhost-style host rule pattern, and the dashboard is commonly exposed on port 8080 in the getting-started setup. (Traefik Labs Documentation)

To stop:

docker compose down

To view logs:

docker compose logs -f

8) What’s happening

When you visit http://app.localhost:

  1. your browser sends a request to port 80
  2. Traefik receives it
  3. Traefik checks Docker-discovered labels
  4. the router rule Host(\app.localhost`)` matches
  5. Traefik forwards the request to the app container

That “dynamic config from Docker labels” model is a central part of Traefik’s configuration overview and Docker provider docs. (Traefik Labs Documentation)

9) Make it feel more real

Add a second app to prove routing works.

Update compose.yml like this:

services:
  traefik:
    image: traefik:v3.0
    command:
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entryPoints.web.address=:80"
    ports:
      - "80:80"
      - "8080:8080"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
  app:
    build:
      context: .
      dockerfile: Dockerfile
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.app.rule=Host(`app.localhost`)"
      - "traefik.http.routers.app.entrypoints=web"
  whoami:
    image: traefik/whoami
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`whoami.localhost`)"
      - "traefik.http.routers.whoami.entrypoints=web"

Then:

  • http://app.localhost → your app
  • http://whoami.localhost → sample Traefik test service

That mirrors Traefik’s own examples for exposing services with Docker labels. (Traefik Labs Documentation)

10) How to deploy this later

For a simple first deployment:

  • get a Linux VM
  • install Docker and Docker Compose
  • copy this project to the server
  • point a domain at the server IP
  • swap localhost routing for your real domain
  • add HTTPS with Traefik + Let’s Encrypt

Traefik documents Docker standalone setup, HTTPS entrypoints, and ACME/Let’s Encrypt support as part of its normal production path. (Traefik Labs Documentation)

Your production router label would look more like:

- "traefik.http.routers.app.rule=Host(`app.yourdomain.com`)"

11) Resume-worthy version of this project

Once this is live, you can honestly describe it like this:

Built and deployed a containerized Node.js service using Docker and Traefik with hostname-based routing and automated image builds via GitHub Actions.

That is a real DevOps project, not tutorial-only practice.

12) Best next upgrades

After this works, do these in order:

  1. add /healthz endpoint
  2. add a test job to GitHub Actions
  3. push built images to GHCR or Docker Hub
  4. deploy on a small cloud VM
  5. add HTTPS with Let’s Encrypt
  6. add Prometheus/Grafana later

GitHub’s Actions docs already provide the build-and-publish direction if you want to turn your build-only workflow into a registry-pushing workflow. (GitHub Docs)

13) The shortest possible checklist

Create files → run:

docker compose up -d --build

Visit:

http://app.localhost
http://localhost:8080

Push to GitHub → Actions builds the image definition automatically.

The 2026 Guide to DevOps Careers

techhadoop Uncategorized , , , ,

The 2026 Guide to DevOps Careers

DevOps isn’t just a job title anymore—it’s a core engineering mindset that companies rely on to ship software faster, safer, and at scale. If you’re thinking about getting into it (or leveling up), here’s a clear, realistic guide to where things stand in 2026.

What DevOps Actually Means (Now)

DevOps sits at the intersection of:

  • Software development
  • Infrastructure / cloud
  • Automation
  • Reliability & monitoring

In practice, you’re:

  • Building CI/CD pipelines
  • Managing cloud infrastructure
  • Improving deployment speed & reliability
  • Fixing production issues
  • Automating everything repetitive

Common DevOps Roles (2026)

DevOps Engineer

  • Focus: CI/CD, automation, infrastructure
  • Tools: GitHub Actions, Jenkins, Terraform
  • Entry → Mid-level role

Cloud Engineer

  • Focus: Cloud platforms, networking, scalability
  • Platforms: AWS, Google Cloud Platform, Microsoft Azure
  • Heavy on infrastructure + cost optimization

Site Reliability Engineer (SRE)

  • Focus: uptime, performance, incident response
  • Origin: Google
  • More coding + systems thinking than typical DevOps

Platform Engineer (fastest-growing)

  • Focus: building internal developer platforms
  • Tools: Kubernetes, Backstage
  • Think: “DevOps as a product”

Core Skills You Need

1. Linux & Networking

  • SSH, processes, file systems
  • HTTP, DNS, load balancing

2. Containers & Orchestration

Image
Image
  • Docker → package apps
  • Kubernetes → run them at scale

3. CI/CD Pipelines

  • Automate build → test → deploy
  • Tools: GitLab CI, CircleCI

4. Infrastructure as Code (IaC)

  • Manage infra like code
  • Tools: Terraform, Ansible

5. Observability

  • Logs, metrics, tracing
  • Tools: Prometheus, Grafana

6. Networking & Routing (where Traefik fits)

  • Reverse proxies like Traefik or NGINX
  • TLS, domains, load balancing

Learning Roadmap (Beginner → Job Ready)

Stage 1: Foundations (2–4 weeks)

  • Linux basics
  • Git + GitHub
  • Basic networking (HTTP, DNS)

Stage 2: Build Stuff (1–2 months)

  • Learn Docker
  • Deploy a simple app locally
  • Add Traefik or NGINX

Stage 3: Cloud + Automation (2–3 months)

  • Use AWS or similar
  • Write basic Terraform
  • Create CI/CD pipeline

Stage 4: Production-Level Skills

  • Learn Kubernetes
  • Add monitoring (Prometheus + Grafana)
  • Practice debugging failures

Salaries (2026 rough ranges)

(varies by country & experience)

  • Entry: $70k–$100k
  • Mid: $100k–$150k
  • Senior: $150k–$220k+
  • Platform/SRE at big tech: even higher

What’s Changed in 2026

  • Platform Engineering > DevOps titles
  • More focus on developer experience (DX)
  • AI is assisting pipelines, but you still need fundamentals
  • Kubernetes is still dominant—but simplified tools are growing

What Actually Gets You Hired

Not certificates—projects:

  • Deploy a full app with:
    • Docker
    • CI/CD
    • HTTPS (Traefik or NGINX)
  • Show logs + monitoring
  • Break things and fix them

Example Project (highly recommended)

Build this:

  • App (Node/Python)
  • Containerized with Docker
  • Routed via Traefik
  • Deployed on AWS
  • Automated with CI/CD
  • HTTPS enabled

That one project alone can outperform most resumes.

Final Reality Check

DevOps is:

  • Less about tools
  • More about systems thinking + automation mindset

If you like:

  • solving messy problems
  • understanding how systems connect
  • building things that just work

—you’ll probably enjoy it.

realistic 30-day DevOps plan that ends with a real project you can show on your resume. No fluff, just what actually builds skill.

30-Day DevOps Roadmap (Hands-on)

Final Goal (what you’ll build)

By Day 30, you’ll have:

  • A containerized app using Docker
  • Routed with Traefik
  • CI/CD pipeline via GitHub Actions
  • Deployed on AWS (or similar)
  • HTTPS enabled
  • Public URL to show employers

Week 1 — Foundations (Linux + Networking)

Learn:

  • Basic Linux commands: cd, ls, grep, chmod
  • Processes & ports
  • What HTTP, DNS, and IP actually do

Do:

  • Install Linux (or use WSL)
  • Run a simple web server:
python3 -m http.server 8000

Outcome:

You understand how apps run and are accessed via ports.

Week 2 — Containers (Docker)

Learn:

  • What containers are
  • Images vs containers
  • Ports & volumes

Do:

Build your first container:

FROM node:18
WORKDIR /app
COPY . .
RUN npm install
CMD ["node", "app.js"]

Run it:

docker build -t myapp .
docker run -p 3000:3000 myapp

Visual idea

Image

Outcome:

You can package and run any app.

Week 3 — Reverse Proxy + Routing (Traefik)

Learn:

  • What a reverse proxy does
  • Routing by domain
  • Basic HTTPS concepts

Do:

Use Traefik with Docker Compose:

  • Route:
    • app.localhost → your app
  • Add second service to understand routing

Outcome:

You understand how real production traffic is routed.

Week 4 — CI/CD + Cloud Deployment

Learn:

  • CI/CD basics (build → test → deploy)
  • Cloud basics (VMs, ports, SSH)

Do:

1. CI/CD with GitHub Actions

Create .github/workflows/deploy.yml:

name: Deploy
on: [push]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: docker build -t myapp .

2. Deploy to AWS

  • Launch EC2 instance
  • SSH in
  • Install Docker
  • Run your app + Traefik setup

3. Add HTTPS

  • Use Traefik + Let’s Encrypt
  • Point your domain to your server

Outcome:

Your app is LIVE on the internet.

Your Final Project Structure

project/
 ├── app/
 ├── Dockerfile
 ├── docker-compose.yml
 ├── traefik/
 ├── .github/workflows/

What to Put on Your Resume

Instead of “learning DevOps,” say:

Deployed a containerized application using Docker and Traefik with automated CI/CD via GitHub Actions on AWS, including HTTPS and reverse proxy routing.

That’s real experience.

Common Mistakes (avoid these)

  • ❌ Only watching tutorials
  • ❌ Skipping networking basics
  • ❌ Not deploying publicly
  • ❌ Over-focusing on Kubernetes too early

If You Want to Go Further (Day 31+)

Next steps:

  • Learn Kubernetes
  • Add monitoring with Prometheus + Grafana
  • Use Terraform

Final Advice

Don’t aim to “learn DevOps.”

Aim to:

Build → Break → Fix → Repeat

That loop is what actually makes you good.

I can guide you step-by-step to build the exact project (code + config) so you can literally copy, run, and understand everything.

I will create a separate post for this

step-by-step to build a project (code + config)

Traefik

techhadoop Uncategorized , , , ,

Traefik is an open-source reverse proxy and load balancer designed for modern cloud-native applications—especially those running in containers.

What that actually means

Think of Traefik as a smart traffic controller sitting in front of your apps:

  • It receives incoming requests (like someone visiting your website)
  • Then routes them to the correct service (e.g., your API, frontend, or another container)
  • It can also balance traffic across multiple instances of the same service

Key features

Automatic service discovery

Image
Image

Traefik integrates directly with tools like:

  • Docker
  • Kubernetes

It automatically detects new containers/services and routes traffic to them—no manual config needed.

Built-in HTTPS (SSL/TLS)

  • Automatically generates and renews certificates using Let’s Encrypt
  • Handles HTTPS setup for you (no manual certificate management)

Load balancing

  • Distributes requests across multiple instances
  • Helps keep your app fast and available

Dynamic configuration

  • Updates routes in real time when services start/stop
  • No restarts required

Dashboard & monitoring

  • Web UI shows routes, services, and traffic
  • Useful for debugging and observability

Simple example (Docker)

If you run a container with labels like:

labels:
  - "traefik.http.routers.myapp.rule=Host(`myapp.local`)"

Traefik will:

  • Detect the container
  • Create a route for myapp.local
  • Start sending traffic there automatically

When people use Traefik

  • Microservices architectures
  • Docker or Kubernetes setups
  • Hosting multiple apps on one server
  • Replacing tools like:
    • NGINX
    • HAProxy

In one sentence

Traefik is a modern, automatic reverse proxy that makes routing traffic to containerized apps simple and dynamic.

Here’s a small working Docker Compose example you can run locally. It follows Traefik’s current Docker quick-start pattern: Traefik listens to Docker, exposes port 80 for app traffic and 8080 for the dashboard, and routes a sample whoami container using labels. (Traefik Docs)

version: "3.9"
services:
  traefik:
    image: traefik:v3.0
    container_name: traefik
    command:
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entryPoints.web.address=:80"
    ports:
      - "80:80"
      - "8080:8080"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
  whoami:
    image: traefik/whoami
    container_name: whoami
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`whoami.localhost`)"
      - "traefik.http.routers.whoami.entrypoints=web"

Run it with:

docker compose up -d

Then open:

  • http://whoami.localhost → sample app
  • http://localhost:8080 → Traefik dashboard

That hostname rule is the key idea: Traefik reads the Docker labels and creates a router so requests for whoami.localhost go to the whoami container. Traefik hot-reloads this dynamic routing config from Docker without restarting. (Traefik Docs)

How to read the important lines:

  • --providers.docker=true tells Traefik to watch Docker for containers/services. (Traefik Docs)
  • --providers.docker.exposedbydefault=false means only containers with traefik.enable=true get exposed. (Traefik Docs)
  • --entryPoints.web.address=:80 creates an HTTP entrypoint on port 80. (Traefik Docs)
  • traefik.http.routers.whoami.rule=Host(\whoami.localhost`)` matches incoming requests by hostname. (Traefik Docs)

A more realistic example is routing two apps:

version: "3.9"
services:
  traefik:
    image: traefik:v3.0
    command:
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entryPoints.web.address=:80"
    ports:
      - "80:80"
      - "8080:8080"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
  app1:
    image: traefik/whoami
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.app1.rule=Host(`app1.localhost`)"
      - "traefik.http.routers.app1.entrypoints=web"
  app2:
    image: traefik/whoami
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.app2.rule=Host(`app2.localhost`)"
      - "traefik.http.routers.app2.entrypoints=web"

Then:

  • http://app1.localhost → app1
  • http://app2.localhost → app2

That is basically the Traefik workflow: define a service, add labels, and Traefik discovers it automatically. The official docs also note that when both containers are in the same Compose file, Docker’s default network is enough for Traefik to reach them. (Traefik Docs)

A couple of useful notes:

  • The dashboard setting shown here is insecure and meant for local learning, not production. (Traefik Docs)
  • For production, people usually add TLS/HTTPS and often Let’s Encrypt certificate automation through Traefik’s config. The routing and certificates are part of Traefik’s dynamic config model. (Traefik Docs)

Want me to show the same thing with HTTPS + Let’s Encrypt, or a Kubernetes Ingress example?

Here’s a real HTTPS + Let’s Encrypt Docker Compose setup for Traefik.

It uses the HTTP-01 challenge, which means your server must be publicly reachable on ports 80 and 443, and your domain’s DNS must point at that server. Traefik’s docs also note that certificate resolvers are defined in static config, the router must have TLS enabled, and the router references the resolver by name. (Traefik Docs)

version: "3.9"
services:
  traefik:
    image: traefik:v3.4
    container_name: traefik
    command:
      - "--api.dashboard=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      # entrypoints
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      # redirect http -> https
      - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      # lets encrypt
      - "--certificatesresolvers.le.acme.email=you@example.com"
      - "--certificatesresolvers.le.acme.storage=/letsencrypt/acme.json"
      - "--certificatesresolvers.le.acme.httpchallenge=true"
      - "--certificatesresolvers.le.acme.httpchallenge.entrypoint=web"
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"   # optional dashboard
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "./letsencrypt:/letsencrypt"
  app:
    image: traefik/whoami
    container_name: whoami
    labels:
      - "traefik.enable=true"
      # router
      - "traefik.http.routers.app.rule=Host(`whoami.yourdomain.com`)"
      - "traefik.http.routers.app.entrypoints=websecure"
      - "traefik.http.routers.app.tls=true"
      - "traefik.http.routers.app.tls.certresolver=le"

Then run:

mkdir -p letsencrypt
touch letsencrypt/acme.json
chmod 600 letsencrypt/acme.json
docker compose up -d

Then open:

  • https://whoami.yourdomain.com
  • http://whoami.yourdomain.com will redirect to HTTPS

Why this works:

  • providers.docker=true makes Traefik watch Docker containers and labels. (Traefik Docs)
  • entrypoints.web and entrypoints.websecure create listeners on ports 80 and 443. (Traefik Docs)
  • certificatesresolvers.le.acme.* configures Let’s Encrypt and stores certs in acme.json. (Traefik Docs)
  • tls.certresolver=le tells that router to request and renew a cert through the le resolver. (Traefik Docs)

A couple of important gotchas:

  • Replace you@example.com with your real email.
  • Replace whoami.yourdomain.com with a real domain/subdomain you control.
  • That hostname needs an A or AAAA record pointing to your server. Traefik’s docs say ACME domains must point to Traefik. (Traefik Docs)
  • Port 8080 is just for the dashboard. Don’t leave it openly exposed on the internet unless you secure it. (Traefik Docs)

For Kubernetes, the usual Traefik-native route is an IngressRoute or standard Ingress; Traefik also documents a working Kubernetes + Let’s Encrypt example with CRDs. (Traefik Docs)

OCP vs EKS vs AKS — clear, real-world comparison

techhadoop Uncategorized , , , ,

OCP vs EKS vs AKS — clear, real-world comparison

Image

You’re comparing three ways to run Kubernetes:

  • Red Hat OpenShift (OCP) → full enterprise platform
  • Amazon EKS → AWS-managed Kubernetes
  • Azure Kubernetes Service (AKS) → Azure-managed Kubernetes

One-line mental model

  • OCP = Kubernetes + platform + opinionated tooling
  • EKS / AKS = Kubernetes as a service

Core architecture difference

OpenShift (OCP)

  • You manage:
    • cluster (unless using ROSA/ARO)
    • nodes
  • Comes with:
    • registry
    • CI/CD
    • security policies
    • operators
  • Runs:
    • on-prem, cloud, hybrid

EKS (AWS)

  • AWS manages:
    • control plane
  • You manage:
    • worker nodes (or use Fargate)
  • Uses AWS ecosystem:
    • IAM
    • ALB / NLB
    • VPC networking

AKS (Azure)

  • Azure manages:
    • control plane
  • You manage:
    • node pools
  • Uses Azure ecosystem:
    • Azure AD
    • Load Balancer
    • VNets

Security model

FeatureOCPEKSAKS
Default security🔒 Very strictModerateModerate
Pod restrictionsSCC (strong)PSP/OPA (optional)Azure policies
IdentityRBAC + OAuthIAM rolesAzure AD

OCP is the most locked-down by default.

Networking & exposure

FeatureOCPEKSAKS
External accessRoutesIngress + ALBIngress + Azure LB
CNIOVN-KubernetesAWS VPC CNIAzure CNI / Kubenet

OCP’s Routes = simpler developer experience
EKS/AKS = more cloud-native integrations

CI/CD & Developer Experience

FeatureOCPEKSAKS
Built-in CI/CD✅ Yes (BuildConfig, pipelines)❌ No❌ No
Container registry✅ Built-in❌ (ECR external)❌ (ACR external)
Developer UI✅ StrongMinimalMinimal

OCP is a developer platform, not just infra.

Operations & automation

FeatureOCPEKSAKS
OperatorsCore conceptOptionalOptional
Cluster upgradesOperator-drivenAWS-managedAzure-managed
Add-onsBuilt-inAWS add-onsAzure add-ons

Cost model (important)

  • OCP
    • license + infra cost
  • EKS
    • control plane fee + AWS resources
  • AKS
    • control plane often free + Azure resources

OCP is usually the most expensive.

Where each shines

Use OpenShift when:

  • enterprise / regulated environments
  • on-prem or hybrid cloud
  • need built-in CI/CD + security
  • platform engineering teams

Use EKS when:

  • you’re deep in AWS ecosystem
  • want flexibility + AWS integrations
  • prefer DIY platform setup

Use AKS when:

  • you’re in Azure ecosystem
  • want simplest managed Kubernetes
  • using Azure AD, DevOps, etc.

Real-world differences that matter

1. Developer experience
  • OCP → “push code → app runs”
  • EKS/AKS → you wire everything yourself
2. Security defaults
  • OCP → restrictive (safe by default)
  • EKS/AKS → flexible (you configure security)
3. Lock-in
  • OCP → Red Hat ecosystem
  • EKS → AWS lock-in
  • AKS → Azure lock-in

Interview-ready answer

“OpenShift is a full Kubernetes platform with built-in CI/CD, registry, and strong security, while EKS and AKS are managed Kubernetes services where the cloud provider manages the control plane. OCP is more opinionated and enterprise-focused, whereas EKS and AKS provide more flexibility but require assembling additional components.”

OpenShift (OCP) – Ingress

techhadoop Uncategorized , , , ,

In OpenShift (OCP), Ingress is the mechanism that allows external traffic (HTTP/HTTPS) to reach services inside your cluster. While Kubernetes has a standard “Ingress” resource, OpenShift has historically used its own evolved version called Routes.

As of 2026, the landscape has expanded to include the Gateway API, which is the modern successor to both.

1. The Three Ways to Expose Apps

FeatureRoute (Native OCP)Ingress (K8s Standard)Gateway API (The Future)
SimplicityHigh (Very easy to use)MediumMedium/High
FlexibilityGoodLimited (Needs annotations)Extreme (Fine-grained control)
StandardRed Hat ProprietaryKubernetes LegacyKubernetes Modern (GA 2026)
Best ForStandard OCP appsCross-platform migrationComplex routing/Canary/Blue-Green

2. How the “Router” Works

The Ingress Controller in OCP is an Operator-managed deployment of HAProxy (by default).

  • It sits at the edge of the cluster.
  • It watches for new Routes or Ingresses.
  • It automatically updates its configuration and starts proxying traffic to the correct Pods.

3. Key Concepts for Admins

Ingress Controller Sharding

In large clusters, a single router can become a bottleneck. You can “shard” your ingress traffic by creating multiple Ingress Controllers.

  • Example: Create one router for *.public.example.com and a separate, isolated router for *.internal.example.com.
  • Benefit: Performance isolation and security (e.g., PCI-compliant traffic on specific nodes).

TLS Termination Patterns

Routes support four types of security:

  1. Edge: SSL is decrypted at the Router. Traffic to the Pod is plain HTTP. (Most common).
  2. Passthrough: SSL is sent directly to the Pod. The Router doesn’t see the data.
  3. Re-encryption: SSL is decrypted at the Router, inspected, then re-encrypted before being sent to the Pod.
  4. None: Simple plain HTTP.

4. Interview “Pro” Tips

  • The “503 Service Unavailable” Error: If a developer sees this on their Route, it almost always means the Readiness Probe for the Pod is failing. The Router won’t send traffic to a Pod that isn’t “Ready.”
  • Host vs Path Routing: OCP Routes excel at host-based routing (app-a.com vs app-b.com). If you need complex path-based routing (e.g., app.com/v1/api going to one service and /v2/api to another), the Gateway API is now the recommended tool over standard Routes.
  • Wildcard DNS: OCP creates a default wildcard (e.g., *.apps.cluster.example.com). Every time you create a Route without a host, OCP generates one for you using this pattern.

5. Troubleshooting Command Cheat Sheet

# Check the status of the Ingress Operator
oc get ingresscontroller -n openshift-ingress-operator
# See the HAProxy pods actually doing the work
oc get pods -n openshift-ingress
# Check if a Route is "Admitted" (Successfully configured)
oc get route <route-name> -o yaml | grep -A 5 status
# Look at router logs to see traffic errors
oc logs -n openshift-ingress deployment/router-default
In a high-scale enterprise environment, you often want to isolate traffic. For example, your **Internal HR App** 
shouldn't share the same entry point (the router) as your **Public Marketing Site**. This is called **Ingress Sharding**.
In OpenShift, we achieve this by creating a second **Ingress Controller** and using **Namespace Selectors**.
---
### 1. The Strategy: Router Sharding
By default, OpenShift has a `default` Ingress Controller that handles everything. To split traffic, we will:
1.  Create a new Ingress Controller named `sharded-router`.
2.  Tell it to only watch namespaces with a specific label (e.g., `type: public`).
3.  Label our target namespace.
---
### 2. Implementation Steps
#### **Step A: Label the Namespace**
First, identify which projects should use the new, isolated router.
```bash
oc label namespace my-public-app type=public
```
#### **Step B: Create the New Ingress Controller**
This YAML creates a new set of HAProxy pods that only serve traffic for namespaces with the `type=public` label.
```yaml
apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  name: sharded-public
  namespace: openshift-ingress-operator
spec:
  domain: public.apps.mycluster.com  # A dedicated subdomain
  endpointPublishingStrategy:
    type: LoadBalancerService       # Creates a new Cloud Load Balancer
  namespaceSelector:
    matchLabels:
      type: public                  # The magic filter
  nodePlacement:                    # Optional: Run on specific "DMZ" nodes
    nodeSelector:
      matchLabels:
        node-role.kubernetes.io/infra: ""
```
---
### 3. Verification
Once applied, OpenShift will spin up new pods in the `openshift-ingress` namespace.
* **Check Pods:** `oc get pods -n openshift-ingress` (You'll see `router-sharded-public-...`)
* **Check Load Balancer:** `oc get svc -n openshift-ingress` (You'll see a new service with a unique External IP).
Now, any Route created in the `my-public-app` namespace will be picked up by the **new** router and ignored by the **default** one.
---
### 💡 Interview Questions on Sharding
* **Q: Why would you shard an Ingress Controller?**
    * **A:** For **Security** (isolating internal vs. external traffic), **Performance** (preventing a noisy neighbor from hogging the CPU/bandwidth of the router), and **Compliance** (ensuring certain data only flows through nodes that meet specific regulatory standards).
* **Q: How does the Route know which router to use?**
    * **A:** The Route doesn't "choose." The **Ingress Controllers** choose the Routes based on the `namespaceSelector` or `routeSelector` defined in their configuration.
* **Q: Can a single Route be served by two different Routers?**
    * **A:** Yes, if both Ingress Controllers have selectors that match that Route's namespace or labels. This is sometimes used during migrations.
---

OCP architecture

techhadoop Uncategorized , , , ,

OpenShift (OCP) Architecture — Clear, Practical Breakdown

Red Hat OpenShift (OCP) is a Kubernetes-based platform with extra layers for:

  • security
  • developer workflows
  • enterprise operations

Think of it as:

Kubernetes + opinionated enterprise tooling + automation

High-Level Architecture

At the highest level, OpenShift has 3 main layers:

1. Control Plane (Master Nodes)

Manages the cluster

2. Worker Nodes

Run your applications

3. Infrastructure Layer

Networking, storage, registry, ingress

1. Control Plane (Master Nodes)

Core brain of the cluster:

Key components:
  • kube-apiserver
    • entry point for all API calls
  • etcd
    • stores cluster state
  • kube-scheduler
    • assigns pods to nodes
  • kube-controller-manager
    • maintains desired state
OpenShift-specific additions:
  • OpenShift API Server
    • adds OCP-specific APIs (routes, builds, etc.)
  • Controller Manager (OpenShift)
    • handles builds, deployments, image streams

2. Worker Nodes

Where workloads run.

Components:
  • kubelet
    • manages pods on node
  • Container runtime
    • usually CRI-O (default in OpenShift)
  • Pods
    • your apps + sidecars

3. Networking Layer

Key pieces:

  • Cluster Network
    • pod-to-pod communication
  • Service Network
    • stable virtual IPs
  • Ingress / Routes (OpenShift-specific)

OpenShift uses Routes instead of standard Ingress:

  • external traffic → router → service → pod

OpenShift Router (Ingress Controller)

  • based on HAProxy
  • handles:
    • TLS termination
    • load balancing
    • external exposure

4. Image & Build System (OCP unique)

This is where OpenShift stands out.

Image Registry
  • internal container registry
Image Streams
  • track image versions
  • trigger deployments automatically
BuildConfig
  • builds images from:
    • Git
    • Dockerfile
    • Source-to-Image (S2I)

5. Security Layer (very important)

OpenShift is stricter than Kubernetes.

Features:

  • Security Context Constraints (SCC)
    • control what pods can do
    • similar to Pod Security Policies
  • No root containers by default
  • SELinux enforced
  • integrated RBAC

6. Operators (Automation Engine)

OpenShift heavily uses Operators.

  • manage apps like:
    • databases
    • monitoring
    • logging

Built-in operators:

  • cluster version operator
  • ingress operator
  • etc.

7. Observability & Logging

Built-in:

  • Prometheus (monitoring)
  • Grafana (dashboards)
  • EFK / Loki stack (logging)

Full Flow Example

Deploying an app:

  1. Push code to Git
  2. BuildConfig builds image
  3. Image stored in registry
  4. Deployment created
  5. Pod runs on worker node
  6. Service exposes pod internally
  7. Route exposes app externally

OpenShift vs Kubernetes (quick view)

FeatureKubernetesOpenShift
IngressIngress resourceRoutes
Securityflexiblestrict by default
Buildsexternal toolsbuilt-in
Registryoptionalbuilt-in
UIoptionalstrong web console

Simple mental model

  • Kubernetes = engine
  • OpenShift = full platform

Interview-ready summary

OpenShift architecture is built on Kubernetes with control plane and worker nodes, but adds enterprise features like integrated registry, build pipelines, enhanced security via SCC, and a routing layer for external traffic. It also uses operators extensively to automate cluster management.

OCP troubleshooting

techhadoop Uncategorized , , , ,

In an interview, the ability to walk through a logical “drilling down” process is more important than knowing the exact answer immediately. Here is a classic scenario for an OpenShift Admin role.

The Scenario: “The Disappearing Images”

The Symptom: You are paged because developers cannot push or pull images to the internal OpenShift registry. You run oc get co and see that the image-registry operator is Degraded.

Your Task: Walk me through how you find the root cause and fix it.

Your Mock Troubleshooting Response

1. The High-Level Check

“First, I’ll check the high-level error message provided by the ClusterOperator resource. This usually gives a hint if it’s a configuration issue or a backend failure.”

oc describe clusteroperator image-registry

Interview Result: The message says: “Progressing: Unable to apply resources: storage backend not configured” or “Degraded: error creating registry pod: persistentvolumeclaim “image-registry-storage” not found.”

2. Investigate the Operator Configuration

“Since the error mentions storage, I need to look at the Image Registry’s custom configuration to see where it’s trying to store data.”

oc get configs.imageregistry.operator.openshift.io cluster -o yaml

What you are looking for: Check the spec.storage section. Is it set to pvc, s3, azure, or emptyDir?

3. Deep Dive into the Namespace

“I’ll jump into the openshift-image-registry namespace to check the health of the actual registry pods and the status of the PVC.”

oc get pods,pvc -n openshift-image-registry

Case A (PVC is Pending): “If the PVC is Pending, I’ll run oc describe pvc <pvc-name>. Usually, this reveals that the requested StorageClass doesn’t exist or there is no capacity left in the storage provider.”

Case B (Pod is CrashLoopBackOff): “If the pod is crashing, I’ll check the logs: oc logs <pod_name>. Often, this is a permission issue where the registry container can’t write to the mounted volume due to UID mismatches.”

4. The Fix

“Depending on the find, I would:”

  • If storage was missing: Update the configs.imageregistry to point to a valid StorageClass.
  • If it’s a bare-metal install: Patch the registry to use emptyDir (for non-prod) or configure a manual PV.
  • If it’s Cloud (AWS/Azure): Check if the Operator has the right IAM permissions to create the S3 bucket or Blob storage.

Bonus “Pro” Answer: The Authentication Operator

If you want to impress the interviewer, mention the Authentication Operator and Certificates.

The Scenario: Authentication is degraded because of expired certificates.

The Pro Tip: “I would check the v4-0-config-system-router-certs secret in the openshift-authentication namespace. If the Ingress wildcard cert was manually replaced but the Auth operator wasn’t updated, it will go Degraded because it can no longer validate the OAuth callback URL. I’d fix this by ensuring the router-ca is correctly synced.”

Interviewer Follow-up:

“What if you fix the storage, but the Operator is still showing Degraded after 10 minutes?”

Your Answer: “Sometimes the Operator’s ‘Sync’ loop gets stuck. I would try a graceful restart of the operator pod itself by running oc delete pod -l name=cluster-image-registry-operator -n openshift-image-registry-operator. Since it’s a deployment, a new pod will spin up, re-scan the environment, and should clear the Degraded status if the underlying issue is resolved.”

A “Pending” pod is one of the most common issues you’ll face. In an interview, the key is to show you understand that Pending = A Scheduling Problem, whereas CrashLoopBackOff = An Application Problem.

Here is how to handle this scenario like a seasoned admin.

1. The Core Diagnostic: oc describe

The first thing you must say is: “I check the Events section.” The scheduler is very vocal about why it can’t place a pod.

oc describe pod <pod-name>

Look at the very bottom under “Events”. You will usually see a FailedScheduling warning with a specific reason.

2. Common Reasons (The “Big Four”)

A. Insufficient Resources (CPU/Memory)

  • The Message: 0/6 nodes are available: 3 Insufficient cpu, 3 Insufficient memory.
  • The Reality: Kubernetes schedules based on Requests, not actual usage. Even if a node looks idle, if other pods have “reserved” that space via high requests, the scheduler won’t touch it.
  • The Fix: Scale up the cluster (Autoscaler), add nodes, or ask the developer to lower their resources.requests.

B. Mismatched NodeSelectors / Affinity

  • The Message: 0/6 nodes are available: 6 node(s) didn't match node selector.
  • The Reality: The pod is looking for a label like disktype=ssd, but no nodes have that label.
  • The Fix: Label the nodes or fix the typo in the Deployment YAML.

C. Taints and Tolerations

  • The Message: 0/6 nodes are available: 6 node(s) had taints that the pod didn't tolerate.
  • The Reality: You might have “Infra” nodes or “GPU” nodes that are tainted to keep regular apps off them. If the pod doesn’t have a matching “Toleration,” it’s banned from those nodes.
  • The Fix: Add the correct tolerations to the pod spec.

D. Unbound PersistentVolumeClaims (PVC)

  • The Message: pod has unbound immediate PersistentVolumeClaims.
  • The Reality: The pod is waiting for a disk. Maybe the StorageClass is wrong, or the disk is in US-East-1a while the nodes are in US-East-1b.
  • The Fix: Check the PVC status with oc get pvc.

3. Advanced Troubleshooting: “Resource Quotas”

If oc describe doesn’t show a scheduling error, check the Namespace Quota.

oc get quota

The Scenario: If a project has a limit of 10 CPUs and existing pods are already using 9.5, a new pod requesting 1 CPU will stay Pending because it would violate the project’s “budget,” even if the physical nodes have plenty of room.

4. Summary for the Interviewer

“To summarize, if I see a Pending pod, I follow this hierarchy:”

  1. Check Events: Use oc describe to see the scheduler’s ‘FailedScheduling’ message.
  2. Check Resources: Compare pod requests against node allocatable capacity.
  3. Check Constraints: Verify nodeSelectors, Taints, and Affinity rules.
  4. Check Storage: Ensure the PVC is bound and in the correct zone.
  5. Check Quotas: Ensure the namespace hasn’t hit its hard limit.

In an OpenShift (OCP) admin interview, “Networking” is the area where theory meets reality. By 2026, the focus has shifted entirely to OVN-Kubernetes (the default network provider) and complex traffic patterns like Egress Control.

Here are the most common networking scenarios and questions you’ll encounter.

1. OVN-Kubernetes: The Modern Standard

OpenShift transitioned from the legacy “OpenShift SDN” to OVN-Kubernetes. Interviewers will expect you to know why.

  • Question: Why did OpenShift move to OVN-Kubernetes?
    • Answer: OVN-K is built on Open vSwitch (OVS) and provides better scalability for large clusters, native support for IPv6, and advanced features like Egress IPs and IPsec encryption for pod-to-pod traffic.
  • Troubleshooting Tip: If networking feels “sluggish,” check the OVN Northbound and Southbound databases. These are the “brain” of the network. If they get out of sync, pods might have IPs but can’t talk to each other.
    • Command: oc get pods -n openshift-ovn-kubernetes (Check for failing ovnkube-node or ovnkube-control-plane pods).

2. Egress Traffic: “How do we leave the cluster?”

In enterprise environments, security teams often demand that traffic leaving the cluster has a predictable, static IP for firewall whitelisting.

  • Question: How do you give a specific Project a dedicated external IP?
    • Answer: By using an Egress IP. You assign an IP to a Namespace, and any traffic leaving that namespace to the outside world will appear to come from that specific IP, rather than the node’s IP.
  • The “Egress Firewall” (EgressNetworkPolicy):
    • This is used to prevent pods from reaching specific external destinations (e.g., “Allow pods to talk to the corporate DB, but block all other internet access”).
    • Limit: You can only have one EgressNetworkPolicy per project.

3. Service vs. Route vs. Ingress

This is a classic “bread and butter” question.

  • The Problem: A developer says their application is unreachable from the internet.
  • The Admin Drill:
    1. Check the Route: Does it exist? Is it “Admitted” by the Ingress Controller? (oc get route)
    2. Check the Service: Does the Route point to a valid Service? Does that Service have Endpoints? (oc get endpoints)
    3. Check the Pod: Are the pods running? Are they passing their Readiness Probes? If a probe fails, the endpoint is removed, and the Route will return a 503 Service Unavailable.

4. Common Failure: MTU Mismatches

If you can ping a service but large data transfers (like file uploads) hang or fail, it is almost always an MTU (Maximum Transmission Unit) mismatch.

  • Scenario: You are running OCP on a platform (like Azure or a specific VPC) that uses encapsulation (VXLAN/GENEVE).
  • The Fix: The cluster network MTU must be smaller than the physical network MTU to account for the “header overhead.” If the physical network is 1500, your OVN-K network should usually be 1400.

5. Network Observability (The 2026 Edge)

In 2026, admins don’t just guess; they use the Network Observability Operator.

  • Question: How do you find out which pod is hogging all the bandwidth?
    • Answer: I use the Network Observability Operator (based on Loki). It provides a flow-collector that visualizes traffic in the OCP Console. I can see a “Top Talkers” graph to identify which pod or namespace is causing network congestion.

The “Pro” Interview Summary

If you want to sound like an expert, use these keywords:

  • East-West Traffic: Communication between pods (secured by NetworkPolicies).
  • North-South Traffic: Communication into or out of the cluster (managed by Routes/EgressIP).
  • Hairpinning: When a pod tries to reach itself via the external Route (can cause loops if not configured correctly).

In an OpenShift (OCP) interview, storage is a “Day 2” topic. By 2026, the discussion has moved from simply “how to attach a disk” to software-defined storage and data resilience.

Administrators are expected to understand the abstraction layers between the physical disk and the application.

1. The Core Abstraction (PV, PVC, and StorageClass)

Interviewers will start with the basics to ensure you know the “Kubernetes way” of handling state.

  • StorageClass (SC): The “template” for storage. It defines the provider (AWS EBS, VMware vSphere, Azure Disk) and parameters like reclaimPolicy (Delete vs. Retain).
  • PersistentVolumeClaim (PVC): The developer’s request. “I need 10GB of RWO storage.”
  • PersistentVolume (PV): The actual slice of storage that gets bound to the PVC.

2. OpenShift Data Foundation (ODF)

This is the “Enterprise” way to do storage in OCP. It is based on Ceph and Rook.

  • Question: Why use ODF instead of just direct cloud-native CSI drivers?
    • Answer: ODF provides a unified layer. It gives you Block (RWO), File (RWX), and Object (S3) storage regardless of where the cluster is running. It also enables advanced features like data replication, snapshots, and disaster recovery (DR) across clusters.
  • Key Component (NooBaa): Mention “Multicloud Object Gateway” (NooBaa). It allows you to store data across different cloud providers (e.g., AWS S3 and Azure Blob) while presenting a single S3 endpoint to the app.

3. Access Modes: RWO vs. RWX

This is a frequent “trap” question in interviews.

  • ReadWriteOnce (RWO): Can be mounted by a single node. Best for databases (PostgreSQL, MongoDB).
  • ReadWriteMany (RWX): Can be mounted by many nodes simultaneously. Essential for shared file systems or web servers serving the same static content.
    • Note: Cloud block storage (EBS/Azure Disk) is almost always RWO. To get RWX, you usually need ODF (CephFS) or a managed service like AWS EFS.

4. Critical Admin Tasks & Commands

An interviewer might ask: “A developer says their database is out of space. Walk me through the fix.”

  1. Check Capability: oc get sc <storage-class-name> -o yaml. Look for allowVolumeExpansion: true.
  2. The Fix: Edit the PVC directly: oc edit pvc <pvc-name>.
  3. The Result: If the CSI driver supports it, the PV will expand automatically, and the file system inside the pod will grow without a restart (usually).

5. Advanced: LVM Storage vs. Local Storage Operator

For bare metal or Single Node OpenShift (SNO):

  • LVM Storage Operator (LVMS): The modern (2025/2026) choice. It takes local disks and turns them into a Volume Group, allowing dynamic provisioning of small chunks of local storage.
  • Local Storage Operator (LSO): The “old” way. It binds a whole raw disk to a single PV. It’s less flexible than LVMS because it lacks dynamic resizing.

6. Storage Troubleshooting Checklist

  • PVC stuck in “Pending”:
    • Check oc describe pvc.
    • Cause: No PV available that matches the request, or the StorageClass doesn’t support “Wait For First Consumer” (scheduling issues).
  • Volume stuck in “Terminating”:
    • Cause: A pod is still using the volume. You must find the pod (oc get pods -A | grep <pvc-name>) and delete it before the storage can be released.
  • Multi-Zone Issues:
    • Cause: In AWS/Azure, a volume created in Zone A cannot be mounted by a node in Zone B. This is why “topology-aware” scheduling is critical.

Understanding OCP Backup: Two Essential Layers

techhadoop OCP , , , ,

Here’s a comprehensive breakdown of OCP backup — covering the two distinct layers you need to protect.

The two backup layers in OCP

OCP backup is not a single thing — you need two separate strategies working together:

LayerWhat it protectsTool
Control plane (etcd)Cluster state — all Kubernetes/OCP objects, CRDs, configs, RBACcluster-backup.sh / EtcdBackup CR
Application dataNamespaces, workloads, PVs/PVCs, imagesOADP (OpenShift API for Data Protection)

Use etcd backups with automated snapshots to protect and recover the cluster itself. Use OADP to protect and recover your applications and their data on top of a healthy cluster. — they are complementary, not interchangeable. OADP will not successfully backup and restore operators or etcd.

Layer 1 — etcd backup (control plane)

etcd is the key-value store for OpenShift Container Platform, which persists the state of all resource objects. An etcd backup plays a crucial role in disaster recovery.

What the backup produces

Running cluster-backup.sh on a control plane node generates two files:

  • snapshot_<timestamp>.db — the etcd snapshot (all cluster state)
  • static_kuberesources_<timestamp>.tar.gz — static pod manifests + encryption keys (if etcd encryption is enabled)

How to take a manual backup

# SSH into any control plane node
ssh core@master-0.example.com
# Run the built-in backup script
sudo /usr/local/bin/cluster-backup.sh /home/core/backup
# Copy the backup off-cluster immediately
scp core@master-0:/home/core/backup/* /safe/offsite/location/

Automated scheduled backup (OCP 4.14+)

You can create a CRD to define the schedule and retention type of automated backups:

# 1. Create a PVC for backup storage
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: etcd-backup-pvc
  namespace: openshift-etcd
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 200Gi
---
# 2. Schedule recurring backups
apiVersion: config.openshift.io/v1alpha1
kind: Backup
metadata:
  name: etcd-recurring-backup
spec:
  etcd:
    schedule: "20 4 * * *"   # Daily at 04:20 UTC
    timeZone: "UTC"
    pvcName: etcd-backup-pvc
    retentionPolicy:
      retentionType: RetentionNumber
      retentionNumber:
        maxNumberOfBackups: 15

Key rules for etcd backups

Do not take an etcd backup before the first certificate rotation completes, which occurs 24 hours after installation, otherwise the backup will contain expired certificates. It is also recommended to take etcd backups during non-peak usage hours, as it is a blocking action.

  • Backups only need to be taken from one master — there is no need to run on every master. Store backups in either an offsite location or somewhere off the server.
  • Be sure to take an etcd backup after you upgrade your cluster. When you restore your cluster, you must use an etcd backup that was taken from the same z-stream release — for example, an OCP 4.14.2 cluster must use a backup taken from 4.14.2.

Restore procedure (high level)

# On the designated recovery control plane node:
sudo -E /usr/local/bin/cluster-restore.sh /home/core/backup
# After restore completes, force etcd redeployment:
oc edit etcd cluster
# Add under spec:
#   unsupportedConfigOverrides:
#     forceRedeploymentReason: recovery-2025-04-17
# Monitor etcd pods coming back up
oc get pods -n openshift-etcd | grep -v quorum

Layer 2 — OADP (application backup)

OADP uses Velero to perform both backup and restore tasks for either resources and/or internal images, while also being capable of working with persistent volumes via Restic or with snapshots.

Install OADP via OperatorHub

Operators → OperatorHub → search "OADP" → Install

Configure a backup location (S3 example)

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: dpa-cluster
  namespace: openshift-adp
spec:
  configuration:
    velero:
      defaultPlugins:
        - openshift    # Required for OCP-specific resources
        - aws
    nodeAgent:
      enable: true
      uploaderType: kopia   # Preferred over restic in OADP 1.3+
  backupLocations:
    - name: default
      velero:
        provider: aws
        default: true
        objectStorage:
          bucket: my-ocp-backups
          prefix: cluster-1
        credential:
          name: cloud-credentials
          key: cloud
  snapshotLocations:
    - name: default
      velero:
        provider: aws
        config:
          region: ca-central-1

Taking an application backup

# Backup a specific namespace
apiVersion: velero.io/v1
kind: Backup
metadata:
  name: my-app-backup
  namespace: openshift-adp
spec:
  includedNamespaces:
    - my-app
    - my-app-db
  defaultVolumesToFsBackup: true   # Use kopia/restic for PVs
  storageLocation: default
  ttl: 720h0m0s                   # 30-day retention
# Scheduled backup (daily at 2am)
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: daily-app-backup
  namespace: openshift-adp
spec:
  schedule: "0 2 * * *"
  template:
    includedNamespaces:
      - "*"                        # All namespaces
    excludedNamespaces:
      - openshift-*                # Exclude platform namespaces
      - kube-*
    defaultVolumesToFsBackup: true
    storageLocation: default
    ttl: 168h0m0s                  # 7-day retention

Restoring from OADP

apiVersion: velero.io/v1
kind: Restore
metadata:
  name: my-app-restore
  namespace: openshift-adp
spec:
  backupName: my-app-backup
  includedNamespaces:
    - my-app
  restorePVs: true

PV backup methods

MethodHow it worksBest for
CSI SnapshotsPoint-in-time volume snapshot via storage driverCloud PVs (AWS EBS, Azure Disk, Ceph RBD)
Kopia/Restic (fs backup)File-level copy streamed to object storageAny PV, slower but universal

Supported backup storage targets

OADP supports AWS, MS Azure, GCP, Multicloud Object Gateway, and S3-compatible object storage (MinIO, NooBaa, etc.). Snapshot backups can be performed for AWS, Azure, GCP, and CSI snapshot-enabled cloud storage such as Ceph FS and Ceph RBD.

Best practices summary

PracticeDetail
3-2-1 rule3 copies, 2 media types, 1 offsite — etcd snapshots must be stored outside the cluster
Test restoresRegularly restore to a test cluster — an untested backup is not a backup
Version locketcd restores must use a backup from the same OCP z-stream version
Frequencyetcd: at minimum daily; before every upgrade; OADP: daily or per RPO requirement
Exclude platform namespacesDon’t include openshift-* in OADP — OADP doesn’t restore operators or etcd
EncryptionEncrypt backup storage at rest; etcd snapshot includes encryption keys if etcd encryption is on
Monitor backup jobsSet up alerts on failed Schedule or EtcdBackup CRs