Automated DR Setup with Terraform, Velero, and OCP

A solid Terraform + Velero + OCP automated DR setup usually splits into three lanes:

  1. Terraform rebuilds the cluster infrastructure and base OCP install.
  2. OADP/Velero backs up and restores applications, namespaces, and PV data.
  3. etcd backup/restore protects the control plane state and must use a backup from the same OCP z-stream when restoring. (Red Hat Documentation)

Recommended architecture

Git / CI
├─ Terraform
│ ├─ network, subnets, DNS, LB, IAM
│ ├─ OCP install prerequisites
│ └─ optional object storage + KMS
├─ OCP bootstrap/install
├─ Post-install automation
│ ├─ OADP Operator
│ ├─ DataProtectionApplication
│ ├─ BackupStorageLocation
│ └─ VolumeSnapshotLocation
├─ Scheduled protection
│ ├─ etcd snapshots
│ ├─ Velero/OADP schedules
│ └─ CSI snapshots or file-system backup
└─ DR pipeline
├─ Terraform recreate infra
├─ reinstall OCP
├─ etcd restore if doing full cluster rollback
└─ Velero restore for apps/data

That layout matches Red Hat’s split between control plane backup/restore and application backup/restore via OADP, and OADP exposes the main objects you automate: Backup, Restore, Schedule, BackupStorageLocation, and VolumeSnapshotLocation. (Red Hat Documentation)

What each piece should own

Terraform should manage

  • cloud network, subnets, routes, load balancers, DNS, IAM, object storage, encryption, and the repeatable OCP install scaffolding. This keeps rebuilds deterministic. The OCP install docs cover cluster-wide installation configuration, while backup guidance expects you to recover onto working infrastructure. (Red Hat Documentation)

OADP/Velero should manage

  • namespace-scoped app backups, cluster resources related to apps, and PV protection. Red Hat recommends OADP for application backup/restore on OpenShift, and Velero supports both CSI snapshots and file-system backup. (Red Hat Documentation)

etcd should be separate

  • use OpenShift’s control-plane backup flow for etcd. Red Hat explicitly says a restore must use an etcd backup from the same z-stream release, and OpenShift provides cluster-restore.sh and quorum-restore.sh to simplify recovery. (Red Hat Documentation)

Best-practice deployment pattern

Use Terraform for infra, then GitOps or post-install automation to apply OADP resources. I would not use Terraform to micromanage every backup object forever; it is better for bootstrap and guardrails than for day-to-day backup lifecycle.

A practical pattern is:

  • Terraform creates bucket, IAM, KMS, DNS, LB, install config, and optional cluster manifests.
  • OCP comes up.
  • A post-install job applies:
    • OADP Operator
    • cloud credentials secret
    • DataProtectionApplication
    • one BackupStorageLocation
    • one or more VolumeSnapshotLocation
    • Schedule objects per app tier.
      This lines up with Red Hat’s OADP install flow and Velero’s native schedule model. (Red Hat Documentation)

Reference implementation

1) Terraform: object storage and IAM

This is the part Terraform is best at. Exact provider blocks vary by cloud, but the minimum is:

  • object storage bucket for backups
  • encryption
  • versioning / lifecycle
  • IAM role or credentials for Velero/OADP
resource "aws_s3_bucket" "velero" {
bucket = var.velero_bucket_name
}
resource "aws_s3_bucket_versioning" "velero" {
bucket = aws_s3_bucket.velero.id
versioning_configuration {
status = "Enabled"
}
}
resource "aws_s3_bucket_server_side_encryption_configuration" "velero" {
bucket = aws_s3_bucket.velero.id
rule {
apply_server_side_encryption_by_default {
sse_algorithm = "aws:kms"
kms_master_key_id = var.kms_key_arn
}
}
}

2) OADP install on OpenShift

On current OpenShift, app backup/restore is done through the OADP Operator, which provides the main backup objects and integrates Velero with supported storage providers. (Red Hat Documentation)

3) DataProtectionApplication

This is the core OADP object that wires backup and snapshot locations.

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
name: dpa
namespace: openshift-adp
spec:
backupLocations:
- velero:
provider: aws
default: true
objectStorage:
bucket: infra-cloud-velero-prod
prefix: ocp-prod
config:
region: us-east-1
snapshotLocations:
- velero:
provider: aws
config:
region: us-east-1
configuration:
velero:
defaultPlugins:
- openshift
- aws
- csi

OADP’s API surface includes BackupStorageLocation and VolumeSnapshotLocation, and CSI snapshot support is the preferred volume path when your storage supports it. (Red Hat Documentation)

4) Scheduled backups

Velero schedules are cron-based repeatable backup requests. (Velero)

Example for critical apps:

apiVersion: velero.io/v1
kind: Schedule
metadata:
name: apps-hourly
namespace: openshift-adp
spec:
schedule: "0 * * * *"
template:
includedNamespaces:
- payments
- customer-api
snapshotVolumes: true
ttl: 720h

Example for lower-priority namespaces:

apiVersion: velero.io/v1
kind: Schedule
metadata:
name: apps-daily
namespace: openshift-adp
spec:
schedule: "0 2 * * *"
template:
includedNamespaces:
- reporting
- internal-tools
snapshotVolumes: true
ttl: 2160h

Velero also supports filtering by namespace, labels, and resource type, which is useful for separating critical workloads from everything else. (Velero)

5) etdc backup automation

Keep this outside Velero. OpenShift’s backup docs separate control-plane backup from OADP app backup, and Red Hat says you only need to save the etcd backup from a single control plane host. (Red Hat Documentation)

Typical automation pattern:

  • privileged automation job or external runner
  • SSH to one control plane node
  • run cluster-backup.sh
  • copy backup artifacts off-cluster to encrypted object storage
  • tag with OCP version and timestamp

Recovery workflow

App-only DR

Use this when the cluster still exists:

  1. Reinstall missing operator/app prerequisites if needed.
  2. Run Velero/OADP restore for selected namespaces or apps.
  3. CSI-backed PV restore happens through the CSI plugin during PVC restore. (Velero)

Full-cluster DR

Use this when the cluster is gone:

  1. Terraform recreates infra.
  2. Reinstall OCP.
  3. Restore etcd from a same-z-stream backup.
  4. Reconcile operators.
  5. Use OADP/Velero to restore app data and resources that are outside or after the control-plane restore point. (Red Hat Documentation)

Practical backup policy

A good production baseline is:

  • etcd: daily plus pre-upgrade snapshot
  • tier-1 apps: hourly schedule
  • tier-2 apps: daily schedule
  • PVs: CSI snapshots where supported, file-system backup where snapshots are unavailable or portability matters. Velero documents both CSI snapshot support and file-system backup, including snapshot-data movement options. (Red Hat Documentation)

Guardrails to add

  • Encrypt the backup bucket.
  • Keep backups off-cluster.
  • Tag every etcd backup with OCP z-stream.
  • Separate schedules by business tier, not “back up everything hourly.”
  • Test both restore and full rebuild regularly. Red Hat’s backup docs are explicitly framed around recovering from disaster scenarios, not just creating backups. (Red Hat Documentation)

What I would automate first

If you want the highest payoff with the least complexity, automate this order:

  1. Terraform for infra + bucket + IAM
  2. OCP install
  3. OADP Operator + DataProtectionApplication
  4. namespace-based Schedule objects
  5. etcd backup job to off-cluster storage
  6. one restore drill for app-only recovery
  7. one restore drill for full cluster rebuild

My recommendation

For OpenShift, use OADP on top of Velero rather than installing raw Velero by hand unless you have a very specific reason. That is the supported OpenShift path for application backup/restore, while etcd remains a separate control-plane backup stream. (Red Hat Documentation)

Understanding OVN in OpenShift: A Networking Overview

In OpenShift Container Platform (OCP), OVN refers to Open Virtual Network, used via OVN-Kubernetes. It’s the default networking solution in modern OpenShift clusters.


What OVN is (in simple terms)

OVN is a software-defined networking (SDN) system that:

  • Creates virtual networks for pods
  • Handles routing, switching, and network policies
  • Replaces older OpenShift SDN implementations

Think of it as the “network brain” of your cluster.


How OVN works in OCP

Core components

  • OVN Northbound DB → stores high-level network config
  • OVN Southbound DB → translates config to actual rules
  • OVN Controller (on each node) → programs networking rules
  • Open vSwitch (OVS) → does the actual packet forwarding

What it actually does

1. Pod networking

Each pod gets:

  • Its own IP address
  • Ability to talk to other pods across nodes

2. Routing

Handles:

  • Pod-to-pod communication
  • Pod-to-service traffic
  • External access (ingress/egress)

3. Network policies

Controls traffic like:

  • “Only allow frontend → backend”
  • “Block everything except specific ports”

4. Load balancing

Implements Kubernetes Services internally


Traffic flow (simplified)

  1. Pod sends traffic
  2. Goes through Open vSwitch (OVS) on the node
  3. OVN rules decide:
    • Where it goes
    • Whether it’s allowed
  4. Traffic reaches destination pod/service

Why OCP uses OVN

Compared to older SDN:

  • Better performance and scalability
  • Native support for Kubernetes NetworkPolicy
  • More flexible routing and IP management
  • Cleaner architecture (decoupled control + data plane)

OVN vs OpenShift SDN

FeatureOVN-KubernetesOpenShift SDN (legacy)
PerformanceHigherLower
NetworkPolicyFull supportLimited
IPv6SupportedLimited
Future support✅ Active❌ Deprecated

Key concepts to remember

  • OVN = control plane for networking
  • OVS = data plane (moves packets)
  • Pods communicate via virtual networking managed by OVN
  • Policies are enforced at the network layer

Real-world example

If you deploy:

  • frontend pod
  • backend pod

With OVN:

  • Both get IPs
  • OVN ensures routing between them
  • A NetworkPolicy can allow only frontend → backend traffic

Debugging OVN in OpenShift Container Platform (via OVN-Kubernetes) can feel overwhelming at first, but there’s a clear, structured way to approach it.

Below is a practical, field-tested workflow you can follow.


0. Start with the symptom

Before touching OVN internals, identify the issue type:

  • ❌ Pod can’t reach another pod
  • ❌ Pod can’t reach a service
  • ❌ External traffic not working
  • ❌ DNS failing
  • ❌ Only some nodes affected

This helps narrow the scope fast.


1. Check cluster networking health

oc get co network
  • Should be Available=True
  • If Degraded → OVN problem likely

2. Check OVN pods

oc get pods -n openshift-ovn-kubernetes

Look for:

  • CrashLoopBackOff
  • NotReady pods

Key pods:

  • ovnkube-node (runs on every node)
  • ovnkube-master

3. Check logs (most important step)

Node-level (data plane issues)

oc logs -n openshift-ovn-kubernetes <ovnkube-node-pod>

Control plane

oc logs -n openshift-ovn-kubernetes <ovnkube-master-pod>

Look for:

  • Flow programming errors
  • DB connection failures
  • OVS issues

4. Validate pod networking

Get pod IPs:

oc get pods -o wide

Test connectivity:

oc exec -it <pod> -- ping <other-pod-ip>

If this fails:

  • Likely OVN routing or policy issue

5. Check NetworkPolicies

oc get networkpolicy -A

Common mistake:

  • Policy blocking traffic unintentionally

Test by temporarily removing policy or creating an allow-all:

kind: NetworkPolicy
spec:
podSelector: {}
ingress:
- {}
egress:
- {}

6. Check Open vSwitch (OVS)

SSH into a node:

oc debug node/<node-name>
chroot /host

Then:

ovs-vsctl show

Look for:

  • Bridges (like br-int)
  • Missing interfaces = problem

7. Inspect OVN DB state

From master node:

ovn-nbctl show

Check:

  • Logical switches
  • Ports for pods

If missing → OVN not programming correctly


8. Check services & kube-proxy replacement

OVN replaces kube-proxy.

Check:

oc get svc

Test:

curl <service-cluster-ip>

If service fails but pod IP works:
→ Load balancing issue in OVN


9. Check egress / external connectivity

From pod:

curl google.com

If fails:

  • Check EgressFirewall / EgressIP
  • Check node routing

10. Use must-gather (for deep issues)

oc adm must-gather -- /usr/bin/gather_network_logs

This collects:

  • OVN DB state
  • OVS config
  • Logs

Common real-world issues

1. MTU mismatch

Symptoms:

  • Intermittent connectivity
  • Large packets fail

2. NetworkPolicy blocking traffic

Very common in production


3. OVN DB not syncing

Symptoms:

  • Pods exist but no routes

4. Node-specific issues

  • Only pods on one node fail → check that node’s ovnkube-node

5. DNS issues (often misdiagnosed as OVN)

Check:

oc get pods -n openshift-dns

Debugging mindset (this is key)

Always go in this order:

  1. Is cluster networking healthy?
  2. Are OVN pods running?
  3. Is traffic blocked (policy)?
  4. Is routing broken (OVN/OVS)?
  5. Is it actually DNS or app issue?

Pro tip

Use a debug pod:

oc run test --image=busybox -it --rm -- sh

From there:

  • ping
  • nslookup
  • curl

This isolates networking from your app.

KONG

Kong (often called Kong API Gateway) is a tool that sits in front of your APIs and manages all incoming requests—kind of like a smart gatekeeper for APIs.


Simple explanation

Instead of clients calling your backend services directly, they go through Kong first:

Client → Kong → Your APIs

Kong decides:

  • where the request goes
  • whether it’s allowed
  • how it should be handled

🔧 What Kong actually does

1. Routing (like Traefik, but API-focused)

Image
Image
  • Routes requests to the correct backend service
  • Supports paths, hosts, headers, etc.

Example:

/users → user-service
/orders → order-service

2. Authentication & Security

  • API keys
  • OAuth2 / JWT
  • Rate limiting (prevent abuse)

3. Plugins (this is Kong’s superpower)

Kong uses plugins to add features like:

  • logging
  • caching
  • transformations
  • analytics

4. Load balancing

  • Distributes traffic across multiple service instances

5. Observability

  • Logs requests
  • Tracks usage
  • Helps debug API issues

Kong vs Traefik

FeatureKongTraefik
FocusAPIsGeneral web traffic
PluginsVery powerfulMore limited
AuthBuilt-in strongBasic
Use caseMicroservices APIsContainers & routing

Quick takeaway:

  • Traefik → routing + infrastructure
  • Kong → API management + security

Where Kong fits in a system

Frontend / Mobile App
Kong
Microservices (Node, Python, etc.)
Database

Example use case

Imagine you’re building an app with:

  • user service
  • payment service
  • order service

Kong can:

  • route requests to each service
  • require authentication
  • limit requests per user
  • log all API calls

In DevOps terms

Kong is part of:

  • API Gateway layer
  • Often used with:
    • Kubernetes
    • Docker

In one sentence

Kong is an API gateway that controls, secures, and manages traffic to your backend services.


Here’s a working Kong Docker example you can compare directly with Traefik.

The cleanest starter setup is Kong Gateway in DB-less mode. In this mode, Kong runs without a database and reads its routes/services/plugins from a single declarative YAML file, which Kong documents as a supported deployment mode and a good fit for automation and CI/CD. (Kong Docs)

What you’ll build

Client → Kong → Your app

Kong will:

  • listen on port 8000 for proxied API traffic
  • expose an Admin API on port 8001 for local management/testing
  • route /api to your Node app
  • optionally apply plugins like rate limiting or key auth later

Kong’s Docker docs show Compose-based installs, and Kong’s gateway overview describes it as sitting in front of upstream services to control, analyze, and route requests. (Kong Docs)


Project structure

kong-starter/
├── app/
│ ├── package.json
│ └── server.js
├── kong/
│ └── kong.yml
├── Dockerfile
└── compose.yml

1) app/package.json

{
  "name": "kong-starter",
  "version": "1.0.0",
  "main": "server.js",
  "scripts": {
    "start": "node server.js"
  }
}



2) app/server.js

const http = require("http");
const PORT = process.env.PORT || 3000;
const server = http.createServer((req, res) => {
const body = {
ok: true,
message: "Hello from app behind Kong",
method: req.method,
url: req.url,
host: req.headers.host,
time: new Date().toISOString()
};
res.writeHead(200, { "Content-Type": "application/json" });
res.end(JSON.stringify(body, null, 2));
});
server.listen(PORT, () => {
console.log(`Server listening on ${PORT}`);
});

3) Dockerfile

FROM node:20-alpine
WORKDIR /app
COPY app/package.json ./
RUN npm install --omit=dev
COPY app/server.js ./
ENV PORT=3000
EXPOSE 3000
CMD ["npm", "start"]

4) kong/kong.yml

This is the declarative config Kong loads in DB-less mode.

_format_version: "3.0"
services:
- name: app-service
url: http://app:3000
routes:
- name: app-route
paths:
- /api

This tells Kong:

  • there is an upstream service at http://app:3000
  • requests hitting /api should be proxied there

Kong’s DB-less docs explain that entities are configured through a declarative YAML or JSON file when database=off. (Kong Docs)


5) compose.yml

services:
kong:
image: kong:3.10
environment:
KONG_DATABASE: "off"
KONG_DECLARATIVE_CONFIG: /kong/declarative/kong.yml
KONG_PROXY_ACCESS_LOG: /dev/stdout
KONG_ADMIN_ACCESS_LOG: /dev/stdout
KONG_PROXY_ERROR_LOG: /dev/stderr
KONG_ADMIN_ERROR_LOG: /dev/stderr
KONG_ADMIN_LISTEN: 0.0.0.0:8001
ports:
- "8000:8000" # proxy
- "8001:8001" # admin api
volumes:
- ./kong/kong.yml:/kong/declarative/kong.yml:ro
app:
build:
context: .
dockerfile: Dockerfile

Kong’s Docker install docs support Docker Compose installs, and Kong’s read-only/DB-less docs show using database=off with a declarative config file passed into the container. (Kong Docs)


6) Run it

docker compose up -d --build

Then test it:

curl http://localhost:8000/api

You should get JSON back from your Node app.

You can also inspect Kong locally through the Admin API:

curl http://localhost:8001/services

One important note: in DB-less mode, Kong documents that you cannot use the Admin API to write configuration the normal way, because config comes from the declarative file instead. (Kong Docs)


7) Add rate limiting

One of Kong’s main strengths is plugins. Kong’s overview emphasizes its plugin-based approach for implementing API traffic policies. (Kong Docs)

Update kong/kong.yml like this:

_format_version: "3.0"
services:
- name: app-service
url: http://app:3000
routes:
- name: app-route
paths:
- /api
plugins:
- name: rate-limiting
config:
minute: 5
policy: local

Then reload the stack:

docker compose up -d

Now Kong will rate-limit requests through the gateway.


8) Kong vs Traefik in this exact setup

Traefik version

You used labels on the app container:

- "traefik.http.routers.app.rule=Host(`app.localhost`)"

Traefik discovers Docker containers automatically and builds routing from labels. That is the core of its Docker provider model.

Kong version

You define a service and route in kong.yml:

services:
- name: app-service
url: http://app:3000
routes:
- paths:
- /api

So the practical difference is:

  • Traefik feels more infrastructure-native and auto-discovery-driven
  • Kong feels more API-platform-driven, with explicit services, routes, and plugins

Kong’s docs center services, routes, plugins, and deployment modes as the main model for managing API traffic. (Kong Docs)


9) When to use which

Use Traefik when you want:

  • simple reverse proxying
  • automatic Docker/Kubernetes discovery
  • quick app routing
  • built-in HTTPS for web apps

Use Kong when you want:

  • API gateway features
  • auth, rate limiting, transformations, analytics
  • a plugin-heavy API management layer
  • more explicit API governance

That’s an inference from how each product is documented: Traefik emphasizes reverse proxying and dynamic service discovery, while Kong emphasizes API traffic policies through plugins and gateway entities. (Kong Docs)


10) The easiest mental model

  • Traefik = “send traffic to my containers”
  • Kong = “manage and secure my APIs”

11) Resume-worthy project line

Built a containerized API service behind Kong Gateway in DB-less mode using declarative configuration for routing and traffic policy management.


Here’s the same Kong project, but now with API key auth + rate limiting — which is where Kong starts to feel very different from Traefik.

Kong’s Key Authentication plugin can require clients to send an API key in a header, query string, or request body, and Kong’s Rate Limiting plugin can throttle requests by time window. In DB-less mode, you define all of that declaratively in the config file Kong loads at startup. (Kong Docs)

What this version does

Requests to your app will:

  • go through Kong on http://localhost:8000
  • require an API key
  • be limited to 5 requests per minute
  • route to your Node app on /api

In Kong’s rate-limiting docs, if there is an auth layer, the plugin uses the authenticated Consumer for identifying clients; otherwise it falls back to client IP. (Kong Docs)

Updated kong/kong.yml

_format_version: "3.0"
services:
- name: app-service
url: http://app:3000
routes:
- name: app-route
paths:
- /api
plugins:
- name: key-auth
service: app-service
config:
key_names:
- apikey
- name: rate-limiting
service: app-service
config:
minute: 5
policy: local
consumers:
- username: demo-client
keyauth_credentials:
- key: super-secret-demo-key

Why this works:

  • key-auth protects the service with API key authentication. (Kong Docs)
  • key_names: [apikey] tells Kong to look for the API key under that name. Kong documents that keys can be supplied in headers, query params, or request body. (Kong Docs)
  • rate-limiting enforces request quotas over periods like seconds, minutes, hours, and more. (Kong Docs)
  • policy: local stores counters in-memory on the node; Kong notes this has minimal performance impact but is less accurate across multiple nodes. (Kong Docs)
  • consumers plus keyauth_credentials gives the client an identity and an API key in DB-less declarative config. That fits Kong’s DB-less model where config is the source of truth. (Kong Docs)

compose.yml

You can keep the same Compose file structure as before:

services:
kong:
image: kong:3.10
environment:
KONG_DATABASE: "off"
KONG_DECLARATIVE_CONFIG: /kong/declarative/kong.yml
KONG_PROXY_ACCESS_LOG: /dev/stdout
KONG_ADMIN_ACCESS_LOG: /dev/stdout
KONG_PROXY_ERROR_LOG: /dev/stderr
KONG_ADMIN_ERROR_LOG: /dev/stderr
KONG_ADMIN_LISTEN: 0.0.0.0:8001
ports:
- "8000:8000"
- "8001:8001"
volumes:
- ./kong/kong.yml:/kong/declarative/kong.yml:ro
app:
build:
context: .
dockerfile: Dockerfile

Kong’s Docker install docs support Compose installs, and DB-less deployments use KONG_DATABASE=off plus a declarative config file path. (Kong Docs)

Start it

docker compose up -d --build

Test without an API key

curl -i http://localhost:8000/api

This should fail because the route is protected by key-auth. Kong’s Key Auth plugin requires a valid key for access. (Kong Docs)

Test with the API key

Send the key in the apikey header:

curl -i \
-H "apikey: super-secret-demo-key" \
http://localhost:8000/api

That should succeed.

You can also pass the key as a query string because Kong’s Key Auth plugin supports query string auth too. (Kong Docs)

curl -i "http://localhost:8000/api?apikey=super-secret-demo-key"

Test the rate limit

Run this several times quickly:

for i in {1..6}; do
curl -s -o /dev/null -w "%{http_code}\n" \
-H "apikey: super-secret-demo-key" \
http://localhost:8000/api
done

You should see the first few succeed and then a 429 once you exceed the per-minute limit. Kong’s rate-limiting plugin is designed to cap requests over configured windows like minute: 5. (Kong Docs)

Why this is more “API gateway” than reverse proxy

With Traefik, the main idea was: “route traffic to the right service.” With this Kong setup, the gateway is also enforcing who can call the API and how often they can call it. Kong’s docs frame plugins like Key Auth and Rate Limiting as first-class traffic policy features for services and routes. (Kong Docs)

A practical mental model

  • Traefik: “Send requests to the right app.”
  • Kong: “Control access to the API, then send requests to the app.”

That is an inference from their documented feature emphasis: Traefik centers dynamic routing and service discovery, while Kong centers API traffic policy through gateway entities and plugins. (Kong Docs)

Good next upgrades

The next Kong features that are most worth learning are:

  • JWT auth
  • request/response transformation
  • ACLs by consumer group
  • logging plugins
  • declarative config managed from Git

Those all build naturally on Kong’s plugin model and DB-less configuration workflow. (Kong Docs)

Understanding OCP Backup: Two Essential Layers

Here’s a comprehensive breakdown of OCP backup — covering the two distinct layers you need to protect.


The two backup layers in OCP

OCP backup is not a single thing — you need two separate strategies working together:

LayerWhat it protectsTool
Control plane (etcd)Cluster state — all Kubernetes/OCP objects, CRDs, configs, RBACcluster-backup.sh / EtcdBackup CR
Application dataNamespaces, workloads, PVs/PVCs, imagesOADP (OpenShift API for Data Protection)

Use etcd backups with automated snapshots to protect and recover the cluster itself. Use OADP to protect and recover your applications and their data on top of a healthy cluster. — they are complementary, not interchangeable. OADP will not successfully backup and restore operators or etcd.


Layer 1 — etcd backup (control plane)

etcd is the key-value store for OpenShift Container Platform, which persists the state of all resource objects. An etcd backup plays a crucial role in disaster recovery.

What the backup produces

Running cluster-backup.sh on a control plane node generates two files:

  • snapshot_<timestamp>.db — the etcd snapshot (all cluster state)
  • static_kuberesources_<timestamp>.tar.gz — static pod manifests + encryption keys (if etcd encryption is enabled)

How to take a manual backup

# SSH into any control plane node
ssh core@master-0.example.com
# Run the built-in backup script
sudo /usr/local/bin/cluster-backup.sh /home/core/backup
# Copy the backup off-cluster immediately
scp core@master-0:/home/core/backup/* /safe/offsite/location/

Automated scheduled backup (OCP 4.14+)

You can create a CRD to define the schedule and retention type of automated backups:

# 1. Create a PVC for backup storage
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: etcd-backup-pvc
namespace: openshift-etcd
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 200Gi
---
# 2. Schedule recurring backups
apiVersion: config.openshift.io/v1alpha1
kind: Backup
metadata:
name: etcd-recurring-backup
spec:
etcd:
schedule: "20 4 * * *" # Daily at 04:20 UTC
timeZone: "UTC"
pvcName: etcd-backup-pvc
retentionPolicy:
retentionType: RetentionNumber
retentionNumber:
maxNumberOfBackups: 15

Key rules for etcd backups

Do not take an etcd backup before the first certificate rotation completes, which occurs 24 hours after installation, otherwise the backup will contain expired certificates. It is also recommended to take etcd backups during non-peak usage hours, as it is a blocking action.

  • Backups only need to be taken from one master — there is no need to run on every master. Store backups in either an offsite location or somewhere off the server.
  • Be sure to take an etcd backup after you upgrade your cluster. When you restore your cluster, you must use an etcd backup that was taken from the same z-stream release — for example, an OCP 4.14.2 cluster must use a backup taken from 4.14.2.

Restore procedure (high level)

# On the designated recovery control plane node:
sudo -E /usr/local/bin/cluster-restore.sh /home/core/backup
# After restore completes, force etcd redeployment:
oc edit etcd cluster
# Add under spec:
# unsupportedConfigOverrides:
# forceRedeploymentReason: recovery-2025-04-17
# Monitor etcd pods coming back up
oc get pods -n openshift-etcd | grep -v quorum

Layer 2 — OADP (application backup)

OADP uses Velero to perform both backup and restore tasks for either resources and/or internal images, while also being capable of working with persistent volumes via Restic or with snapshots.

Install OADP via OperatorHub

Operators → OperatorHub → search "OADP" → Install

Configure a backup location (S3 example)

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
name: dpa-cluster
namespace: openshift-adp
spec:
configuration:
velero:
defaultPlugins:
- openshift # Required for OCP-specific resources
- aws
nodeAgent:
enable: true
uploaderType: kopia # Preferred over restic in OADP 1.3+
backupLocations:
- name: default
velero:
provider: aws
default: true
objectStorage:
bucket: my-ocp-backups
prefix: cluster-1
credential:
name: cloud-credentials
key: cloud
snapshotLocations:
- name: default
velero:
provider: aws
config:
region: ca-central-1

Taking an application backup

# Backup a specific namespace
apiVersion: velero.io/v1
kind: Backup
metadata:
name: my-app-backup
namespace: openshift-adp
spec:
includedNamespaces:
- my-app
- my-app-db
defaultVolumesToFsBackup: true # Use kopia/restic for PVs
storageLocation: default
ttl: 720h0m0s # 30-day retention
# Scheduled backup (daily at 2am)
apiVersion: velero.io/v1
kind: Schedule
metadata:
name: daily-app-backup
namespace: openshift-adp
spec:
schedule: "0 2 * * *"
template:
includedNamespaces:
- "*" # All namespaces
excludedNamespaces:
- openshift-* # Exclude platform namespaces
- kube-*
defaultVolumesToFsBackup: true
storageLocation: default
ttl: 168h0m0s # 7-day retention

Restoring from OADP

apiVersion: velero.io/v1
kind: Restore
metadata:
name: my-app-restore
namespace: openshift-adp
spec:
backupName: my-app-backup
includedNamespaces:
- my-app
restorePVs: true

PV backup methods

MethodHow it worksBest for
CSI SnapshotsPoint-in-time volume snapshot via storage driverCloud PVs (AWS EBS, Azure Disk, Ceph RBD)
Kopia/Restic (fs backup)File-level copy streamed to object storageAny PV, slower but universal

Supported backup storage targets

OADP supports AWS, MS Azure, GCP, Multicloud Object Gateway, and S3-compatible object storage (MinIO, NooBaa, etc.). Snapshot backups can be performed for AWS, Azure, GCP, and CSI snapshot-enabled cloud storage such as Ceph FS and Ceph RBD.


Best practices summary

PracticeDetail
3-2-1 rule3 copies, 2 media types, 1 offsite — etcd snapshots must be stored outside the cluster
Test restoresRegularly restore to a test cluster — an untested backup is not a backup
Version locketcd restores must use a backup from the same OCP z-stream version
Frequencyetcd: at minimum daily; before every upgrade; OADP: daily or per RPO requirement
Exclude platform namespacesDon’t include openshift-* in OADP — OADP doesn’t restore operators or etcd
EncryptionEncrypt backup storage at rest; etcd snapshot includes encryption keys if etcd encryption is on
Monitor backup jobsSet up alerts on failed Schedule or EtcdBackup CRs