OpenShift (OCP) Backup & Disaster Recovery Best Practices

One of the biggest mistakes in OpenShift environments is assuming:

“Velero/OADP backups = complete cluster backup.”

They are not.

A proper OpenShift backup strategy must protect:

1. Cluster Configuration (etcd)
2. Applications
3. Persistent Storage
4. Container Images
5. GitOps Configuration
6. Secrets & Certificates

Enterprise OpenShift Backup Architecture

                OpenShift Cluster
                       |
      +----------------+----------------+
      |                                 |
   etcd Backup                    OADP/Velero
      |                                 |
      v                                 v
 Object Storage                 Application Backup
      |
      v
 Secondary Region / DR Site

Layer 1: etcd Backup (Most Important)

What is stored in etcd?

Everything about the cluster:

• Namespaces
• Deployments
• Services
• Routes
• RBAC
• SCCs
• Secrets
• Operators
• CRDs

If etcd is lost:

Cluster State = Lost

Recommended Frequency

Production:

• Daily backup
• Before upgrades
• Before major changes

OpenShift provides:

/usr/local/bin/cluster-backup.sh

Example:

sudo -E /usr/local/bin/cluster-backup.sh

Output:

snapshot_<date>.db
static_kuberesources.tar.gz

Files generated:

etcd snapshot

snapshot.db

Static resources

static_kuberesources.tar.gz

Best Practice

Store backups outside cluster:

• S3
• Azure Blob
• NFS
• Secondary Data Center

Never keep the only backup on a master node.

Layer 2: Application Backups

Use:

OpenShift API for Data Protection

(OADP)

Built on:

Velero

What OADP Backs Up

Kubernetes Resources

• Deployments
• Services
• Routes
• Secrets
• ConfigMaps
• PVC definitions

Storage Snapshots

Cloud:

• Azure Disk
• AWS EBS
• GCP PD

or CSI snapshots.

OADP Architecture

Applications
      |
      v
    OADP
      |
      v
 Object Storage
(S3/Azure Blob/GCS)

Layer 3: Persistent Data

Critical distinction:

Backup of YAMLs

NOT enough.

Backup of storage

Required.

Examples:

• PostgreSQL
• Oracle
• MongoDB
• Kafka
• Elasticsearch

Recommended Approach

Application-consistent backup

Example:

Database dump
+
Volume snapshot

Not volume snapshot alone.

Storage Backup Strategies

OpenShift Data Foundation

Using:

OpenShift Data Foundation

Options:

• Ceph snapshots
• Ceph mirroring
• ODF DR

Cloud Storage

Azure:

• Managed Disk Snapshot

AWS:

• EBS Snapshot

GCP:

• Persistent Disk Snapshot

GitOps as Backup

One of the best architect answers.

Use:

OpenShift GitOps

Store:

• Deployments
• Operators
• Network Policies
• Routes
• Helm charts

in Git.

Benefits:

Git Repository
      =
Cluster Source of Truth

Recovery becomes much faster.

Container Registry Backup

Do not forget images.

Protect:

• Internal registry
• Quay repositories

For:

Red Hat Quay

Backup:

• Registry storage
• Database
• Configuration

Secrets Backup

Protect:

• TLS certificates
• OAuth configuration
• LDAP/OIDC configuration
• Service accounts

Examples:

oc get secrets -A

Secrets are stored in etcd, but many organizations also export critical certificates separately.

Backup Frequency Recommendations

Disaster Recovery Targets

Interviewers love this.

RPO

Recovery Point Objective

How much data can be lost?

Example:

15 minutes

RTO

Recovery Time Objective

How long to recover?

Example:

1 hour

Recommended Enterprise Targets

Multi-Cluster DR

Best enterprise architecture:

Primary OCP Cluster
         |
      OADP
         |
   Object Storage
         |
         v
Secondary OCP Cluster

Governed by:

Red Hat Advanced Cluster Management

Capabilities:

• DR orchestration
• Policy management
• Application placement
• Failover

Upgrade Backup Checklist

Before every OpenShift upgrade:

1

Backup etcd

cluster-backup.sh

2

Verify backup integrity

3

Run OADP backup

4

Validate storage snapshots

5

Export critical manifests

6

Document rollback plan

Common Mistakes

Mistake #1

Only backing up Velero/OADP.

Missing etcd.

Mistake #2

Only backing up etcd.

Missing application data.

Mistake #3

No offsite copy.

Mistake #4

No restore testing.

A backup that has never been restored is unproven.

Mistake #5

Ignoring database consistency.

Crash-consistent snapshots are not always enough.

Interview Question

“How would you design OpenShift backup and disaster recovery?”

A strong architect answer:

I use a layered backup strategy. Cluster state is protected through daily etcd backups and pre-upgrade snapshots using cluster-backup.sh. Application resources and Kubernetes objects are backed up using OADP, with backups stored in external object storage such as S3 or Azure Blob. Persistent data is protected through storage snapshots and application-consistent database backups. GitOps repositories serve as the source of truth for cluster configuration and application deployment. Container registries such as Quay are backed up separately. All backups are replicated to a secondary region or data center, and restore procedures are tested regularly. For enterprise environments, I target an RPO of less than 15 minutes and an RTO of less than one hour for critical applications.

This is the level of answer expected from a Senior OpenShift Architect, Platform Architect, or Principal Engineer.