k8s

Understanding OCP Backup: Two Essential Layers

techhadoop OCP , , , ,

Here’s a comprehensive breakdown of OCP backup — covering the two distinct layers you need to protect.

The two backup layers in OCP

OCP backup is not a single thing — you need two separate strategies working together:

LayerWhat it protectsTool
Control plane (etcd)Cluster state — all Kubernetes/OCP objects, CRDs, configs, RBACcluster-backup.sh / EtcdBackup CR
Application dataNamespaces, workloads, PVs/PVCs, imagesOADP (OpenShift API for Data Protection)

Use etcd backups with automated snapshots to protect and recover the cluster itself. Use OADP to protect and recover your applications and their data on top of a healthy cluster. — they are complementary, not interchangeable. OADP will not successfully backup and restore operators or etcd.

Layer 1 — etcd backup (control plane)

etcd is the key-value store for OpenShift Container Platform, which persists the state of all resource objects. An etcd backup plays a crucial role in disaster recovery.

What the backup produces

Running cluster-backup.sh on a control plane node generates two files:

  • snapshot_<timestamp>.db — the etcd snapshot (all cluster state)
  • static_kuberesources_<timestamp>.tar.gz — static pod manifests + encryption keys (if etcd encryption is enabled)

How to take a manual backup

# SSH into any control plane node
ssh core@master-0.example.com
# Run the built-in backup script
sudo /usr/local/bin/cluster-backup.sh /home/core/backup
# Copy the backup off-cluster immediately
scp core@master-0:/home/core/backup/* /safe/offsite/location/

Automated scheduled backup (OCP 4.14+)

You can create a CRD to define the schedule and retention type of automated backups:

# 1. Create a PVC for backup storage
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: etcd-backup-pvc
  namespace: openshift-etcd
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 200Gi
---
# 2. Schedule recurring backups
apiVersion: config.openshift.io/v1alpha1
kind: Backup
metadata:
  name: etcd-recurring-backup
spec:
  etcd:
    schedule: "20 4 * * *"   # Daily at 04:20 UTC
    timeZone: "UTC"
    pvcName: etcd-backup-pvc
    retentionPolicy:
      retentionType: RetentionNumber
      retentionNumber:
        maxNumberOfBackups: 15

Key rules for etcd backups

Do not take an etcd backup before the first certificate rotation completes, which occurs 24 hours after installation, otherwise the backup will contain expired certificates. It is also recommended to take etcd backups during non-peak usage hours, as it is a blocking action.

  • Backups only need to be taken from one master — there is no need to run on every master. Store backups in either an offsite location or somewhere off the server.
  • Be sure to take an etcd backup after you upgrade your cluster. When you restore your cluster, you must use an etcd backup that was taken from the same z-stream release — for example, an OCP 4.14.2 cluster must use a backup taken from 4.14.2.

Restore procedure (high level)

# On the designated recovery control plane node:
sudo -E /usr/local/bin/cluster-restore.sh /home/core/backup
# After restore completes, force etcd redeployment:
oc edit etcd cluster
# Add under spec:
#   unsupportedConfigOverrides:
#     forceRedeploymentReason: recovery-2025-04-17
# Monitor etcd pods coming back up
oc get pods -n openshift-etcd | grep -v quorum

Layer 2 — OADP (application backup)

OADP uses Velero to perform both backup and restore tasks for either resources and/or internal images, while also being capable of working with persistent volumes via Restic or with snapshots.

Install OADP via OperatorHub

Operators → OperatorHub → search "OADP" → Install

Configure a backup location (S3 example)

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: dpa-cluster
  namespace: openshift-adp
spec:
  configuration:
    velero:
      defaultPlugins:
        - openshift    # Required for OCP-specific resources
        - aws
    nodeAgent:
      enable: true
      uploaderType: kopia   # Preferred over restic in OADP 1.3+
  backupLocations:
    - name: default
      velero:
        provider: aws
        default: true
        objectStorage:
          bucket: my-ocp-backups
          prefix: cluster-1
        credential:
          name: cloud-credentials
          key: cloud
  snapshotLocations:
    - name: default
      velero:
        provider: aws
        config:
          region: ca-central-1

Taking an application backup

# Backup a specific namespace
apiVersion: velero.io/v1
kind: Backup
metadata:
  name: my-app-backup
  namespace: openshift-adp
spec:
  includedNamespaces:
    - my-app
    - my-app-db
  defaultVolumesToFsBackup: true   # Use kopia/restic for PVs
  storageLocation: default
  ttl: 720h0m0s                   # 30-day retention
# Scheduled backup (daily at 2am)
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: daily-app-backup
  namespace: openshift-adp
spec:
  schedule: "0 2 * * *"
  template:
    includedNamespaces:
      - "*"                        # All namespaces
    excludedNamespaces:
      - openshift-*                # Exclude platform namespaces
      - kube-*
    defaultVolumesToFsBackup: true
    storageLocation: default
    ttl: 168h0m0s                  # 7-day retention

Restoring from OADP

apiVersion: velero.io/v1
kind: Restore
metadata:
  name: my-app-restore
  namespace: openshift-adp
spec:
  backupName: my-app-backup
  includedNamespaces:
    - my-app
  restorePVs: true

PV backup methods

MethodHow it worksBest for
CSI SnapshotsPoint-in-time volume snapshot via storage driverCloud PVs (AWS EBS, Azure Disk, Ceph RBD)
Kopia/Restic (fs backup)File-level copy streamed to object storageAny PV, slower but universal

Supported backup storage targets

OADP supports AWS, MS Azure, GCP, Multicloud Object Gateway, and S3-compatible object storage (MinIO, NooBaa, etc.). Snapshot backups can be performed for AWS, Azure, GCP, and CSI snapshot-enabled cloud storage such as Ceph FS and Ceph RBD.

Best practices summary

PracticeDetail
3-2-1 rule3 copies, 2 media types, 1 offsite — etcd snapshots must be stored outside the cluster
Test restoresRegularly restore to a test cluster — an untested backup is not a backup
Version locketcd restores must use a backup from the same OCP z-stream version
Frequencyetcd: at minimum daily; before every upgrade; OADP: daily or per RPO requirement
Exclude platform namespacesDon’t include openshift-* in OADP — OADP doesn’t restore operators or etcd
EncryptionEncrypt backup storage at rest; etcd snapshot includes encryption keys if etcd encryption is on
Monitor backup jobsSet up alerts on failed Schedule or EtcdBackup CRs

Kong troubleshooting

techhadoop Uncategorized , , , ,

A 502 Bad Gateway error in your setup:

Client → LB → KONG → LB → Server

means that Kong Gateway cannot successfully connect to the upstream server behind the second load balancer. Here’s how to troubleshoot and resolve it step by step:

Understanding the Flow

  1. Client hits the first LB (e.g., AWS ELB, NGINX).
  2. That LB forwards traffic to Kong Gateway.
  3. Kong receives the request and proxies it to a second LB.
  4. That LB routes to your actual upstream service.

Troubleshooting Checklist

1. Check Upstream URL in Kong

  • Run: curl -s http://<KONG_ADMIN>:8001/services/<service-name>
  • Ensure the upstream URL (url or host) points to a valid, resolvable address (DNS or IP) and correct port.

Example:

{
  "host": "internal-lb.yourdomain.local",
  "port": 8080,
  "protocol": "http"
}

2. Test Kong’s Network Reachability

From the Kong container or host:

curl -v http://internal-lb.yourdomain.local:8080/

If this fails:

  • DNS resolution might be broken
  • Port might be blocked
  • LB might not route correctly

3. Enable Debug Logs in Kong

In kong.conf or via environment variable:

log_level = debug

Then check:

tail -f /usr/local/kong/logs/error.log

Look for messages like:

  • upstream timed out
  • could not resolve host
  • connection refused

4. Check Health of Second LB and Backend

  • Ensure second LB is up
  • Verify backend servers are healthy and accepting connections
  • Check if Kong’s IP is allowed (firewall or security group)

5. Check Kong Route & Service Configuration

Validate route is defined correctly:

curl -s http://&lt;KONG_ADMIN&gt;:8001/routes

Make sure paths, hosts, or methods match the request.

Example Kong Service & Route Setup

# Service pointing to internal load balancer
curl -i -X POST http://localhost:8001/services \
  --data name=upstream-service \
  --data url=http://internal-lb.yourdomain.local:8080

# Route for the service
curl -i -X POST http://localhost:8001/services/upstream-service/routes \
  --data paths[]=/api

Common Causes of 502 with LB Behind Kong

ProblemSolution
DNS resolution failureUse IP or fix /etc/resolv.conf or CoreDNS
Port not exposed or wrongConfirm port with nc or curl
Second LB not forwarding correctlyCheck LB target groups and health checks
Kong plugins (e.g., OIDC, rate-limit) errorDisable plugins temporarily to isolate
HTTP vs HTTPS mismatchEnsure protocol matches (http vs https)
Timeout too shortIncrease proxy_read_timeout or similar

Final Tips

  • Try curl directly from Kong to the backend server.
  • Use Kong’s health check endpoint if you’re using upstream targets: curl http://localhost:8001/upstreams/<name>/health

If you share:

  • the exact curl call to Kong
  • the relevant Kong service/route config
  • error.log content from Kong

The error message “upstream prematurely closed connection while reading response header from upstream” in Kong Gateway indicates that Kong attempted to read the response headers from the upstream service, but the connection was closed unexpectedly before the headers were fully received. This typically results in a 502 Bad Gateway error.

Common Causes

  1. Upstream Service Crashes or Terminates Connection Early:
    • The upstream application may crash, encounter an error, or intentionally close the connection before sending a complete response.
  2. Timeouts:
    • The upstream service takes too long to respond, exceeding Kong’s configured timeouts.
  3. Keepalive Connection Issues:
    • Persistent connections (keepalive) between Kong and the upstream service may be closed unexpectedly by the upstream, leading to this error.
  4. Protocol Mismatch:
    • Kong expects a certain protocol (e.g., HTTP/1.1), but the upstream service responds differently or uses an incompatible protocol.
  5. Large Response Headers:
    • The upstream service sends headers that exceed Kong’s buffer sizes, causing the connection to be closed prematurely.

Understanding the Issue

When Kong is configured to connect to an upstream service, it uses the protocol specified in the service’s configuration. If the upstream service expects HTTPS connections and Kong is configured to use HTTP, the SSL/TLS handshake will fail, leading to the connection being closed prematurely.

Yes, a mismatch between the protocol specified in Kong’s service configuration and the actual protocol used by the upstream service can lead to the error:

“upstream prematurely closed connection while reading response header from upstream”

This typically occurs when Kong attempts to communicate with an upstream service over HTTP, but the upstream expects HTTPS, or vice versa.

Understanding the Issue

When Kong is configured to connect to an upstream service, it uses the protocol specified in the service’s configuration. If the upstream service expects HTTPS connections and Kong is configured to use HTTP, the SSL/TLS handshake will fail, leading to the connection being closed prematurely.

For example, if your upstream service is accessible at https://api.example.com, but Kong is configured with:

bashCopyEditcurl -i -X POST http://localhost:8001/services \
  --data name=example-service \
  --data url=http://api.example.com

Kong will attempt to connect over HTTP, resulting in a failed connection.

Solution

Ensure that the protocol in Kong’s service configuration matches the protocol expected by the upstream service.

If the upstream service requires HTTPS, configure the service in Kong accordingly:

bashCopyEditcurl -i -X POST http://localhost:8001/services \
  --data name=example-service \
  --data url=https://api.example.com

This ensures that Kong establishes a secure connection using HTTPS, aligning with the upstream service’s expectations.