Understanding OpenShift Certificate Management

techhadoop OCP

This is a Senior OpenShift Administrator/Architect interview question. The interviewer wants to know whether you understand how OpenShift automatically manages certificates, what certificates exist, how they are rotated, and how to troubleshoot failures.

A strong answer should explain:

  • Types of certificates
  • Who owns them
  • Automatic rotation
  • Manual rotation
  • Failure scenarios
  • Troubleshooting commands

Why Certificate Rotation Matters

Every OpenShift component communicates over TLS.

Examples:

  • kubelet → API Server
  • API Server → etcd
  • Controllers → API Server
  • Ingress Router → Applications
  • Operators → API Server
  • Nodes → Registry
  • Prometheus → kubelets

If certificates expire:

  • Nodes may become NotReady
  • API access can fail
  • etcd members may stop communicating
  • Operators can become Degraded
  • Applications using Routes may become inaccessible

Automatic certificate rotation minimizes this risk.

OpenShift Certificate Hierarchy

                    Root CA
                       │
        ┌──────────────┼──────────────┐
        │              │              │
   API Server CA    etcd CA      Ingress CA
        │              │              │
        ▼              ▼              ▼
 API Server Cert   etcd Peer Cert   Router Cert
        │
        ▼
 kubelet Client Certs
        │
        ▼
 Service Account / Client Authentication

OpenShift maintains multiple certificate authorities (CAs), each with a specific trust domain rather than using a single CA for everything.

Major Certificate Types

1. API Server Certificates

Used for:

  • oc CLI
  • kubelets
  • Operators
  • controllers
  • scheduler

Example:

oc login https://api.cluster.example.com:6443
2. kubelet Client Certificates

Every node has its own client certificate.

Used for:

  • Node authentication
  • Node registration
  • Pod lifecycle operations
  • Status updates
3. etcd Certificates

etcd uses:

  • server certificates
  • peer certificates
  • client certificates

These secure:

etcd1 <----TLS----> etcd2
4. Ingress Certificates

The OpenShift router presents certificates to external clients.

Example:

Browser
↓
Router
↓
Application

These may be:

  • default wildcard certificates
  • custom wildcard certificates
  • certificates from enterprise PKI or cert-manager
5. Service Serving Certificates

OpenShift can automatically generate serving certificates for Services.

Applications use them for:

  • internal HTTPS
  • Prometheus scraping
  • webhook communication
6. Service Account Tokens

Although not X.509 certificates, service account credentials are also rotated and managed automatically by Kubernetes.

Who Rotates Certificates?

Several Operators share responsibility:

OperatorResponsibility
Cluster Version Operator (CVO)Coordinates platform lifecycle
Machine Config Operator (MCO)Distributes trust bundles and node configuration when required
kube-controller-managerApproves and rotates kubelet client certificates
kube-apiserver OperatorManages API server certificates
etcd OperatorManages etcd certificates
Ingress OperatorManages router certificates
Authentication OperatorManages OAuth-related certificates

kubelet Certificate Rotation

The process is automatic.

Current Certificate
↓
Near Expiration
↓
kubelet Creates CSR
↓
API Server
↓
CSR Approved
↓
New Certificate Issued
↓
kubelet Reloads Certificate

Useful commands:

oc get csr
oc describe csr

API Server Certificate Rotation

The kube-apiserver Operator continuously monitors certificate validity.

When a certificate approaches expiration:

Detect Expiration
↓
Generate New Certificate
↓
Roll Out API Server Pods
↓
Wait for Readiness
↓
Old Pods Removed

The rollout is performed one API server instance at a time to maintain availability.

etcd Certificate Rotation

Each etcd member receives new certificates.

etcd-1
↓
New Certificate
↓
Restart
↓
Healthy
↓
etcd-2
↓
etcd-3

The Operator maintains quorum throughout the process.

Ingress Certificate Rotation

The Ingress Operator updates router certificates.

New Wildcard Certificate
↓
Secret Updated
↓
Router Reload
↓
Traffic Continues

Routers typically reload the certificate without requiring a full cluster outage.

Certificate Lifetime

Exact validity periods vary by OpenShift version and certificate type, but the important interview point is:

  • OpenShift monitors expiration.
  • Certificates are rotated automatically before they expire.
  • Rotation is staggered to avoid service interruption.

What Happens If Rotation Fails?

Possible symptoms include:

Node NotReady
API Unavailable
Operator Degraded
Authentication Failure
TLS Handshake Failure
Expired Certificate Errors

Examples:

x509: certificate has expired

or

remote error: tls: bad certificate

Troubleshooting Commands

Check Cluster Operators
oc get co
Check Certificate Signing Requests
oc get csr
Inspect API Server
oc get pods -n openshift-kube-apiserver
Check kubelet
journalctl -u kubelet
Examine Certificate Expiration
openssl x509 -in tls.crt -noout -dates

or

openssl s_client -connect api.cluster.example.com:6443
Check Machine Config Pools
oc get mcp

Real Interview Scenario

Question: After an OpenShift upgrade, several worker nodes become NotReady with TLS errors. How would you troubleshoot?

A good answer:

  1. Verify node status:oc get nodes
  2. Check for pending CSRs:oc get csr
  3. Inspect kubelet logs:journalctl -u kubelet
  4. Confirm the kubelet client certificate hasn’t expired.
  5. Verify API server availability and health.
  6. Check Machine Config Operator and Cluster Operators:oc get co oc get mcp
  7. Ensure system time is synchronized (for example, with chronyd), because significant clock skew can cause certificate validation failures.

Best Practices

  • Never manually replace OpenShift-managed certificates unless following Red Hat’s documented recovery procedures.
  • Monitor certificate expiration proactively.
  • Use enterprise PKI or cert-manager only for application and ingress certificates where appropriate, not for internal platform certificates.
  • Validate certificate rotation during disaster recovery and upgrade testing.
  • Keep cluster time synchronized with NTP/chrony.

Interview Answer (2-Minute Version)

“OpenShift secures communication between all platform components using multiple certificate authorities and X.509 certificates. Platform certificates include API server, kubelet, etcd, ingress, and service-serving certificates, each managed by the appropriate Operator. The kubelet automatically requests a new client certificate through a Certificate Signing Request before expiration, while Operators such as the kube-apiserver Operator, etcd Operator, and Ingress Operator rotate their respective certificates using rolling updates to maintain availability. Rotation is coordinated so components are updated incrementally rather than all at once. If rotation fails, common symptoms include NotReady nodes, degraded Operators, or TLS handshake errors. I would troubleshoot by checking ClusterOperators, pending CSRs, kubelet logs, certificate validity, MachineConfigPools, and time synchronization before considering any manual intervention.”

Leave a Reply