OpenShift (OCP) Monitoring & Observability Best Practices

For an enterprise OpenShift platform, monitoring is much more than Prometheus and Grafana. A mature observability strategy includes:

1. Metrics (Prometheus/Grafana)
2. Logs (Loki, ELK, Splunk)
3. Traces (OpenTelemetry/Jaeger)
4. Alerts (Alertmanager)
5. Synthetic Monitoring
6. Business KPI Monitoring
7. SRE Practices (SLI/SLO/Error Budget)

Enterprise Observability Architecture

Five Pillars

Infrastructure Monitoring

Monitor:

• Nodes
• CPU
• Memory
• Disk
• Network
• kubelet
• CRI-O

Platform Monitoring

Monitor:

• API Server
• etcd
• Ingress
• DNS
• OVN-Kubernetes
• Machine Config Operator
• Cluster Operators

Application Monitoring

Monitor:

• Latency
• Error rates
• Throughput
• Business transactions

Logging

Collect:

• Container logs
• Audit logs
• Application logs
• Security logs

Distributed Tracing

Track:

• Microservice calls
• API latency
• Database latency

Built-In OpenShift Monitoring Stack

OpenShift includes:

Prometheus

Metrics collection

Alertmanager

Alerting

Grafana

Visualization (newer releases)

Thanos

Long-term metrics retention

kube-state-metrics

Kubernetes object metrics

node-exporter

Host metrics

Telemetry

Red Hat support telemetry

What to Monitor First

Cluster Health Dashboard

Monitor:

API Server

oc get co

Critical Metrics:

• API latency
• Request rate
• Error rate

Example Prometheus metric:

apiserver_request_duration_seconds

etcd Monitoring

The most important component.

Monitor:

• DB size
• Disk latency
• Leader changes
• Quorum

Key metrics:

etcd_server_leader_changes_seen_total
etcd_disk_backend_commit_duration_seconds

Warning signs:

• Frequent leader elections
• High disk latency

Node Monitoring

Monitor:

CPU

node_cpu_seconds_total

Memory

node_memory_MemAvailable_bytes

Disk

node_filesystem_avail_bytes

Load

node_load1

Thresholds:

OpenShift Operator Monitoring

Monitor:

oc get co

Critical Operators:

• Authentication
• Ingress
• Monitoring
• DNS
• Network
• Machine Config

Example alert:

ClusterOperatorDegraded

Ingress Monitoring

Monitor router health.

Namespace:

openshift-ingress

Key metrics:

• Request rate
• 4xx errors
• 5xx errors
• SSL failures

Prometheus:

haproxy_server_http_responses_total

OVN-Kubernetes Monitoring

Monitor:

• Packet drops
• Encapsulation issues
• Routing problems

Critical components:

oc get pods -n openshift-ovn-kubernetes

Watch:

• ovnkube-master
• ovnkube-node

Logging Best Practices

Option 1 (Recommended)

Use:

Red Hat OpenShift Logging

Components:

• Loki
• Vector
• OpenShift Console

Benefits:

• Fully supported
• Integrated

Option 2

ELK Stack

Useful when you already run ELK.

Architecture:

Applications

↓

Vector / Fluentd

↓

Logstash

↓

Elasticsearch

↓

Kibana

Since you’ve worked extensively with ELK and Filebeat, this is often the easiest integration path.

Log Retention

Recommended:

Audit Logging

Critical in banking and government.

Enable:

audit:
  profile: WriteRequestBodies

Monitor:

• Login failures
• Privilege escalation
• Secret access
• API abuse

Distributed Tracing

Use:

OpenTelemetry

and

Jaeger

Monitor:

• End-to-end request latency
• Database calls
• External API latency

Golden Signals

Every architect should know these.

Latency

Response time

Traffic

Requests per second

Errors

Failure rate

Saturation

Resource utilization

SLI / SLO Example

Banking API

SLI:

Successful requests / Total requests

SLO:

99.95% availability

Error budget:

0.05%

Interviewers love hearing SLI/SLO terminology.

Alerting Best Practices

Do NOT alert on everything.

Use:

Critical

• API down
• etcd unhealthy
• Node NotReady
• Storage unavailable

Warning

• CPU > 85%
• Memory > 90%
• Disk > 80%

Info

• Pod restarts
• Deployment changes

Capacity Planning

Track:

Cluster

• CPU allocation
• Memory allocation
• Pod density

Growth

• Monthly workload growth
• Storage consumption
• Network throughput

For enterprise clusters, keep worker utilization around:

• CPU: 60–70%
• Memory: 70–75%

to leave room for failures and upgrades.

OpenShift + Enterprise Observability Stack

A common enterprise design:

Metrics

• Prometheus
• Thanos
• Grafana

Logs

• Loki or ELK/Splunk

Traces

• OpenTelemetry
• Jaeger

Alerts

• Alertmanager
• PagerDuty
• ServiceNow

SIEM

• Splunk ES
• Microsoft Sentinel
• IBM QRadar

Architect Interview Answer

“How would you design monitoring for a production OpenShift cluster?”

I would implement a three-layer observability strategy.

At the platform layer, I would use OpenShift Monitoring with Prometheus, Alertmanager, and Thanos to monitor API servers, etcd, operators, nodes, and ingress.

At the application layer, I would instrument services using OpenTelemetry and collect metrics, logs, and traces.

For centralized logging, I would use OpenShift Logging with Loki or integrate with ELK/Splunk. Alerts would be based on SLI/SLO objectives rather than raw infrastructure thresholds. Audit logs would be forwarded to the enterprise SIEM for compliance and threat detection. Long-term metrics retention would be handled by Thanos, and dashboards would provide visibility into infrastructure, platform, application, and business KPIs. This approach provides complete observability across metrics, logs, traces, and security events while supporting enterprise-scale operations.**

This is the level of answer typically expected from a Senior OpenShift Architect or Principal Platform Engineer.