The Cluster Monitoring Operator (CMO) is a core, default-installed OpenShift operator responsible for deploying, managing, and maintaining the platform’s entire observability stack.

It implements a fully managed Prometheus-based monitoring solution designed to monitor core OpenShift platform components, node infrastructure, and (optionally) user-defined workload applications.

1. Architecture and Key Components

The CMO acts as a high-level orchestrator. It manages custom resources (CRs), deployments, daemonsets, and configurations across the cluster (primarily within the openshift-monitoring and openshift-user-workload-monitoring namespaces).

Plaintext

                               ┌────────────────────────────────┐
                               │  Cluster Monitoring Operator   │
                               │             (CMO)              │
                               └───────────────┬────────────────┘
                                               │ Manages
             ┌─────────────────────────────────┼─────────────────────────────────┐
             ▼                                 ▼                                 ▼
┌─────────────────────────┐       ┌─────────────────────────┐       ┌─────────────────────────┐
│     Prometheus Core     │       │      Alertmanager       │       │      Metrics Exporters  │
│  (Platform Metrics DB)  │       │    (Routing & Alerts)   │       │ (kube-state-metrics,    │
└────────────┬────────────┘       └─────────────────────────┘       │ node-exporter, cAdvisor)│
             │ Scrapes                                              └─────────────────────────┘
             ▼
┌─────────────────────────┐
│ User Workload Monitoring│
│  (Thanos / Prometheus)  │
└─────────────────────────┘

The stack deployed by CMO includes:

• Prometheus: High-performance time-series database configured to scrape platform metrics (control plane services, etcd, OVN-Kubernetes, routers).
• Thanos Querier: Provides a unified query interface (via PromQL) across both platform metrics and user workload metrics.
• Alertmanager: Handles alerting logic, deduplication, grouping, and notification routing (via PagerDuty, Slack, Webhooks, Email).
• Metrics Exporters: Includes node-exporter (host system metrics), kube-state-metrics (Kubernetes object state), and OpenShift component exporters.
• Telemeter Client: Sends a curated, anonymized subset of platform health metrics back to Red Hat Insights for proactive support.

2. Core Responsibilities

1. Automated Configuration & Reconciliation: You do not manage raw Prometheus YAML configs directly. Instead, you modify a central ConfigMap (cluster-monitoring-config), and the CMO reconciles and applies those changes across all underlying Prometheus and Alertmanager instances.
2. Platform Health Enforcement: CMO ensures critical alert rules (e.g., etcd high latency, node disk pressure, API server errors) are always active and cannot be accidentally deleted by cluster users.
3. User Workload Monitoring (UWM): By default, CMO only monitors OpenShift platform components. However, administrators can enable UWM in the CMO configuration to allow developers to deploy ServiceMonitor and PodMonitor CRDs for custom application metrics in their own namespaces.

3. Key Configuration Example

To configure CMO settings (such as setting retention periods, persistent volume claims, or enabling User Workload Monitoring), administrators edit the cluster-monitoring-config ConfigMap in the openshift-monitoring namespace.

YAML

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    # Enable User Workload Monitoring for application metrics
    enableUserWorkload: true
    
    # Configure Platform Prometheus Retention and Storage
    prometheusK8s:
      retention: 15d
      volumeClaimTemplate:
        metadata:
          name: prometheus-pvc
        spec:
          storageClassName: fast-ssd
          resources:
            requests:
              storage: 100Gi
    
    # Configure Alertmanager Storage
    alertmanagerMain:
      volumeClaimTemplate:
        metadata:
          name: alertmanager-pvc
        spec:
          storageClassName: fast-ssd
          resources:
            requests:
              storage: 10Gi

4. Useful oc Commands for Troubleshooting CMO

• Check CMO Operator Status:
  oc get clusteroperator monitoring
• View Monitoring Pods:
  oc get pods -n openshift-monitoring
• View User Workload Monitoring Pods (if enabled):
  oc get pods -n openshift-user-workload-monitoring
• Inspect CMO Logs: oc logs -n openshift-monitoring deployment/cluster-monitoring-operator -c cluster-monitoring-operator