Choosing the Right OpenShift Logging Solution

You need a central log collection and storage solution, but it does not necessarily have to be ELK or Splunk.

OpenShift nodes retain local container logs only temporarily. For centralized searching, retention, alerting, auditing and compliance, install the Red Hat OpenShift Logging Operator and send logs to one or more supported backends.

For a new OpenShift deployment, the main choices are:

1. Vector + LokiStack
2. Vector + Splunk
3. Vector + external Elasticsearch/OpenSearch
4. Vector + multiple destinations

OpenShift Logging collects three primary categories:

  • Application logs — logs from application containers
  • Infrastructure logs — OpenShift, Kubernetes, CRI-O and node-service logs
  • Audit logs — Kubernetes API, OpenShift API, Linux audit and OVN audit logs

The collector runs as a DaemonSet on cluster nodes and forwards selected logs using ClusterLogForwarder. Red Hat’s current logging architecture uses Vector as the collector; Fluentd is legacy/deprecated. (Red Hat Documentation)

Recommended Native OpenShift Solution

For most new OpenShift environments, use:

OpenShift nodes
│
▼
Vector collector
│
▼
LokiStack
│
▼
OpenShift web console

Components

Red Hat OpenShift Logging Operator
│
├── Vector collectors
│ └── One collector pod per node
│
├── ClusterLogForwarder
│ └── Routing and filtering rules
│
└── LokiStack
└── Central log storage

Red Hat recommends LokiStack as the supported on-cluster log store for newer deployments. The old OpenShift-managed Elasticsearch and Kibana logging stack is no longer the strategic choice; Kibana is not supported for OpenShift logging beginning with OCP 4.16, and the OpenShift Elasticsearch Operator is no longer supported as the default logging storage solution. (Red Hat Documentation)

When LokiStack is sufficient

Choose LokiStack when:

  • You need operational troubleshooting.
  • You want logs visible in the OpenShift console.
  • You want a Red Hat-supported OpenShift-native solution.
  • Your developers primarily search logs by namespace, pod, container and labels.
  • You do not need a full enterprise SIEM.
  • You want lower infrastructure complexity than Elasticsearch.

A typical implementation:

Application logs ──────┐
Infrastructure logs ───┼──> Vector ──> LokiStack
Audit logs ────────────┘

For sensitive environments, I would normally separate audit-log retention from short-term operational logging.


Splunk Solution

Use Splunk when the organization already uses Splunk as its centralized logging or SIEM platform.

OpenShift
│
▼
Vector
│ TLS
▼
Splunk HEC
│
▼
Splunk indexers
│
▼
Search / alerts / SIEM

OpenShift Logging supports forwarding directly to Splunk HTTP Event Collector, using Vector and a ClusterLogForwarder resource. (Red Hat Documentation)

Splunk is preferable when you need
  • Enterprise SIEM integration
  • Security correlation across OpenShift, firewalls, IAM and endpoints
  • Long audit retention
  • SOC dashboards
  • Threat detection
  • Compliance reporting
  • Integration with Splunk Enterprise Security
  • Existing Splunk operational teams and licenses
Advantages
  • Mature enterprise searching and alerting
  • Strong security analytics
  • Central correlation across many technologies
  • Long-term indexed retention
  • Mature RBAC and compliance functions
Disadvantages
  • Licensing can be expensive, especially when based on ingested volume.
  • Kubernetes logs can generate very high daily volumes.
  • Poor filtering can send unnecessary debug logs into costly indexes.
  • Splunk becomes an external dependency for operational troubleshooting.

ELK or External Elasticsearch

You can forward OpenShift logs to an externally managed Elasticsearch deployment:

OpenShift nodes
│
▼
Vector
│
▼
External Elasticsearch
│
▼
Kibana

Supported OpenShift logging versions have provided Elasticsearch output from ClusterLogForwarder; however, this should be an external Elasticsearch service that your organization operates, rather than relying on the old OpenShift Elasticsearch Operator for new log-storage deployments. (Red Hat Documentation)

ELK is appropriate when
  • Your organization already operates Elasticsearch.
  • Application teams require detailed full-text searching.
  • You need Kibana dashboards.
  • You want more control over indices and data models.
  • You can operate Elasticsearch clusters reliably.
  • Splunk licensing is not justified.
Additional components you must manage
  • Elasticsearch sizing
  • Data nodes and master nodes
  • Storage capacity
  • Shard counts
  • Index lifecycle management
  • Kibana
  • TLS and authentication
  • Backup and restore
  • Version upgrades
  • Cluster health
  • Index mappings
  • Disk watermarks

ELK can cost less in licensing than Splunk, but it carries more operational responsibility.


Loki vs ELK vs Splunk

AreaLokiStackExternal ELKSplunk
OpenShift-nativeExcellentExternal integrationExternal integration
Red Hat-supported on-cluster storeYesNo for new managed ES deploymentsForwarding supported
Operational troubleshootingExcellentExcellentExcellent
Full-text analyticsMore limitedStrongStrong
SIEM capabilityLimitedRequires additional security toolsExcellent
Infrastructure complexityLow to mediumHighMedium externally
Licensing costGenerally lowerProduct-dependentUsually highest
Kubernetes label searchesExcellentGoodGood
Long-term audit retentionPossibleGoodExcellent
SOC integrationLimitedModerateExcellent
Best usePlatform operationsCustom analyticsEnterprise security

Recommended Enterprise Architecture

For a bank or regulated organization, I would use a dual-destination design:

                     OpenShift nodes
                           │
                           ▼
                        Vector
                    ┌──────┴───────┐
                    │              │
                    ▼              ▼
                LokiStack       Splunk HEC
                    │              │
             Operational logs   Audit/security logs
                    │              │
              7–30 days         1–7 years

Example routing:

Log typeDestinationPurpose
ApplicationLokiStackDeveloper and operational troubleshooting
InfrastructureLokiStack and SplunkOperations and security correlation
AuditSplunkCompliance and security monitoring
Selected critical applicationsLokiStack and SplunkOperational and business-security analysis
Debug logsLokiStack onlyAvoid expensive Splunk ingestion

This gives the platform team fast OpenShift-native log access while the security team receives relevant logs in the enterprise SIEM.

OpenShift supports pipelines that select application, infrastructure and audit inputs and route them to chosen external outputs. (Red Hat Documentation)


Example Splunk Forwarding Design

The exact API fields can vary by OpenShift Logging Operator release, so validate them against your installed Operator version.

apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
name: splunk-forwarder
namespace: openshift-logging
spec:
serviceAccount:
name: log-collector
outputs:
- name: splunk
type: splunk
splunk:
url: https://splunk-hec.example.com:8088
authentication:
token:
key: hecToken
secretName: splunk-hec-secret
pipelines:
- name: audit-to-splunk
inputRefs:
- audit
outputRefs:
- splunk
- name: infrastructure-to-splunk
inputRefs:
- infrastructure
outputRefs:
- splunk

Store the HEC token in a Secret, use TLS certificate validation and do not place the token directly in the YAML.


Sizing Considerations

Before selecting the product, estimate:

Daily volume =
nodes
× containers per node
× average log rate
× 86,400 seconds

For example:

50 nodes
× 40 containers
× 1 KB/second
≈ 173 GB per day before filtering

Actual volume varies significantly, but this demonstrates why filtering is critical.

Consider:

  • Number of nodes
  • Number of pods
  • Average events per second
  • Average event size
  • Retention period
  • Replication factor
  • Compression
  • Audit volume
  • Debug logging
  • Search concurrency
  • Availability requirements

Do not send every log to long-term expensive storage by default.


Practical Recommendation

Small or medium OpenShift platform
Vector + LokiStack

Use a 7–30-day retention period and forward only audit/security logs elsewhere when required.

Existing enterprise ELK platform
Vector + external Elasticsearch

Do not deploy the legacy OpenShift Elasticsearch Operator for a new solution.

Existing Splunk/SOC environment
Vector + LokiStack + Splunk

Use LokiStack for platform operations and Splunk for audit, security and compliance.

Regulated production environment

My preferred design is:

Application logs → LokiStack
Infrastructure logs → LokiStack + Splunk
Audit logs → Splunk
Critical app logs → LokiStack + Splunk

This balances operational usability, compliance and ingestion cost.

Interview Answer

OpenShift does not require Splunk or ELK specifically, but production clusters normally require centralized log storage because node-local container logs are temporary. I would install the Red Hat OpenShift Logging Operator and use Vector collectors deployed as a DaemonSet. For a new native OpenShift logging implementation, I would use LokiStack rather than the legacy Elasticsearch and Kibana stack.

In an enterprise already using Splunk, I would use ClusterLogForwarder to send audit and security-relevant infrastructure logs to Splunk HEC, while keeping application and operational logs in LokiStack. This avoids sending all high-volume container logs to Splunk and reduces licensing costs. External Elasticsearch remains an option when the organization already operates ELK, but its capacity, lifecycle, security and upgrades must be managed separately.

Leave a Reply