Here are the complete implementations for setting up SSM Session Manager (eliminating SSH port 22), automating Lynis via Cron with email delivery, and remediating key Prowler IAM findings.

1. Configure AWS SSM Session Manager (Disable SSH Port 22)

Using AWS Systems Manager (SSM) Session Manager allows you to open interactive shell sessions to your EC2 instances through the AWS Management Console or AWS CLI—without opening inbound port 22 in your Security Groups.

Step 1: Attach the SSM IAM Policy to your EC2 Role

1. Open the AWS IAM Console > Roles.
2. Select (or create) the IAM Role attached to your EC2 instance.
3. Attach the AWS-managed policy: AmazonSSMManagedInstanceCore.

Step 2: Verify the SSM Agent is Active

Amazon Linux 2/2023 and modern Ubuntu AMIs come with the SSM Agent pre-installed. You can verify its status on the host:

sudo systemctl status amazon-ssm-agent

Verification: The service status should show active (running).

Step 3: Remove SSH Port 22 Inbound Rule

1. Open the AWS EC2 Console > Security Groups.
2. Edit the Inbound Rules for your instance’s security group.
3. Delete the rule allowing TCP port 22.

Step 4: Connect via SSM Session Manager

From your local machine (with AWS CLI and the Session Manager plugin

installed):

aws ssm start-session --target i-0123456789example

Verification: You will receive a secure interactive bash prompt inside the instance over AWS HTTPS APIs without any open inbound ports.

2. Automate Lynis Audits via Cron with Email Reports

This setup automates a weekly Lynis security scan and emails the summarized report to your system administrator.

Step 1: Install Mail Utilities

Ensure mailx or postfix/mailutils is installed on your server:

# Amazon Linux / RHEL / CentOS:
sudo dnf install -y s-nail
# Ubuntu / Debian:
sudo apt-get update && sudo apt-get install -y mailutils

Step 2: Create the Automated Audit Script (/usr/local/bin/cron_lynis.sh)

Create the script using your preferred editor:

#!/bin/bash
# Description: Runs Lynis weekly audit and emails the report summary
ADMIN_EMAIL="admin@example.com"
REPORT_FILE="/var/log/lynis-reports/weekly_report.log"
mkdir -p /var/log/lynis-reports
# Run Lynis audit non-interactively
lynis audit system --quick --warnings-only > "${REPORT_FILE}"
# Send report via email
mail -s "Weekly Lynis Security Audit Report - $(hostname)" "${ADMIN_EMAIL}" < "${REPORT_FILE}"

Make the script executable:

sudo chmod +x /usr/local/bin/cron_lynis.sh

Step 3: Schedule a Weekly Cron Job

Open the root user’s crontab:

sudo crontab -e

Add the following entry to run the script every Sunday at 2:00 AM:

Code snippet

0 2 * * 0 /usr/local/bin/cron_lynis.sh >/dev/null 2>&1
Verification: Test the script execution manually by running sudo /usr/local/bin/cron_lynis.sh and checking /var/log/lynis-reports/weekly_report.log as well as your email inbox.

Verification: Test the script execution manually by running sudo /usr/local/bin/cron_lynis.sh and checking /var/log/lynis-reports/weekly_report.log as well as your email inbox.

3. Remediate Critical Prowler IAM Findings

Below are the three most frequent high-severity IAM flags reported by Prowler and their respective remediation steps.

Finding 1: iam_user_console_access_unused / Inactive IAM Users with Console/Access Keys

• The Risk: Inactive credentials increase the attack surface for credential theft and unauthorized access.
• Remediation: Enforce a policy to deactivate credentials unused for >90 days.
  Bash# Delete or deactivate unused access keys via AWS CLI aws iam update-access-key --user-name <username> --access-key-id <key-id> --status Inactive

Finding 2: iam_policy_no_star_privileges / Policies containing AdministratorAccess or Wildcards ("Action": "*")

• The Risk: Violates the principle of least privilege, allowing compromised instances or roles full control over the AWS account.
• Remediation: Replace overly permissive inline or attached policies with scoped-down IAM policies.
  • Example fix: Limit S3 operations to a single bucket rather than all AWS resources:
  JSON{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject" ], "Resource": "arn:aws:s3:::my-app-logs-bucket/*" } ] }

Finding 3: iam_root_hardware_mfa_enabled / Root Account Active or Missing MFA

• The Risk: The root account has unrestricted access to all resources and billing. If compromised without Multi-Factor Authentication (MFA), the entire cloud environment can be lost.
• Remediation:
  1. Log into the AWS Console as the Root User.
  2. Navigate to IAM > Dashboard > Root user environment.
  3. Enable MFA using a hardware key (FIDO2/YubiKey) or virtual authenticator app.
  4. Delete all root access keys and lock down root usage exclusively for emergency tasks.

Verification: Re-run Prowler specifically targeting IAM checks to verify remediations:

prowler aws --services iam

Purpose

This script is a pre-flight gate before triggering an OCP upgrade. It runs 7 sequential checks and blocks the upgrade if any critical condition is detected. Think of it as a checklist that a senior SRE would run manually — automated.

1. Verify Active CLI Session & Cluster Admin Access

2. Check ClusterVersion Operator (CVO) Status

What is the CVO?
The Cluster Version Operator is the top-level operator that manages the OCP version and coordinates all upgrades. If it’s unhealthy or busy, no upgrade should start.

What the jsonpath queries extract:

The clusterversion object has a status.conditions array:

yaml

status:
  conditions:
    - type: Available
      status: "True"       ← extracted by first query
    - type: Progressing
      status: "False"      ← extracted by second query
    - type: Degraded
      status: "False"

The condition check:

Equivalent manual check:

oc get clusterversion
# NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
# version   4.14.12   True        False         5d      Cluster version is 4.14.12

3. Audit Core Cluster Operators

4. Check MachineConfigPools (MCP)

What are MCPs?

MachineConfigPools define groups of nodes and the configuration applied to them. During an upgrade, the MCP controller drains and reboots each node to apply the new RHCOS and MachineConfig.

master MCP → controls all 3 master nodes
worker MCP → controls all worker nodes
infra MCP  → controls infra nodes (if defined)

Degraded MCP check:
A degraded MCP means at least one node in the pool failed to apply its MachineConfig — it didn’t reboot correctly, got stuck, or had a rendering error. Upgrading on top of a degraded MCP compounds the problem.

5. Check Node Readiness

6. Audit PodDisruptionBudgets (PDB) for Potential Deadlocks

What is a PDB?

A PodDisruptionBudget is a policy that limits how many pods of an application can be voluntarily disrupted (evicted) at once:

7. Check Active Critical Alerts

What oc get alerts does:

This uses the OpenShift alerts API resource — a custom OCP resource that surfaces Prometheus AlertManager alerts via the Kubernetes API:

oc get alerts -A
# NAMESPACE    NAME                    STATE    SEVERITY   AGE
# openshift-*  etcdHighNumberOfFailed  firing   critical   5m  ← caught
# openshift-*  NodeNotReady            pending  warning

The 2>/dev/null || echo "":
oc get alerts is not available on all OCP versions or configurations. The 2>/dev/null suppresses errors and || echo "" ensures FIRING_CRITICALS is empty (not unset) if the command fails — avoiding the -u unset variable trap.

Why critical alerts block an upgrade:

A firing critical alert means something is actively broken in the cluster. Upgrading on top of an existing critical condition risks:

• Making a broken component worse during its own operator-driven upgrade
• Masking the original problem behind upgrade noise
• A critical alert like etcdMemberDown means your etcd quorum is at risk — the worst time to upgrade

Complete flow:

START
│
├─ [1/7] oc whoami → HARD EXIT if not logged in
├─ [2/7] CVO Available+Idle → ERRORS++ if degraded or progressing
├─ [3/7] Cluster Operators → ERRORS++ if any Degraded or Unavailable
├─ [4/7] MachineConfigPools → ERRORS++ if Degraded / WARN if Paused
├─ [5/7] Node Readiness → ERRORS++ if any NotReady
├─ [6/7] PDB Deadlocks → WARN only (no ERRORS++)
└─ [7/7] Critical Alerts → ERRORS++ if any firing
│
├─ ERRORS == 0 → exit 0 (safe to upgrade)
└─ ERRORS > 0 → exit 1 (do not upgrade)

The script :

#!/usr/bin/env bash
#
# OCP Pre-Upgrade Health Check Automation Script
# Validates cluster readiness before initiating an OpenShift platform update.
#
set -euo pipefail
# ANSI Color Codes for Scannable Output
RED='\030[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
ERRORS=0
echo -e "${YELLOW}====================================================${NC}"
echo -e "${YELLOW}   OpenShift Pre-Upgrade Automated Health Check     ${NC}"
echo -e "${YELLOW}====================================================${NC}\n"
# 1. Verify Active CLI Session & Cluster Admin Access
echo -n "[1/7] Checking OpenShift CLI authentication... "
if ! oc whoami &>/dev/null; then
    echo -e "${RED}[FAILED]${NC} Not logged into an OpenShift cluster. Run 'oc login' first."
    exit 1
fi
echo -e "${GREEN}[OK]${NC} Authenticated as $(oc whoami)"
# 2. Check ClusterVersion Operator (CVO) Status
echo -n "[2/7] Checking ClusterVersion status... "
CVO_STATUS=$(oc get clusterversion -o jsonpath='{.items[0].status.conditions[?(@.type=="Available")].status}')
CVO_PROGRESSING=$(oc get clusterversion -o jsonpath='{.items[0].status.conditions[?(@.type=="Progressing")].status}')
if [[ "$CVO_STATUS" == "True" && "$CVO_PROGRESSING" == "False" ]]; then
    echo -e "${GREEN}[OK]${NC} CVO is Available and idle."
else
    echo -e "${RED}[FAILED]${NC} CVO is degraded or an update is already in progress."
    ((ERRORS++))
fi
# 3. Audit Core Cluster Operators
echo "[3/7] Auditing Cluster Operators state..."
DEGRADED_OPS=$(oc get clusteroperator -o json | jq -r '.items[] | select(.status.conditions[] | select(.type=="Degraded" and .status=="True")) | .metadata.name')
UNAVAILABLE_OPS=$(oc get clusteroperator -o json | jq -r '.items[] | select(.status.conditions[] | select(.type=="Available" and .status=="False")) | .metadata.name')
if [[ -n "$DEGRADED_OPS" ]]; then
    echo -e "  ${RED}✗ Degraded Operators found:${NC}\n$DEGRADED_OPS"
    ((ERRORS++))
else
    echo -e "  ${GREEN}✓ Zero Degraded operators.${NC}"
fi
if [[ -n "$UNAVAILABLE_OPS" ]]; then
    echo -e "  ${RED}✗ Unavailable Operators found:${NC}\n$UNAVAILABLE_OPS"
    ((ERRORS++))
else
    echo -e "  ${GREEN}✓ All operators are Available.${NC}"
fi
# 4. Check MachineConfigPools (MCP)
echo "[4/7] Validating MachineConfigPools (MCP)..."
DEGRADED_MCP=$(oc get mcp -o jsonpath='{.items[?(@.status.conditions[?(@.type=="Degraded")].status=="True")].metadata.name}')
PAUSED_MCP=$(oc get mcp -o jsonpath='{.items[?(@.spec.paused==true)].metadata.name}')
if [[ -n "$DEGRADED_MCP" ]]; then
    echo -e "  ${RED}✗ Degraded MachineConfigPools:${NC} $DEGRADED_MCP"
    ((ERRORS++))
else
    echo -e "  ${GREEN}✓ All MachineConfigPools healthy.${NC}"
fi
if [[ -n "$PAUSED_MCP" ]]; then
    echo -e "  ${YELLOW}! Warning: Paused MachineConfigPools detected:${NC} $PAUSED_MCP"
fi
# 5. Check Node Readiness
echo "[5/7] Checking Node status across cluster..."
NOT_READY_NODES=$(oc get nodes --no-headers | awk '$2 != "Ready" {print $1}')
if [[ -n "$NOT_READY_NODES" ]]; then
    echo -e "  ${RED}✗ Nodes in NotReady state:${NC}\n$NOT_READY_NODES"
    ((ERRORS++))
else
    echo -e "  ${GREEN}✓ All nodes report Ready status.${NC}"
fi
# 6. Audit PodDisruptionBudgets (PDB) for Potential Deadlocks
echo "[6/7] Checking PodDisruptionBudgets (PDBs) for drain lock risks..."
DEADLOCKED_PDBS=$(oc get pdb -A -o json | jq -r '.items[] | select(.status.disruptionsAllowed == 0) | "\(.metadata.namespace)/\(.metadata.name)"')
if [[ -n "$DEADLOCKED_PDBS" ]]; then
    echo -e "  ${YELLOW}! PDBs currently allowing 0 disruptions (May block node drains):${NC}"
    echo "$DEADLOCKED_PDBS" | sed 's/^/    /'
else
    echo -e "  ${GREEN}✓ No blocking PDBs detected.${NC}"
fi
# 7. Check Active Critical Alerts
echo "[7/7] Checking Prometheus Alerts..."
FIRING_CRITICALS=$(oc get alerts -A -o json 2>/dev/null | jq -r '.items[] | select(.status.state=="firing" and .labels.severity=="critical") | .labels.alertname' || echo "")
if [[ -n "$FIRING_CRITICALS" ]]; then
    echo -e "  ${RED}✗ Firing Critical Alerts:${NC}\n$FIRING_CRITICALS"
    ((ERRORS++))
else
    echo -e "  ${GREEN}✓ Zero firing critical alerts.${NC}"
fi
# Final Summary
echo -e "\n${YELLOW}====================================================${NC}"
if [[ $ERRORS -eq 0 ]]; then
    echo -e "${GREEN} SUCCESS: Cluster passed all health checks. Safe to proceed with upgrade.${NC}"
    echo -e "${YELLOW}====================================================${NC}"
    exit 0
else
    echo -e "${RED} FAILURE: Found $ERRORS issue(s). Resolve all errors before upgrading.${NC}"
    echo -e "${YELLOW}====================================================${NC}"
    exit 1
fi

This is one of the hardest OpenShift interview questions. It tests whether you understand etcd, static pods, Machine Config Operator, Operators, backups, and disaster recovery.

Scenario: All three control plane (master) nodes are lost due to a datacenter failure, storage corruption, or accidental deletion. Worker nodes still exist, but the Kubernetes API is unavailable.

A strong answer should emphasize that the critical asset is the etcd backup. Without a valid etcd backup, you cannot restore the cluster’s Kubernetes state (objects such as Deployments, Secrets, ConfigMaps, CRDs, Routes, etc.).

Recovery Strategy Overview

                Disaster
                    │
                    ▼
        All Control Plane Lost
                    │
                    ▼
     Provision New Control Plane VMs
                    │
                    ▼
      Install Matching RHCOS Version
                    │
                    ▼
 Restore etcd from Snapshot + Static Pod Resources
                    │
                    ▼
 API Server Available Again
                    │
                    ▼
 Control Plane Operators Recover
                    │
                    ▼
 Workers Reconnect
                    │
                    ▼
 Applications Recover

Step 1 – Assess the Failure

First determine:

• Is etcd data still available?
• Were all control plane VMs lost?
• Are worker nodes still running?
• Is the load balancer intact?
• Is the infrastructure (DNS, storage, networking) still available?

Remember:

Workers do not contain the cluster state.

The authoritative cluster state is stored in etcd.

Step 2 – Verify Backups

OpenShift supports backing up:

Platform

• etcd snapshot
• static pod resources (required alongside the snapshot)

Applications

• OADP/Velero
• CSI snapshots
• Database-native backups

A typical backup includes:

etcd snapshot
+
static_kuberesources.tar.gz

The static Kubernetes resources archive contains the manifests and certificates needed by the control plane.

Step 3 – Rebuild the Infrastructure

Provision replacement control plane nodes with:

• Same OpenShift version
• Same RHCOS version
• Similar CPU/RAM sizing
• Correct networking
• Same DNS names (or update infrastructure accordingly)

Example:

master-1
master-2
master-3

Step 4 – Restore the First Control Plane Node

You restore the cluster from one control plane node first.

Typical high-level process:

• Boot RHCOS.
• Place the etcd snapshot and static resources on the node.
• Use the documented OpenShift restore procedure (cluster-restore.sh in supported versions) to restore etcd and recreate the static control plane components.
• Start the control plane services.

This restores:

• Kubernetes objects
• Secrets
• ConfigMaps
• CRDs
• RBAC
• Routes
• Operator state

Step 5 – Recover the API Server

Once etcd is restored:

etcd
↓
API Server
↓
Controller Manager
↓
Scheduler

The API should become available again.

Verify:

oc get nodes

Initially, only the restored control plane may appear Ready.

Step 6 – Rejoin Remaining Control Plane Nodes

Provision the remaining control plane nodes so they join the restored cluster.

OpenShift rebuilds:

• kube-apiserver
• controller-manager
• scheduler
• etcd members

One node at a time until the control plane regains quorum and high availability.

Step 7 – Cluster Operators Reconcile

After the API is available, Operators begin reconciling automatically.

Examples:

• Authentication Operator
• DNS Operator
• Monitoring Operator
• Network Operator
• Ingress Operator
• Image Registry Operator
• Machine Config Operator

Check:

oc get co

The goal is:

Available=True
Progressing=False
Degraded=False

Step 8 – Worker Nodes Reconnect

Worker kubelets continuously attempt to reconnect.

Worker
↓
API Server Restored
↓
TLS Authentication
↓
Node Ready

Verify:

oc get nodes

Step 9 – Restore Applications (If Needed)

If persistent storage or application data was also lost:

Restore using:

• OADP / Velero
• CSI snapshots
• Database backups

Typical order:

1. Storage
2. Databases
3. Stateful applications
4. Stateless applications

Step 10 – Validate the Cluster

Check:

oc get clusterversion

oc get co

oc get mcp

oc get nodes

oc get pods -A

Confirm:

• All Operators healthy
• No degraded MachineConfigPools
• Nodes Ready
• Applications running
• Routes responding

What Happens Internally?

When etcd is restored:

etcd Snapshot
↓
API Objects
↓
Deployments
↓
ReplicaSets
↓
Pods
↓
Services
↓
Routes

Kubernetes controllers and Operators recreate the runtime state from the restored desired state.

If Workers Were Running During the Outage

During API downtime:

• Existing containers generally continue running.
• kubelets continue managing local pods.
• No new scheduling occurs.
• No configuration changes can be applied.
• Controllers cannot reconcile state.

After the API returns:

API Available
↓
Workers Reconnect
↓
Status Updated
↓
Normal Scheduling Resumes

What If etcd Is Lost and No Backup Exists?

This is effectively a cluster rebuild.

You can recreate:

• Control plane
• Worker nodes
• Operators

However, you cannot recover Kubernetes objects such as:

• Deployments
• Secrets
• ConfigMaps
• Routes
• CRDs
• RBAC
• Persistent resource definitions

Applications must be redeployed from GitOps, manifests, Helm charts, or other deployment artifacts, and application data must come from separate backups.

Best Practices

• Back up etcd regularly and verify the backups.
• Store etcd snapshots off-cluster and off-site.
• Back up the required static pod resources together with the snapshot.
• Use OADP/Velero for application-level backup and recovery.
• Test disaster recovery procedures periodically in a non-production environment.
• Maintain infrastructure as code (Terraform/Ansible) to rebuild the underlying infrastructure consistently.
• Use GitOps (for example, Argo CD) so application manifests can be redeployed quickly after platform recovery.

Interview Answer (2-Minute Version)

“If all control plane nodes are lost, my first priority is to recover the Kubernetes control plane from a valid etcd snapshot and the associated static pod resources. I would provision replacement control plane nodes running the same OpenShift and RHCOS versions, restore the etcd snapshot on the first control plane node using Red Hat’s documented restore procedure, and bring the API server back online. After the API is available, I would add the remaining control plane nodes back into the cluster to restore high availability. The Cluster Operators then reconcile the platform automatically, and worker nodes reconnect using their existing kubelet certificates. If application storage was also lost, I would restore it separately using OADP/Velero, CSI snapshots, or database-native backups. Finally, I’d validate ClusterOperators, MachineConfigPools, node health, and application functionality. Without a valid etcd backup, the platform must be rebuilt and Kubernetes objects cannot be recovered, so regular tested backups are essential.”

Platform-level automation in Red Hat OpenShift Container Platform (OCP) represents the core difference between running upstream vanilla Kubernetes and running an enterprise application platform. In a standard Kubernetes environment, platform engineers must build, maintain, and glue together external tools for provisioning infrastructure, managing operating systems, rotating certificates, and scaling compute nodes.

OpenShift treats the entire infrastructure stack as software. It automates the lifecycle of the cluster through a specialized hierarchy of operators, enabling true declarative Day-2 operations.

1. The Automation Hierarchy (The Control Loop)

OpenShift’s platform automation operates on a hierarchical loop. If a lower level drifts or encounters an issue, the higher levels orchestrate the remediation automatically.

 ┌────────────────────────────────────────────────────────┐
 │ 1. Cluster Version Operator (CVO)                      │ ─── Tracks Cluster Version & Top-Level Operators
 └───────────────────────────┬────────────────────────────┘
                             ▼
 ┌────────────────────────────────────────────────────────┐
 │ 2. Machine Config Operator (MCO)                       │ ─── Translates state into OS configurations
 └───────────────────────────┬────────────────────────────┘
                             ▼
 ┌────────────────────────────────────────────────────────┐
 │ 3. Machine API Operator                                │ ─── Provisions/Destroys actual Infrastructure VMs
 └────────────────────────────────────────────────────────┘

• The Brain (CVO): The Cluster Version Operator enforces the exact software footprint of the cluster, checking the central release payload and updating core operators in a strict dependency sequence.
• The OS Configurer (MCO): The Machine Config Operator ensures the underlying operating system (RHCOS) exactly matches the machine configs. It manages node reboots, kernel patches, and file injections via rpm-ostree.
• The Infrastructure Provider (Machine API): The Machine API Operator bridges the gap between the software cluster and the cloud provider (AWS, Azure, GCP, or vSphere), programmatically provisioning physical or virtual hardware resources.

2. Dynamic Infrastructure: The Machine API Operator

In traditional environments, scaling a cluster requires logging into a cloud console, spinning up a VM, configuring the network, running an installation script, and joining it to the cluster.

OpenShift automates this by bringing the concept of Custom Resource Definitions (CRDs) directly to virtual machines via the Machine API.

The Machine API Resource Stack:

• Machine: A declarative definition of a single node (VM or bare-metal). If a Machine object is deleted from the cluster, the Machine API actively calls the cloud provider’s API to terminate the underlying VM instance.
• MachineSet: Similar to a Kubernetes ReplicaSet, a MachineSet maintains a desired count of identical Machine objects. If you change the replica count from 3 to 10, the operator instantly communicates with your infrastructure provider to spin up 7 new instances, configures them with Fedora/Red Hat CoreOS, and provisions them into the cluster cluster data plane.

3. Automated Horizontal Scaling: ClusterAutoscaler and MachineAutoscaler

To truly achieve automated platform operations, you can decouple human intervention from capacity planning by implementing the ClusterAutoscaler.

When a surge of user traffic hits your application, HPA (Horizontal Pod Autoscaler) will scale up your pods. If those pods fail to schedule because your current worker nodes are completely out of CPU or Memory resources, they transition to a Pending state. The autoscaling loop detects this bottleneck and reacts instantly:

Plaintext

 ┌─────────────────────┐       ┌──────────────────────┐       ┌───────────────────────┐
 │ Pods enter PENDING  │ ────► │  ClusterAutoscaler   │ ────► │   MachineAutoscaler   │
 │ Due to No Node Room │       │ Evaluates Cluster Max│       │ Scales Target Match   │
 └─────────────────────┘       └──────────────────────┘       └───────────┬───────────┘
                                                                          │
                                                                          ▼
 ┌─────────────────────┐       ┌──────────────────────┐       ┌───────────────────────┐
 │   Cloud Provider    │ ◄──── │ MachineSet Replicas  │ ◄──── │ Dynamic Machine VM    │
 │ Provisions New VM   │       │  Increments (+1)     │       │ Created in API        │
 └─────────────────────┘       └──────────────────────┘       └───────────────────────┘

The Declarative Implementation

First, you establish a global ClusterAutoscaler limit policy to define the total resource boundaries for the entire cluster fleet:

YAML

apiVersion: autoscaling.openshift.io/v1
kind: ClusterAutoscaler
metadata:
  name: default
spec:
  podPriorityThreshold: -10 # Ensures low-priority batch jobs don't trigger costly node scaling
  resourceLimits:
    maxNodesTotal: 100       # Hard ceiling limit for cluster size expansion
    cores:
      min: 16
      max: 800              # Total CPU capacity safety cap
    memory:
      min: 64
      max: 3200             # Total Memory capacity safety cap
  scaleDown:
    enabled: true           # Scale down and delete nodes when traffic drops to save money
    delayAfterAdd: 10m
    unneededTime: 5m        # How long a node must be completely idle before removal

Next, you map a specific MachineAutoscaler to watch your localized regional MachineSets, granting them the authorization to scale out within those global boundaries:

YAML

apiVersion: autoscaling.openshift.io/v1
kind: MachineAutoscaler
metadata:
  name: ecom-us-east-scaler
  namespace: openshift-machine-api
spec:
  minReplicas: 3
  maxReplicas: 12             # Allows this specific pool to scale up to 12 VMs
  scaleTargetRef:
    apiVersion: machine.openshift.io/v1beta1
    kind: MachineSet
    name: cluster-prod-asdf-worker-us-east-1a # Targeted regional availability zone

4. Bare-Metal Automation: Metal3 and Ironic

While cloud autoscaling is straightforward via vendor APIs, OpenShift also automates physical hardware (Bare-Metal) infrastructure natively. It achieves this using an embedded upstream project named Metal3, combined with OpenStack’s Ironic engine.

The Provisioning Cycle:

1. Discovery: You register the IPMI, iDRAC, or ILO management credentials of raw, un-provisioned physical blade servers into OpenShift as a BareMetalHost custom resource.
2. Power Management: When a new Machine is requested via a bare-metal MachineSet, the Metal3 operator uses IPMI commands to programmatically power on the physical blade server.
3. PXE Boot / Virtual Media: The internal Ironic controller mounts the RHCOS Live ISO directly onto the physical server via virtual media or an internal PXE network network.
4. Flashing the Operating System: The host boots the installer image, writes the core OpenShift Ignition files directly to the server’s raw physical NVMe drives, reboots, configures its local network bonds, and completes registration back to the master control plane entirely over-the-wire without a datacenter technician ever stepping into the server aisle.

5. Automated Day-2 Maintenance Operations

Beyond scaling nodes, OpenShift drives automated operational routines across the cluster lifecycle:

• Certificate Auto-Rotation: The platform contains internal certificate authorities (CAs) that validate communication lines between components like the API server, etcd, and kubelets. These internal certificates have strict expiration windows. OpenShift operators actively monitor these metrics and automatically handle cryptographic CSR generation, signature verification, and hot-renewal execution loops silently in the background before they expire, eliminating manual certificate management outages.
• Self-Healing Descheduler: If an administrator updates a node configuration or nodes get highly crowded over time, a cluster can develop unevenly distributed workloads (e.g., Node 1 is at 95% CPU utilization while Node 2 sits at 15%). The Descheduler Operator automatically audits the active workloads against utilization thresholds and systematically evicts pods from stressed nodes, allowing the default scheduler to re-balance applications across the fleet.

Implementing an ELK Stack (Elasticsearch, Logstash, Kibana) gives you a highly capable platform for log aggregation and analysis. For architecture managing ~20 Linux/Docker servers, you can tie this cleanly into your existing workflow.

A modern production-grade architecture does not typically use Logstash on all 20 nodes because it has a heavy memory footprint. Instead, you deploy lightweight Filebeat agents on your 20 servers to stream data to a central ELK hub.

Part 1: Define the Central ELK Stack

On your central monitoring host, you can deploy the full ELK stack using Portainer or a direct docker-compose.yml file.

1. Configure Host Virtual Memory

Elasticsearch requires memory mapping limits higher than default Linux settings. Before starting the containers, run this on your central host:

sudo sysctl -w vm.max_map_count=262144

To make this setting permanent across server reboots, append vm.max_map_count=262144 to /etc/sysctl.conf.

2. Central docker-compose.yml

Create a directory /opt/elk and save the following file:

YAML

version: '3.8'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.15.0
    container_name: central-elasticsearch
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=true
      - ELASTIC_PASSWORD=SuperSecurePassword123 # Choose a strong password
      - "ES_JAVA_OPTS=-Xms2g -Xmx2g" # Allocates 2GB RAM; adjust to host sizing
    volumes:
      - es-data:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"
    restart: unless-stopped
  logstash:
    image: docker.elastic.co/logstash/logstash:8.15.0
    container_name: central-logstash
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf:ro
    ports:
      - "5044:5044" # Filebeat connection port
    environment:
      - "LS_JAVA_OPTS=-Xms1g -Xmx1g"
    depends_on:
      - elasticsearch
    restart: unless-stopped
  kibana:
    image: docker.elastic.co/kibana/kibana:8.15.0
    container_name: central-kibana
    ports:
      - "5601:5601"
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      - ELASTICSEARCH_USERNAME=elastic
      - ELASTICSEARCH_PASSWORD=SuperSecurePassword123
    depends_on:
      - elasticsearch
    restart: unless-stopped
volumes:
  es-data:

3. Logstash Pipeline Configuration (logstash.conf)

In the same folder (/opt/elk), create logstash.conf. This instructs Logstash to listen for incoming logs from your 20 servers on port 5044 and parse them into Elasticsearch:

Ruby

input {
  beats {
    port => 5044
  }
}
filter {
  if [container][image][name] =~ /nginx/ {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
  }
  # Add other parsing rules/grok filters for system logs here
}
output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    index => "logstash-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "SuperSecurePassword123"
  }
}

Deploy this stack using docker compose up -d or your Portainer Stacks editor.

Part 2: Deploy Filebeat to the 20 Hosts via Ansible

With the central hub running, you need to configure your 20 remote nodes to gather system logs and Docker logs, sending them over the network to Logstash.

1. Filebeat Template File (filebeat.yml.j2)

On your Ansible controller node, create a Jinja2 template file:

YAML

filebeat.inputs:
# Input 1: System syslog and auth logs
- type: log
  enabled: true
  paths:
    - /var/log/syslog
    - /var/log/auth.log
# Input 2: Dynamic Docker container logs
- type: container
  enabled: true
  paths:
    - /var/lib/docker/containers/*/*.log
  processors:
    - add_docker_metadata: ~ # Enriches logs with container names, images, etc.
output.logstash:
  hosts: ["{{ central_logstash_ip }}:5044"]

2. The Ansible Automation Playbook

Create a playbook named deploy-filebeat.yml to automatically install Filebeat across your entire fleet as a native service:

YAML

---
- name: Deploy Filebeat Log Collector
  hosts: docker_servers
  become: true
  vars:
    central_logstash_ip: "192.168.1.10" # Replace with your ELK master server IP
  tasks:
    - name: Download Filebeat Debian package
      ansible.builtin.get_url:
        url: "https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.15.0-amd64.deb"
        dest: "/tmp/filebeat.deb"
        mode: '0644'
    - name: Install Filebeat package
      ansible.builtin.apt:
        deb: "/tmp/filebeat.deb"
        state: present
    - name: Configure Filebeat via Jinja2 Template
      ansible.builtin.template:
        src: filebeat.yml.j2
        dest: /etc/filebeat/filebeat.yml
        owner: root
        group: root
        mode: '0600'
      register: filebeat_config
    - name: Enable and Restart Filebeat Service
      ansible.builtin.systemd:
        name: filebeat
        state: restarted
        enabled: true
        daemon_reload: true
      when: filebeat_config.changed

Execute your playbook to push the log forwarding infrastructure across your servers:

Bash

ansible-playbook -i inventory.ini deploy-filebeat.yml

Part 3: View the Logs in Kibana

1. Open your web browser and navigate to your Kibana portal at http://<central-server-ip>:5601.
2. Log in using the username elastic and the password you defined in your Docker Compose file (SuperSecurePassword123).
3. Navigate to Management -> Stack Management -> Data Views (sometimes labeled Index Patterns).
4. Click Create Data View and define the pattern name as logstash-*. Select @timestamp as your timestamp field.
5. Go to the main menu and click Discover.

You will now see a centralized stream of logs rolling in from your 20 Linux hosts. Because of the add_docker_metadata processor, you can filter logs by specific attributes like container.name, host.name, or container.image.name across your entire network.

Scaling from 20 to 50 Linux servers is a major milestone. While the core architecture pattern (Lightweight Agents $\rightarrow$ Central Queue/Buffer $\rightarrow$ Processing $\rightarrow$ Storage) remains identical, the exact deployment strategy you used for 20 servers will bottle-neck and crash if applied directly to 50 servers.

At 50 servers, you are no longer just handling standard logs; you are running a high-throughput data streaming operation. If 5 servers suddenly experience an app error loop at the same time, they will flood Logstash, exhaust its memory, and cause it to drop logs or crash.

Here is how you must adapt your ELK strategy to safely handle 50 servers.

1. The Architectural Shift: Add a Buffer (Kafka or Redis)

For 20 servers, Filebeat can talk directly to Logstash. For 50 servers, you need to introduce an architecture that includes a Message Queue (like Apache Kafka or Redis) between Filebeat and Logstash.

Why this is mandatory at scale:

• Spike Protection: If your servers generate a sudden burst of millions of log lines, Kafka absorbs the shock. It acts as a shock absorber, writing the logs safely to a temporary disk queue.
• Decoupling: Logstash can now pull logs out of Kafka at its own comfortable pace without getting overwhelmed and crashing.

2. Scale up Elasticsearch Hardware (JVM Heap & Storage)

A single-node Elasticsearch instance running on standard settings will choke on 50 servers. You must adjust your resources:

• Dedicated Production Server: Your ELK Master should be a dedicated machine with at least 32 GB of RAM and fast SSD storage.
• Adjust JVM Heap Size: Update your docker-compose.yml environment variables to give Elasticsearch more memory. A good rule of thumb is giving it 50% of your total system memory, up to 31 GB:YAML- "ES_JAVA_OPTS=-Xms16g -Xmx16g" # Boosted to 16GB for 50 hosts

3. Implement an Index Lifecycle Management (ILM) Policy

50 servers will easily generate 5 GB to 15 GB of raw log data every single day. If you store this indefinitely, Elasticsearch will run out of memory tracking the data indexes.

You must configure a rolling policy in Kibana (Stack Management -> Index Lifecycle Policies) to automatically manage this data lifecycle:

4. Optimize Your Ansible Filebeat Deployments

Your Ansible strategy is still the perfect way to manage 50 hosts, but you need to optimize how Filebeat handles resource boundaries on the target nodes so it doesn’t consume host CPU.

Update your filebeat.yml.j2 template with these enterprise performance tweaks:

YAML

filebeat.inputs:
- type: container
  enabled: true
  paths:
    - /var/lib/docker/containers/*/*.log
  # Performance Tuning for 50+ Hosts:
  backoff: 1s                  # How long to wait before checking a file again after reaching EOF
  max_backoff: 10s
  harvester_buffer_size: 16384 # 16KB memory buffer per log file
  queue.mem:
    events: 4096               # Buffer logs in local memory before shipping over the network
    flush.min_events: 512
    flush.timeout: 5s
# If you implemented a queue, point your hosts to Kafka instead of Logstash
output.kafka:
  hosts: ["192.168.1.15:9092"]
  topic: 'docker-logs'
  partition.round_robin:
    reachable_only: false
  required_acks: 1

Summary Checklist for Scaling to 50 Hosts

1. Do not use a single-node setup without bumping the RAM heap size to at least 16 GB.
2. Deploy Redis or Kafka as a buffer if your logs are business-critical and cannot tolerate dropped lines during traffic spikes.
3. Automate Index Deletion from day one so your central cluster doesn’t experience a storage failure in month two.