Top Linux Interview Questions for All Levels

Preparing for a Linux interview can vary wildly depending on whether you are looking at an entry-level helpdesk role, a DevOps position, or a senior Systems Administrator gig.

To give you the best foundation, here is a breakdown of the most frequently asked Linux interview questions, categorized by difficulty and topic.

1. Fundamentals & Concept Questions

Q: What is the difference between a Hard Link and a Soft (Symbolic) Link?
  • Hard Link: A hard link is a direct pointer to the physical data (the inode) on the disk. If you delete the original file, the hard link still works and can access the data. They cannot cross different filesystems or point to directories.
  • Soft Link: Essentially a shortcut. It points to the file name rather than the underlying inode. If you delete the original file, the soft link becomes “broken” (dangling). They can cross filesystems and point to directories.
Q: Can you explain the Linux Boot Sequence?

Interviewers love this because it proves you know how the OS operates under the hood. The core steps are:

  1. BIOS/UEFI: Performs POST (Power-On Self-Test) and loads the MBR/EFI boot sector.
  2. GRUB (Bootloader): Allows you to select the kernel and loads it into memory.
  3. Kernel: Initializes hardware and mounts the root filesystem (/).
  4. systemd / Init: The primary process (PID 1) starts up, managing system services, targets, and daemons.
Q: What is an Inode, and what happens if a disk runs out of them?

An inode (index node) is a data structure that stores metadata about a file (size, owner, permissions, location on disk), but not the file name or actual data.

Interview Tip: If a system runs out of inodes (which you check via df -i), you cannot create new files, even if df -h shows you have gigabytes of free disk space.

2. Practical Troubleshooting Scenario Questions

Q: “The server is running incredibly slow. What are the first three commands you run?”

This tests your live triage methodology. A great baseline answer is:

  • top (or htop): To instantly see CPU/Memory usage and identify rogue, resource-heavy processes.
  • free -m: To check if the system is running out of physical RAM and aggressively using Swap space, which destroys performance.
  • uptime: To look at the system load averages over the last 1, 5, and 15 minutes.
Q: “An application can’t bind to its port, or users can’t connect. How do you troubleshoot?”
  1. Check if the service is actually running and listening: ss -tulpn or netstat -tulnp.
  2. Check for local firewall restrictions: iptables -L or ufw status / firewall-cmd --list-all.
  3. Check security enforcement modules like SELinux or AppArmor: run getenforce to see if it’s blocking the application.
Q: “How do you securely check logs for a specific service failing in systemd?”

Instead of digging around blindly in /var/log/, use:

Bash

journalctl -u nginx.service -n 50 --no-pager

(Replacing nginx.service with the failing service. -n 50 pulls the last 50 lines).

3. High-Frequency Commands Cheat Sheet

Be ready to explain or write out these daily-use commands:

Scenario / ObjectiveCommand to Use
Search text recursively inside a directorygrep -ri "error_pattern" /var/log/
Follow a log file in real-timetail -f /var/log/syslog
Find large files eating up spacedu -sh * | sort -rh
Change file permissions safelychmod 755 filename or chmod u+rwx,go+rx
Check which process is using a filelsof /path/to/file
See detailed block storage layoutlsblk or fdisk -l

4. Advanced / DevOps Linux Questions

Q: What is the difference between a Process and a Thread in Linux?

In Linux, the kernel doesn’t strictly differentiate between them the way other operating systems do. Both are represented by task_struct. However:

  • A Process gets its own isolated memory space.
  • A Thread is a lightweight unit of execution created inside a process that shares the parent process’s memory, file descriptors, and environment.
Q: What is Zombie Process, and how do you kill it?

A zombie process is a process that has completed execution but still has an entry in the process table because its parent process hasn’t read its exit status yet (via wait()).

The Catch: You cannot kill a zombie process using kill -9 PID because it is already technically dead. To get rid of it, you must either kill its parent process or restart the service owning it.

Top Linux Interview Questions for Aspiring Admins

Here are key Linux interview questions organized by topic:

Core Concepts

What is the difference between a process and a thread? A process is an independent program with its own memory space; a thread is a lightweight unit within a process sharing the same memory.

What are inodes? Inodes store metadata about files (permissions, ownership, timestamps, disk block locations) but not the filename or file data itself.

Explain the boot process. BIOS/UEFI → Bootloader (GRUB) → Kernel loads → init/systemd starts → runlevel/target reached.


File System

  • Hard link vs soft link — Hard links point to the same inode; symlinks point to a path. Deleting the original breaks a symlink but not a hard link.
  • /proc filesystem — A virtual FS exposing kernel and process info in real time (e.g., /proc/cpuinfo, /proc/meminfo).
  • /etc/fstab — Defines filesystems to mount at boot.

Permissions & Users

  • chmod 755 — Owner: rwx, Group: r-x, Others: r-x
  • setuid bit — Runs executable as the file owner (e.g., passwd runs as root).
  • sudo vs susudo runs a single command as root; su switches the entire session to another user.

Processes & Performance

  • ps aux vs topps aux is a snapshot; top is real-time.
  • Zombie process — A process that has finished but its parent hasn’t called wait() to read its exit status.
  • nice / renice — Set/change process scheduling priority (-20 highest, 19 lowest).
  • Check memory usagefree -h, vmstat, /proc/meminfo

Networking

  • netstat -tulnp / ss -tulnp — List open ports and listening services.
  • iptables vs firewalld — Both manage the kernel netfilter; firewalld is a dynamic wrapper with zones.
  • Check routing tableip route or route -n
  • /etc/resolv.conf — Configures DNS nameservers.

Common Commands

TaskCommand
Find files by namefind / -name "file.txt"
Search inside filesgrep -r "pattern" /dir
Disk usagedf -h / du -sh *
Archive & compresstar -czf file.tar.gz dir/
Live log watchingtail -f /var/log/syslog
Schedule taskscrontab -e

systemd

  • systemctl status/start/stop/enable — Manage services.
  • Units — Services (.service), timers (.timer), mounts (.mount), etc.
  • journalctl -u nginx -f — Follow logs for a specific unit.

Shell Scripting

  • $? — Exit code of the last command (0 = success).
  • $@ vs $* — Both hold script arguments; $@ treats each as a separate word, $* treats all as one string.
  • 2>/dev/null — Redirect stderr to discard errors.

Tricky / Advanced Questions

  • What happens when you type a command in the shell? Shell forks a child process → exec() replaces it with the command → parent waits.
  • OOM killer — Kernel kills processes when RAM is exhausted; logs in dmesg.
  • strace — Traces system calls made by a process, useful for debugging.
  • lsof — Lists open files and which processes have them open.

OpenShift Troubleshooting: Essential OC Commands

OC Commands for OpenShift (OCP) Troubleshooting

Here’s a comprehensive reference organized by troubleshooting category:


Cluster & Node Health

# Cluster version and status
oc get clusterversion
oc describe clusterversion
# Node status
oc get nodes
oc get nodes -o wide
oc describe node <node-name>
# Node resource usage
oc adm top nodes
# Check cluster operators
oc get clusteroperators
oc get co # shorthand

Pod Troubleshooting

# List pods (all namespaces)
oc get pods -A
oc get pods -n <namespace> -o wide
# Pod details & events
oc describe pod <pod-name> -n <namespace>
# Logs
oc logs <pod-name> -n <namespace>
oc logs <pod-name> -n <namespace> --previous # crashed container
oc logs <pod-name> -n <namespace> -c <container> # specific container
oc logs <pod-name> -n <namespace> --tail=100 -f # follow last 100 lines
# Exec into pod
oc exec -it <pod-name> -n <namespace> -- /bin/bash
# Pod resource usage
oc adm top pods -n <namespace>

Deployment / DeploymentConfig

# Status
oc get deployments -n <namespace>
oc get dc -n <namespace> # DeploymentConfig
oc rollout status deployment/<name> -n <namespace>
# Rollout history & rollback
oc rollout history deployment/<name> -n <namespace>
oc rollout undo deployment/<name> -n <namespace>
# Scale
oc scale deployment/<name> --replicas=3 -n <namespace>
# Restart (rolling)
oc rollout restart deployment/<name> -n <namespace>

Networking & Routes

# Services
oc get svc -n <namespace>
oc describe svc <service-name> -n <namespace>
# Routes
oc get routes -n <namespace>
oc describe route <route-name> -n <namespace>
# Endpoints (checks pod–service binding)
oc get endpoints -n <namespace>
# Network policies
oc get networkpolicy -n <namespace>

RBAC & Permissions

# Check what a user can do
oc auth can-i <verb> <resource> --as=<user> -n <namespace>
oc auth can-i --list --as=<user> -n <namespace>
# Role bindings
oc get rolebindings -n <namespace>
oc get clusterrolebindings | grep <user>
# Service account permissions
oc get sa -n <namespace>
oc describe sa <sa-name> -n <namespace>
# Add role to user
oc adm policy add-role-to-user <role> <user> -n <namespace>
oc adm policy add-cluster-role-to-user <role> <user>

Events & Alerts

# Namespace events (sorted by time)
oc get events -n <namespace> --sort-by='.lastTimestamp'
# Warning events only
oc get events -n <namespace> --field-selector type=Warning
# Cluster-wide events
oc get events -A --sort-by='.lastTimestamp'

Storage / PVCs

# PVCs and PVs
oc get pvc -n <namespace>
oc get pv
oc describe pvc <pvc-name> -n <namespace>
# Storage classes
oc get storageclass

etcd & Control Plane

# etcd pod health
oc get pods -n openshift-etcd
oc logs -n openshift-etcd <etcd-pod-name> -c etcd
# API server
oc get pods -n openshift-kube-apiserver
oc get pods -n openshift-apiserver
# Scheduler & controller manager
oc get pods -n openshift-kube-scheduler
oc get pods -n openshift-kube-controller-manager

MachineConfig & Node Config

# MachineConfig pools
oc get mcp
oc describe mcp worker
# MachineConfigs
oc get mc
oc describe mc <mc-name>
# Machine health check
oc get machinehealthcheck -n openshift-machine-api
oc get machines -n openshift-machine-api

Resource Inspection (General)

# Get all resources in a namespace
oc get all -n <namespace>
# Output as YAML/JSON
oc get pod <pod-name> -n <namespace> -o yaml
oc get pod <pod-name> -n <namespace> -o json
# jsonpath query
oc get pods -n <namespace> -o jsonpath='{.items[*].metadata.name}'
# Watch resources live
oc get pods -n <namespace> -w
# Diff live vs local config
oc diff -f <manifest.yaml>

Quick Fixes

# Force delete stuck pod
oc delete pod <pod-name> -n <namespace> --grace-period=0 --force
# Cordon / uncordon node
oc adm cordon <node-name>
oc adm uncordon <node-name>
# Drain node
oc adm drain <node-name> --ignore-daemonsets --delete-emptydir-data
# Mark node schedulable
oc adm manage-node <node-name> --schedulable=true

Pro tip: Add -o wide to most get commands for extra detail, and --sort-by='.lastTimestamp' to events for chronological ordering. Use oc explain <resource> to explore any resource’s field schema inline.

Ansible AWX vs AAP: Key Differences for DevOps

Ansible AWX vs AAP, Terraform Integration & CI/CD Pipelines


1. Ansible AWX vs AAP

What is AWX?

AWX is the open-source upstream project for Ansible Automation Platform — think of it like Fedora is to RHEL. It provides a web UI, REST API, and job scheduling on top of Ansible.

What is AAP?

AAP (Ansible Automation Platform) is the Red Hat enterprise product built from AWX — adds support, stability, Automation Hub, EDA, and a hardened release cycle.

Comparison Table
FeatureAWXAAP
CostFree / open sourcePaid subscription
SupportCommunity onlyRed Hat support
Release cycleRapid / unstableStable / tested
Automation HubNoYes (certified content)
Event Driven AnsibleLimitedFull EDA component
Execution EnvironmentsYesYes + certified EEs
Upgrade pathManual / complexSupported upgrade
RBACBasicAdvanced
AnalyticsNoYes (Automation Analytics)
Air-gap installDifficultSupported
Best forLabs / dev / learningEnterprise production

AAP Architecture — Full Picture
┌─────────────────────────────────────────────────────────────┐
│ Ansible Automation Platform │
│ │
│ ┌─────────────────┐ ┌──────────────────────────────┐ │
│ │ Automation Hub │ │ Automation Controller │ │
│ │ │ │ │ │
│ │ - Certified │ │ ┌────────────────────────┐ │ │
│ │ Collections │ │ │ Job Templates │ │ │
│ │ - Private repo │◄────┤ │ Workflows │ │ │
│ │ - Sync from │ │ │ Schedules │ │ │
│ │ Galaxy │ │ │ Credentials │ │ │
│ └─────────────────┘ │ │ Inventories │ │ │
│ │ │ Projects (SCM) │ │ │
│ ┌─────────────────┐ │ └────────────────────────┘ │ │
│ │ Event Driven │ └──────────────────────────────┘ │
│ │ Ansible (EDA) │ │ │
│ │ │ ┌─────────────▼────────────────┐ │
│ │ - Rulebooks │ │ Execution Environments │ │
│ │ - Event sources │ │ (containerized Ansible) │ │
│ │ - Kafka/webhook │ └──────────────────────────────┘ │
│ └─────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ Automation Analytics (optional) │ │
│ │ Job history, savings calculator, host metrics │ │
│ └──────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘

AWX / AAP Key Concepts Deep Dive
Organizations & RBAC
Organization
├── Teams
│ ├── Team: Developers
│ │ └── Role: Execute (Job Templates only)
│ └── Team: Ops
│ └── Role: Admin (all resources)
├── Users
├── Inventories
├── Projects
├── Job Templates
└── Credentials
Credential Types
# Built-in credential types:
Machine # SSH key/password for managed nodes
Source Control # Git username/token/SSH key
Vault # Ansible Vault password
AWS # Access key + secret
GCP # Service account JSON
Azure # Client ID + secret
OpenShift/K8s # Kubeconfig / bearer token
HashiCorp Vault # AppRole / token
CyberArk # API credentials
Job Template — Key Fields
Name: Deploy App
Inventory: Production Inventory
Project: App Deployment (Git)
Playbook: deploy.yml
Credentials:
- Machine: prod-ssh-key
- Vault: prod-vault-pass
- AWS: prod-aws-creds
Extra Vars:
env: production
version: "{{ tower_job_id }}"
Verbosity: 1 (Verbose)
Job Tags: deploy,config
Limit: webservers # Run only on webservers group
Concurrent: false # Prevent parallel runs
Timeout: 3600
Survey: enabled # Runtime input form
Workflow Job Templates
┌─────────────┐ ┌──────────────┐ ┌─────────────────┐
│ Run Tests │────►│ Deploy App │────►│ Smoke Tests │
└─────────────┘ └──────────────┘ └────────┬────────┘
│ │
on_failure on_success
│ │
┌──────▼──────┐ ┌────────▼────────┐
│ Rollback │ │ Notify Slack │
└─────────────┘ └─────────────────┘
# Workflow node conditions:
on_success # Run next node if this one succeeds
on_failure # Run next node if this one fails
always # Run next node regardless of result
Surveys — Runtime Input
# Survey fields on a Job Template
- variable: app_version
question_name: "App Version to Deploy"
type: text
required: true
default: "latest"
- variable: environment
question_name: "Target Environment"
type: multiplechoice
choices:
- dev
- staging
- production
required: true
- variable: replica_count
question_name: "Number of Replicas"
type: integer
min: 1
max: 10
default: 2

AAP REST API — Automation

# Launch a job template via API
curl -X POST \
https://aap.example.com/api/v2/job_templates/42/launch/ \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"extra_vars": {
"env": "production",
"version": "2.1.0"
},
"limit": "webservers"
}'
# Check job status
curl -X GET \
https://aap.example.com/api/v2/jobs/1234/ \
-H "Authorization: Bearer $TOKEN"
# Get job output
curl -X GET \
https://aap.example.com/api/v2/jobs/1234/stdout/?format=txt \
-H "Authorization: Bearer $TOKEN"

2. Ansible + Terraform Integration
Why Combine Ansible and Terraform?
┌────────────────────────────────────────────────────────┐
│ Infrastructure Lifecycle │
│ │
│ ┌───────────────┐ ┌──────────────────────┐ │
│ │ Terraform │ │ Ansible │ │
│ │ │ │ │ │
│ │ - Provision │─────────►│ - Configure OS │ │
│ │ VMs/cloud │ │ - Install software │ │
│ │ - Networks │ │ - Deploy apps │ │
│ │ - Storage │ │ - Manage users │ │
│ │ - Kubernetes │ │ - Day-2 operations │ │
│ │ │ │ │ │
│ │ "Build it" │ │ "Configure it" │ │
│ └───────────────┘ └──────────────────────┘ │
│ │ │ │
│ └────────────────────────────┘ │
│ Works together │
└────────────────────────────────────────────────────────┘
Pattern 1 — Terraform Provisions, Ansible Configures
# main.tf — Terraform creates EC2 instances
resource "aws_instance" "web" {
count = 3
ami = "ami-0c55b159cbfafe1f0"
instance_type = "t3.medium"
tags = {
Name = "web-${count.index + 1}"
Role = "webserver"
Environment = "production"
}
# Run Ansible after instance is created
provisioner "local-exec" {
command = <<-EOT
sleep 30 # Wait for SSH
ansible-playbook \
-i '${self.public_ip},' \
-u ec2-user \
--private-key ~/.ssh/prod-key.pem \
playbooks/configure-webserver.yml \
--extra-vars "env=production"
EOT
}
}
# Output IPs for Ansible dynamic inventory
output "web_ips" {
value = aws_instance.web[*].public_ip
}
Pattern 2 — Terraform Output → Ansible Inventory
# outputs.tf
output "web_servers" {
value = {
for idx, instance in aws_instance.web :
"web-${idx + 1}" => {
ip = instance.public_ip
private = instance.private_ip
az = instance.availability_zone
}
}
}
#!/usr/bin/env python3
# dynamic_inventory.py — reads Terraform state for Ansible inventory
import json
import subprocess
import sys
def get_terraform_output():
result = subprocess.run(
["terraform", "output", "-json"],
capture_output=True, text=True
)
return json.loads(result.stdout)
def build_inventory():
tf_output = get_terraform_output()
web_servers = tf_output["web_servers"]["value"]
inventory = {
"webservers": {
"hosts": list(web_servers.keys()),
},
"_meta": {
"hostvars": {
name: {
"ansible_host": data["ip"],
"private_ip": data["private"],
"az": data["az"],
"ansible_user": "ec2-user"
}
for name, data in web_servers.items()
}
}
}
return inventory
print(json.dumps(build_inventory()))
# Use dynamic inventory
ansible-playbook -i dynamic_inventory.py configure.yml
Pattern 3 — Ansible Calling Terraform
# playbook — use Ansible to run Terraform
- name: Provision infrastructure with Terraform
hosts: localhost
gather_facts: false
tasks:
- name: Terraform init
community.general.terraform:
project_path: ./terraform
state: present
force_init: true
backend_config:
bucket: my-tf-state
key: prod/terraform.tfstate
region: us-east-1
variables:
environment: production
instance_count: "3"
instance_type: t3.medium
register: tf_result
- name: Show outputs
debug:
var: tf_result.outputs
- name: Add new hosts to inventory
add_host:
name: "{{ item.key }}"
ansible_host: "{{ item.value.ip }}"
groups: webservers
loop: "{{ tf_result.outputs.web_servers.value | dict2items }}"
- name: Configure newly provisioned servers
hosts: webservers
become: true
roles:
- common
- webserver
Pattern 4 — Shared State via Terraform Remote State
# data.tf — read existing Terraform state in another playbook
data "terraform_remote_state" "network" {
backend = "s3"
config = {
bucket = "my-tf-state"
key = "network/terraform.tfstate"
region = "us-east-1"
}
}
# Use VPC ID from network state
resource "aws_instance" "web" {
subnet_id = data.terraform_remote_state.network.outputs.subnet_id
}
# Ansible reading Terraform state directly
- name: Read Terraform state
hosts: localhost
tasks:
- name: Get TF state from S3
amazon.aws.s3_object:
bucket: my-tf-state
object: prod/terraform.tfstate
dest: /tmp/terraform.tfstate
mode: get
- name: Parse state
set_fact:
tf_state: "{{ lookup('file', '/tmp/terraform.tfstate') | from_json }}"
- name: Extract outputs
set_fact:
web_ips: >-
{{ tf_state.outputs.web_ips.value }}

3. CI/CD Pipelines with Ansible

Pipeline Patterns
┌──────────────────────────────────────────────────────────────┐
│ CI/CD Pipeline │
│ │
│ Code Push │
│ │ │
│ ▼ │
│ ┌──────┐ ┌────────┐ ┌──────────┐ ┌────────────────┐ │
│ │ SCM │──►│ CI │──►│ Ansible │──►│ Production │ │
│ │(Git) │ │(build/ │ │(deploy/ │ │ (live env) │ │
│ │ │ │ test) │ │ config) │ │ │ │
│ └──────┘ └────────┘ └──────────┘ └────────────────┘ │
│ │
└──────────────────────────────────────────────────────────────┘

Jenkins + Ansible Pipeline
// Jenkinsfile
pipeline {
agent any
environment {
ANSIBLE_HOST_KEY_CHECKING = 'False'
VAULT_PASSWORD_FILE = credentials('ansible-vault-pass')
}
parameters {
choice(
name: 'ENVIRONMENT',
choices: ['dev', 'staging', 'production'],
description: 'Target environment'
)
string(
name: 'APP_VERSION',
defaultValue: 'latest',
description: 'Version to deploy'
)
}
stages {
stage('Checkout') {
steps {
git branch: 'main',
url: 'https://github.com/myorg/ansible-playbooks.git',
credentialsId: 'github-token'
}
}
stage('Lint & Syntax Check') {
steps {
sh '''
pip install ansible-lint
ansible-lint playbooks/deploy.yml
ansible-playbook playbooks/deploy.yml --syntax-check
'''
}
}
stage('Molecule Tests') {
when {
branch 'main'
}
steps {
sh '''
pip install molecule molecule-docker
cd roles/myapp
molecule test
'''
}
}
stage('Deploy to Dev') {
when {
expression { params.ENVIRONMENT == 'dev' }
}
steps {
sh """
ansible-playbook playbooks/deploy.yml \
-i inventories/dev/hosts.yml \
--vault-password-file ${VAULT_PASSWORD_FILE} \
-e app_version=${params.APP_VERSION} \
-e env=dev
"""
}
}
stage('Deploy to Staging') {
when {
expression { params.ENVIRONMENT == 'staging' }
}
steps {
sh """
ansible-playbook playbooks/deploy.yml \
-i inventories/staging/hosts.yml \
--vault-password-file ${VAULT_PASSWORD_FILE} \
-e app_version=${params.APP_VERSION} \
-e env=staging
"""
}
post {
success {
// Trigger smoke tests
sh 'ansible-playbook playbooks/smoke-tests.yml -i inventories/staging/hosts.yml'
}
}
}
stage('Approval') {
when {
expression { params.ENVIRONMENT == 'production' }
}
steps {
timeout(time: 1, unit: 'HOURS') {
input message: 'Deploy to Production?',
ok: 'Deploy',
submitter: 'ops-team'
}
}
}
stage('Deploy to Production') {
when {
expression { params.ENVIRONMENT == 'production' }
}
steps {
sh """
ansible-playbook playbooks/deploy.yml \
-i inventories/production/hosts.yml \
--vault-password-file ${VAULT_PASSWORD_FILE} \
-e app_version=${params.APP_VERSION} \
-e env=production \
--diff
"""
}
}
}
post {
success {
slackSend channel: '#deployments',
message: "✅ Deploy ${params.APP_VERSION} to ${params.ENVIRONMENT} succeeded"
}
failure {
slackSend channel: '#deployments',
message: "❌ Deploy ${params.APP_VERSION} to ${params.ENVIRONMENT} FAILED"
// Auto rollback on production failure
script {
if (params.ENVIRONMENT == 'production') {
sh 'ansible-playbook playbooks/rollback.yml -i inventories/production/hosts.yml'
}
}
}
}
}

GitLab CI + Ansible
# .gitlab-ci.yml
stages:
- lint
- test
- deploy-dev
- deploy-staging
- deploy-prod
variables:
ANSIBLE_HOST_KEY_CHECKING: "false"
PIP_CACHE_DIR: "$CI_PROJECT_DIR/.pip-cache"
# Reusable anchor for Ansible setup
.ansible_setup: &ansible_setup
image: python:3.11-slim
before_script:
- pip install ansible ansible-lint molecule[docker] --quiet
- ansible-galaxy collection install -r requirements.yml
- echo "$VAULT_PASSWORD" > /tmp/vault_pass
# Lint stage
ansible-lint:
<<: *ansible_setup
stage: lint
script:
- ansible-lint
- ansible-playbook site.yml --syntax-check -i inventories/dev/
# Molecule tests
molecule-test:
<<: *ansible_setup
stage: test
services:
- docker:dind
variables:
DOCKER_HOST: tcp://docker:2376
script:
- |
for role in roles/*/; do
echo "Testing role: $role"
cd "$role"
molecule test
cd -
done
only:
- merge_requests
- main
# Deploy to dev
deploy-dev:
<<: *ansible_setup
stage: deploy-dev
script:
- |
ansible-playbook deploy.yml \
-i inventories/dev/hosts.yml \
--vault-password-file /tmp/vault_pass \
-e app_version=$CI_COMMIT_SHA \
-e env=dev
environment:
name: development
url: https://dev.example.com
only:
- main
# Deploy to staging
deploy-staging:
<<: *ansible_setup
stage: deploy-staging
script:
- |
ansible-playbook deploy.yml \
-i inventories/staging/hosts.yml \
--vault-password-file /tmp/vault_pass \
-e app_version=$CI_COMMIT_TAG \
-e env=staging
environment:
name: staging
url: https://staging.example.com
only:
- tags
# Deploy to production — manual gate
deploy-prod:
<<: *ansible_setup
stage: deploy-prod
script:
- |
ansible-playbook deploy.yml \
-i inventories/production/hosts.yml \
--vault-password-file /tmp/vault_pass \
-e app_version=$CI_COMMIT_TAG \
-e env=production \
--diff
environment:
name: production
url: https://example.com
when: manual # Manual trigger
allow_failure: false
only:
- tags

GitHub Actions + Ansible
# .github/workflows/deploy.yml
name: Deploy with Ansible
on:
push:
branches: [main]
workflow_dispatch:
inputs:
environment:
description: 'Environment'
required: true
type: choice
options: [dev, staging, production]
version:
description: 'Version to deploy'
required: true
default: 'latest'
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Install Ansible
run: pip install ansible ansible-lint
- name: Run ansible-lint
run: ansible-lint
- name: Syntax check
run: ansible-playbook deploy.yml --syntax-check -i inventories/dev/
molecule-test:
runs-on: ubuntu-latest
needs: lint
steps:
- uses: actions/checkout@v4
- name: Install dependencies
run: pip install ansible molecule[docker]
- name: Run Molecule tests
run: |
for role in roles/*/; do
cd "$role" && molecule test && cd -
done
deploy:
runs-on: ubuntu-latest
needs: molecule-test
environment: ${{ github.event.inputs.environment || 'dev' }}
steps:
- uses: actions/checkout@v4
- name: Install Ansible + collections
run: |
pip install ansible
ansible-galaxy collection install -r requirements.yml
- name: Write vault password
run: echo "${{ secrets.VAULT_PASSWORD }}" > /tmp/vault_pass
- name: Write SSH key
run: |
mkdir -p ~/.ssh
echo "${{ secrets.SSH_PRIVATE_KEY }}" > ~/.ssh/id_rsa
chmod 600 ~/.ssh/id_rsa
- name: Deploy
run: |
ansible-playbook deploy.yml \
-i inventories/${{ github.event.inputs.environment }}/hosts.yml \
--vault-password-file /tmp/vault_pass \
--private-key ~/.ssh/id_rsa \
-e app_version=${{ github.event.inputs.version }} \
-e env=${{ github.event.inputs.environment }}
- name: Cleanup secrets
if: always()
run: |
rm -f /tmp/vault_pass ~/.ssh/id_rsa

Ansible + AAP in CI/CD Pipeline
# aap_trigger.py trigger AAP job from CI pipeline
import requests
import time
import sys
AAP_URL = "https://aap.example.com"
TOKEN = os.environ["AAP_TOKEN"]
JOB_TEMPLATE_ID = 42
def launch_job(extra_vars):
response = requests.post(
f"{AAP_URL}/api/v2/job_templates/{JOB_TEMPLATE_ID}/launch/",
headers={"Authorization": f"Bearer {TOKEN}"},
json={"extra_vars": extra_vars},
verify=True
)
response.raise_for_status()
return response.json()["id"]
def wait_for_job(job_id, timeout=3600):
start = time.time()
while time.time() - start < timeout:
response = requests.get(
f"{AAP_URL}/api/v2/jobs/{job_id}/",
headers={"Authorization": f"Bearer {TOKEN}"}
)
job = response.json()
status = job["status"]
print(f"Job {job_id} status: {status}")
if status == "successful":
return True
elif status in ["failed", "error", "canceled"]:
return False
time.sleep(15)
raise TimeoutError(f"Job {job_id} timed out")
# Usage in CI
job_id = launch_job({
"env": "production",
"version": os.environ["APP_VERSION"]
})
success = wait_for_job(job_id)
sys.exit(0 if success else 1)

Deployment Strategies with Ansible
Rolling Deploy
- name: Rolling deployment
hosts: webservers
serial: 1 # One host at a time
# serial: "25%" # Or 25% at a time
max_fail_percentage: 0 # Stop on any failure
pre_tasks:
- name: Remove from load balancer
uri:
url: "http://lb.example.com/api/drain/{{ inventory_hostname }}"
method: POST
delegate_to: localhost
roles:
- deploy-app
post_tasks:
- name: Health check
uri:
url: "http://{{ ansible_host }}:8080/health"
status_code: 200
retries: 5
delay: 10
- name: Re-add to load balancer
uri:
url: "http://lb.example.com/api/enable/{{ inventory_hostname }}"
method: POST
delegate_to: localhost
Blue-Green Deploy
- name: Blue-Green deployment
hosts: localhost
vars:
active_env: "{{ lookup('file', '/tmp/active_env') | default('blue') }}"
new_env: "{{ 'green' if active_env == 'blue' else 'blue' }}"
tasks:
- name: Deploy to inactive environment
include_role:
name: deploy-app
vars:
target_hosts: "{{ new_env }}_servers"
app_version: "{{ deploy_version }}"
- name: Smoke test new environment
uri:
url: "http://{{ new_env }}.internal/health"
status_code: 200
- name: Switch load balancer to new environment
uri:
url: http://lb.example.com/api/switch
method: POST
body_format: json
body:
active: "{{ new_env }}"
- name: Record new active environment
copy:
content: "{{ new_env }}"
dest: /tmp/active_env

Interview Quick-Fire

  • AWX vs AAP main difference? AWX is open-source/community; AAP is enterprise with Red Hat support, Automation Hub, and EDA.
  • How does AAP store credentials securely? Credentials are encrypted at rest and never exposed in job output — only injected into the execution environment at runtime.
  • What is an Execution Environment? A container image bundling Ansible, collections, and Python dependencies — ensures consistent runtime across nodes.
  • How do you trigger AAP from a CI pipeline? Via the REST API — POST to /api/v2/job_templates/<id>/launch/ with a bearer token.
  • Terraform vs Ansible — which for provisioning? Terraform for cloud infrastructure (stateful, plan/apply). Ansible for configuration management (agentless, push-based). Use both together.
  • What is the Terraform provisioner risk? Provisioners are a last resort — they break idempotency. Prefer separate Ansible invocation after terraform apply.
  • How do you pass Terraform outputs to Ansible? Via dynamic inventory script reading terraform output -json, or add_host module after community.general.terraform task.
  • What is serial in a playbook? Controls rolling update batch size — serial: 1 updates one host at a time; serial: "25%" does 25% at a time.
  • How do you prevent secrets appearing in CI logs? Use Ansible Vault, store passwords in CI secret variables, use no_log: true on sensitive tasks.
# no_log example
- name: Set database password
mysql_user:
name: appuser
password: "{{ db_password }}"
no_log: true # Suppresses task output in logs

Ansible Interview Prep: Questions by Difficulty

Here is a curated list of Ansible interview questions, broken down by difficulty level, to help you prepare or interview others.

Beginner Level

1. What is Ansible, and how does it differ from other configuration management tools?

Ansible is an open-source automation platform used for configuration management, application deployment, and task automation.

  • Agentless: Unlike Chef or Puppet, which require agent software to be installed on target nodes, Ansible is agentless. It connects via SSH (for Linux/Unix) or WinRM (for Windows) and uses Python on the target machine to execute tasks.
  • Push Architecture: Ansible pushes configurations from a central control node to the managed nodes, whereas many other tools use a pull architecture where clients periodically check a master server for updates.
2. What is an Ansible Inventory?

An inventory is a file (usually in INI or YAML format) that lists the hosts or nodes that Ansible manages. It allows you to organize hosts into groups and assign variables to them.

Example INI Inventory:

Ini, TOML

[webservers]
web1.example.com
web2.example.com
[dbservers]
db1.example.com
3. What is the difference between an Ansible Playbook and a Task?
  • Task: The smallest unit of action in Ansible. It defines a single operation to be executed, like installing a package or starting a service, usually by calling an Ansible module.
  • Playbook: A YAML file containing one or more “plays.” A play maps a group of hosts from the inventory to a specific set of tasks.

Intermediate Level

4. What are Ansible Handlers, and when should you use them?

Handlers are special tasks that only run when triggered by a notify directive from another task, and only if that task actually changed something on the managed node. They are commonly used to restart services after a configuration file is updated.

Note: Handlers run once at the very end of a play, ensuring a service is only restarted once, even if multiple tasks notified it.

YAML

tasks:
- name: Copy Nginx configuration
ansible.builtin.copy:
src: nginx.conf
dest: /etc/nginx/nginx.conf
notify: Restart Nginx
handlers:
- name: Restart Nginx
ansible.builtin.service:
name: nginx
state: restarted
5. What is the difference between variable_files and Ansible Vault?
  • Variable Files: Standard YAML files used to store non-sensitive variables (like port numbers, usernames, or domain names) to keep playbooks clean and reusable.
  • Ansible Vault: A feature that allows you to encrypt sensitive data (like passwords, API keys, and SSL certificates) within your playbooks or variable files. Encrypted files can be safely committed to version control (like Git).
6. Explain the difference between ad-hoc commands and playbooks.
  • Ad-hoc Commands: Quick, one-liner commands used to perform a single task across one or many nodes without saving it for later (e.g., ansible all -m ping). Great for quick checks.
  • Playbooks: Structured YAML files used for complex, repeatable, and multi-step deployments that need to be version-controlled and reused.

Advanced Level

7. How do you handle secrets and sensitive data in CI/CD pipelines using Ansible?

When running Ansible in an automated pipeline (like Jenkins, GitLab CI, or GitHub Actions), you can manage secrets by:

  1. Storing the Ansible Vault password as a protected environment variable or secret in the CI/CD platform.
  2. Passing the password to the playbook run using --vault-password-file pointing to a temporary file, or using a script that echoes the environment variable.
  3. Integrating Ansible with external secret managers like HashiCorp Vault, CyberArk, or AWS Secrets Manager using Ansible lookup plugins.
8. What is the difference between include_tasks and import_tasks?

This is a classic architectural question regarding dynamic vs. static reuse:

  • import_tasks (Static): Happens at playbook parsing time. Ansible pre-processes all tasks before the playbook runs. You cannot use loops with import_tasks, and variables defined earlier in the play might not be available during parsing.
  • include_tasks (Dynamic): Happens at playbook execution time, as the play encounters the statement. This allows you to use loops, conditional statements (when), and runtime variables to decide whether or not to include the tasks.
9. What are Ansible Collections, and how do they differ from Roles?
  • Ansible Roles: A structured way to bundle tasks, variables, handlers, templates, and modules together to make them easily reusable.
  • Ansible Collections: Introduced in Ansible 2.9, collections are a broader distribution format. They can contain multiple roles, custom modules, plugins (like lookup or filter plugins), and playbooks. Collections allow third-party vendors (like AWS, Cisco, or Red Hat) to update their Ansible content independently of the core Ansible release cycle.
10. How can you optimize Ansible performance for a large infrastructure (hundreds of nodes)?
  • Increase Forks: Adjust the forks parameter in ansible.cfg (default is 5) to increase the number of parallel processes Ansible spawns.
  • Enable Pipelining: Turn on pipelining = True in ansible.cfg. This reduces the number of SSH operations required to execute a module by executing it directly in memory without transferring files.
  • Use Mitogen: A third-party plugin for Ansible that replaces the default execution strategy with a highly optimized one, often speeding up runs by 1.5x to 3x.
  • Disable Fact Gathering: If your tasks don’t rely on system facts (like OS version or IP addresses), set gather_facts: false in your playbook to save significant time.

Top Ansible Interview Questions You Need to Know

Ansible Interview Questions — Comprehensive Guide


1. Basic / Fundamental Questions

Q: What is idempotency in Ansible and why does it matter? Idempotency means running the same playbook multiple times produces the same result without unintended side effects. If nginx is already installed, the yum module won’t reinstall it. This makes Ansible safe to run repeatedly — useful for drift correction and CI/CD pipelines.

Q: What is the difference between a Play and a Playbook?

  • Play — maps a group of hosts to a set of tasks
  • Playbook — a YAML file containing one or more plays
# This entire file = Playbook
- name: Play 1 - Configure webservers # ← Play
hosts: webservers
tasks:
- name: Install nginx
yum:
name: nginx
state: present
- name: Play 2 - Configure databases # ← Play
hosts: dbservers
tasks:
- name: Install postgresql
yum:
name: postgresql
state: present

Q: What is inventory and what types exist?

# Static inventory — INI format
[webservers]
web1.example.com
web2.example.com ansible_user=ec2-user ansible_port=2222
[dbservers]
db1.example.com
db2.example.com
[production:children] # Group of groups
webservers
dbservers
[webservers:vars] # Group variables
http_port=80

# Static inventory — YAML format
all:
children:
webservers:
hosts:
web1.example.com:
ansible_user: ec2-user
web2.example.com:
dbservers:
hosts:
db1.example.com:
db2.example.com:
# Dynamic inventory — script or plugin
ansible-inventory -i aws_ec2.yml --list # AWS dynamic inventory
ansible-inventory -i inventory/ --graph # View inventory tree

2. Variables & Precedence

Q: What is the variable precedence order in Ansible?

From lowest to highest priority (higher overrides lower):

1. Role defaults (roles/x/defaults/main.yml)
2. Inventory file vars
3. Inventory group_vars/all
4. Playbook group_vars/all
5. Inventory group_vars/*
6. Playbook group_vars/*
7. Inventory host_vars/*
8. Playbook host_vars/*
9. Host facts (gathered)
10. Play vars
11. Play vars_prompt
12. Play vars_files
13. Role vars (roles/x/vars/main.yml)
14. Block vars
15. Task vars
16. include_vars
17. set_facts / registered vars
18. Role params
19. Extra vars (-e) ← HIGHEST PRIORITY
# Extra vars always win
ansible-playbook site.yml -e "env=prod db_password=secret"

Q: What are magic variables? Special variables Ansible populates automatically:

- debug:
msg: |
Current host: {{ inventory_hostname }}
Short hostname: {{ inventory_hostname_short }}
All groups: {{ group_names }}
All hosts: {{ groups['all'] }}
Playbook dir: {{ playbook_dir }}
Role path: {{ role_path }}
Hostvars of web1: {{ hostvars['web1']['ansible_default_ipv4'] }}

Q: What is set_fact and when do you use it?

- name: Set derived variable
set_fact:
app_url: "https://{{ ansible_hostname }}:{{ app_port }}/{{ app_path }}"
is_production: "{{ env == 'prod' }}"
cacheable: true # Persists fact across plays in same run

3. Tasks & Modules

Q: What is the difference between command, shell, and raw modules?

ModuleUse CaseNotes
commandRun commands without shell featuresNo pipes, redirects, variables
shellRun commands with shell featuresSupports pipes, &&, |, globs
rawLow-level SSH — no Python neededFor bootstrapping, network devices
# command — safe, no shell interpretation
- command: /usr/bin/systemctl restart nginx
# shell — needed for pipes/redirects
- shell: ps aux | grep nginx | wc -l
# raw — no Python required on target
- raw: apt-get install -y python3

Prefer built-in modules over command/shell for idempotency.

Q: Explain commonly used modules.

# File management
- ansible.builtin.copy:
src: files/nginx.conf
dest: /etc/nginx/nginx.conf
owner: root
group: root
mode: '0644'
- ansible.builtin.template:
src: templates/app.conf.j2
dest: /etc/app/app.conf
- ansible.builtin.file:
path: /var/log/myapp
state: directory # absent | directory | file | link | touch
mode: '0755'
# Package management
- ansible.builtin.yum:
name:
- nginx
- python3
state: present # present | absent | latest
- ansible.builtin.apt:
name: nginx
state: present
update_cache: true
# Service management
- ansible.builtin.service:
name: nginx
state: started # started | stopped | restarted | reloaded
enabled: true
# User management
- ansible.builtin.user:
name: appuser
uid: 1001
groups: wheel
shell: /bin/bash
state: present
# Fetch file from remote
- ansible.builtin.fetch:
src: /var/log/app.log
dest: /local/logs/
flat: true

Q: What is the register keyword? Captures module output into a variable:

- name: Check if file exists
ansible.builtin.stat:
path: /etc/nginx/nginx.conf
register: nginx_conf_stat
- name: Show result
debug:
msg: "File exists: {{ nginx_conf_stat.stat.exists }}"
- name: Only run if file missing
ansible.builtin.copy:
src: nginx.conf
dest: /etc/nginx/nginx.conf
when: not nginx_conf_stat.stat.exists

4. Conditionals, Loops & Filters

Q: How do conditionals work in Ansible?

# Simple condition
- name: Install on RedHat only
yum:
name: nginx
state: present
when: ansible_os_family == "RedHat"
# Multiple conditions
- name: Install on RHEL 8+
yum:
name: nginx
state: present
when:
- ansible_os_family == "RedHat"
- ansible_distribution_major_version | int >= 8
# OR condition
- name: Run on web or proxy nodes
service:
name: nginx
state: started
when: >
inventory_hostname in groups['webservers'] or
inventory_hostname in groups['proxies']
# Check registered result
- name: Restart only if config changed
service:
name: nginx
state: restarted
when: config_result.changed
# Check if variable is defined
- name: Use custom port if defined
debug:
msg: "Port is {{ custom_port }}"
when: custom_port is defined

Q: Explain loops in Ansible.

# Simple loop
- name: Create multiple users
ansible.builtin.user:
name: "{{ item }}"
state: present
loop:
- alice
- bob
- carol
# Loop with dictionaries
- name: Create users with attributes
ansible.builtin.user:
name: "{{ item.name }}"
uid: "{{ item.uid }}"
groups: "{{ item.groups }}"
loop:
- { name: alice, uid: 1001, groups: wheel }
- { name: bob, uid: 1002, groups: docker }
# Loop with index
- name: Show item and index
debug:
msg: "Item {{ ansible_loop.index }}: {{ item }}"
loop: "{{ packages }}"
loop_control:
label: "{{ item }}" # Cleaner output
index_var: idx
# Loop over dict
- name: Set sysctl values
sysctl:
name: "{{ item.key }}"
value: "{{ item.value }}"
loop: "{{ sysctl_settings | dict2items }}"
vars:
sysctl_settings:
vm.swappiness: 10
net.core.somaxconn: 65535
# until loop — retry until condition met
- name: Wait for service to respond
uri:
url: http://localhost:8080/health
status_code: 200
register: result
until: result.status == 200
retries: 10
delay: 5

Q: What are Jinja2 filters and give examples?

vars:
my_list: [3, 1, 4, 1, 5, 9]
my_string: " Hello World "
my_dict: {a: 1, b: 2}
packages: ["nginx", "python3"]
tasks:
- debug:
msg:
# String filters
upper: "{{ my_string | upper }}"
lower: "{{ my_string | lower }}"
trim: "{{ my_string | trim }}"
replace: "{{ my_string | replace('World', 'Ansible') }}"
default: "{{ undefined_var | default('fallback') }}"
# List filters
sorted: "{{ my_list | sort }}"
unique: "{{ my_list | unique }}"
joined: "{{ packages | join(', ') }}"
first: "{{ my_list | first }}"
last: "{{ my_list | last }}"
length: "{{ my_list | length }}"
# Type conversion
int_val: "{{ '42' | int }}"
bool_val: "{{ 'true' | bool }}"
list_val: "{{ my_dict | dict2items }}"
# Math
max_val: "{{ my_list | max }}"
min_val: "{{ my_list | min }}"
# Conditional
ternary: "{{ (env == 'prod') | ternary('production', 'staging') }}"
# Path
basename: "{{ '/etc/nginx/nginx.conf' | basename }}"
dirname: "{{ '/etc/nginx/nginx.conf' | dirname }}"

5. Handlers, Tags & Error Handling

Q: How do handlers work?

tasks:
- name: Update nginx config
template:
src: nginx.conf.j2
dest: /etc/nginx/nginx.conf
notify:
- Reload nginx
- Send alert
- name: Update SSL cert
copy:
src: cert.pem
dest: /etc/ssl/cert.pem
notify: Reload nginx # Same handler — only fires ONCE
handlers:
- name: Reload nginx
service:
name: nginx
state: reloaded
- name: Send alert
uri:
url: https://hooks.slack.com/...
method: POST
body: '{"text": "nginx config updated"}'

Handlers run once at the end of a play, even if notified multiple times.

Q: How do you force handlers to run immediately?

tasks:
- name: Update config
template:
src: app.conf.j2
dest: /etc/app/app.conf
notify: Restart app
- name: Flush handlers now
meta: flush_handlers # Runs handlers immediately here
- name: Run health check # Now runs after restart
uri:
url: http://localhost/health

Q: How do tags work?

tasks:
- name: Install packages
yum:
name: nginx
state: present
tags:
- install
- packages
- name: Configure nginx
template:
src: nginx.conf.j2
dest: /etc/nginx/nginx.conf
tags:
- configure
- nginx
- name: Start nginx
service:
name: nginx
state: started
tags:
- start
- always # 'always' tag runs even with --tags filter
# Run only tagged tasks
ansible-playbook site.yml --tags install
ansible-playbook site.yml --tags "install,configure"
# Skip tagged tasks
ansible-playbook site.yml --skip-tags configure
# List all tags in a playbook
ansible-playbook site.yml --list-tags

Q: How do you handle errors in Ansible?

# Ignore errors and continue
- name: Try to stop service (may not exist)
service:
name: myapp
state: stopped
ignore_errors: true
# Custom failure condition
- name: Run script
command: /usr/local/bin/check-status.sh
register: result
failed_when:
- result.rc != 0
- '"CRITICAL" in result.stdout'
# Custom changed condition
- name: Run idempotent script
command: /usr/local/bin/configure.sh
register: result
changed_when: '"already configured" not in result.stdout'
# Blocks for error handling (try/catch/finally)
- block:
- name: Try risky operation
command: /usr/bin/risky-script.sh
- name: Another task in block
service:
name: myapp
state: started
rescue:
- name: Handle the error
debug:
msg: "Something went wrong: {{ ansible_failed_result }}"
- name: Rollback
command: /usr/bin/rollback.sh
always:
- name: Always run cleanup
file:
path: /tmp/lockfile
state: absent

6. Advanced Topics

Q: What is delegate_to and run_once?

# Run task on a DIFFERENT host
- name: Add to load balancer
uri:
url: http://lb.example.com/api/add
method: POST
body: '{"host": "{{ inventory_hostname }}"}'
delegate_to: localhost # Run on control node
- name: Take DB backup before deploy
command: pg_dump mydb > /backup/pre-deploy.sql
delegate_to: db1.example.com # Run on DB server
# Run only once across all hosts
- name: Create database schema
command: psql -f schema.sql
run_once: true # Runs on first host in play only
delegate_to: db1.example.com

Q: Difference between include_tasks and import_tasks?

import_tasksinclude_tasks
ParsedAt playbook load time (static)At runtime (dynamic)
Supports loopsNoYes
Supports conditionalsLimited (on the import)Full
Tags visibilityTags visible at startTags not visible until runtime
Use caseAlways-needed task filesConditional or looped includes
# import — static, parsed upfront
- import_tasks: tasks/setup.yml # No loop/conditional on the import
# include — dynamic, parsed at runtime
- include_tasks: "tasks/{{ ansible_os_family }}.yml" # Dynamic path OK
when: setup_needed
loop: "{{ environments }}"

Q: What is ansible-pull and when is it used? Reverses the push model — nodes pull their configuration from a Git repo. Used for:

  • Large fleets where push is impractical
  • Nodes behind firewalls
  • Self-provisioning scenarios
ansible-pull -U https://github.com/myorg/ansible-config.git \
-C main \
--inventory localhost, \
local.yml

Q: What is a dynamic inventory plugin and how do you configure one?

# aws_ec2.yml — AWS dynamic inventory
plugin: amazon.aws.aws_ec2
regions:
- us-east-1
- us-west-2
filters:
instance-state-name: running
tag:Environment: production
keyed_groups:
- key: tags.Role
prefix: role
- key: placement.region
prefix: region
hostnames:
- private-ip-address
compose:
ansible_host: private_ip_address
# Test dynamic inventory
ansible-inventory -i aws_ec2.yml --graph
ansible-inventory -i aws_ec2.yml --list

7. Ansible for OpenShift/Kubernetes

Q: How do you manage OCP resources with Ansible?

- name: Manage OpenShift resources
hosts: localhost
collections:
- kubernetes.core
- redhat.openshift
tasks:
- name: Create namespace
kubernetes.core.k8s:
state: present
definition:
apiVersion: v1
kind: Namespace
metadata:
name: my-app
labels:
environment: production
- name: Apply manifest from file
kubernetes.core.k8s:
state: present
src: /path/to/deployment.yaml
- name: Apply from template
kubernetes.core.k8s:
state: present
definition: "{{ lookup('template', 'deployment.j2') }}"
- name: Wait for deployment rollout
kubernetes.core.k8s_rollout_status:
name: my-deployment
namespace: my-app
timeout: 300
- name: Get pod info
kubernetes.core.k8s_info:
kind: Pod
namespace: my-app
label_selectors:
- app=my-app
register: pod_info
- name: Scale deployment
kubernetes.core.k8s_scale:
name: my-deployment
namespace: my-app
replicas: 5

8. Performance & Best Practices

Q: How do you speed up Ansible playbooks?

# ansible.cfg optimizations
[defaults]
# SSH pipelining — reduces SSH connections
pipelining = True
# Parallel execution
forks = 20
# Fact caching — don't re-gather every run
fact_caching = jsonfile
fact_caching_connection = /tmp/ansible_facts
fact_caching_timeout = 3600
[ssh_connection]
# Reuse SSH connections
ssh_args = -o ControlMaster=auto -o ControlPersist=60s
control_path_dir = /tmp/ansible-ssh
# Disable fact gathering when not needed
- hosts: webservers
gather_facts: false # Skip if you don't need facts
tasks: ...
# Gather only specific facts
- hosts: webservers
gather_facts: true
gather_subset:
- network
- hardware
- "!all" # Only gather what's listed

Q: What are Ansible best practices?

  • Use roles for reusable, testable code
  • Store secrets in Vault, never plaintext
  • Use requirements.yml for collections and roles
  • Use tags for selective execution
  • Prefix MCs with numbers for ordering (00-base, 99-custom)
  • Use handlers instead of always restarting services
  • Set become: true at play level, not task level
  • Use block/rescue/always for error handling
  • Test roles with Molecule
  • Pin collection versions in requirements.yml
  • Use --check (dry run) and --diff before applying changes
# Dry run
ansible-playbook site.yml --check --diff
# Syntax check
ansible-playbook site.yml --syntax-check
# List hosts that would be affected
ansible-playbook site.yml --list-hosts
# Step through tasks one at a time
ansible-playbook site.yml --step

9. Molecule — Role Testing

# Install
pip install molecule molecule-docker
# Initialize molecule in a role
cd roles/my-role
molecule init scenario --driver-name docker
# molecule/default/molecule.yml
driver:
name: docker
platforms:
- name: rhel8-instance
image: registry.access.redhat.com/ubi8/ubi
pre_build_image: true
privileged: true
provisioner:
name: ansible
playbooks:
converge: converge.yml
verify: verify.yml
verifier:
name: ansible
# molecule/default/verify.yml
- name: Verify
hosts: all
tasks:
- name: Check nginx is running
service_facts:
- name: Assert nginx is active
assert:
that:
- "'nginx' in services"
- "services['nginx'].state == 'running'"
- name: Check port 80 is listening
wait_for:
port: 80
timeout: 5
# Run full test cycle
molecule test
# Individual stages
molecule create # Spin up containers
molecule converge # Run the role
molecule verify # Run tests
molecule destroy # Tear down
molecule login # SSH into instance for debugging

Quick-Fire Round

  • when vs failed_when? when controls if a task runs; failed_when defines what counts as failure.
  • How to run a task on localhost? delegate_to: localhost or hosts: localhost.
  • What is ansible_facts? Auto-collected system info — OS, IP, memory, CPU.
  • Difference between notify and direct task? notify triggers handlers at end of play only if task changed; direct task always runs.
  • What is check mode? --check — dry run, no changes made, shows what would change.
  • What is become? Privilege escalation — equivalent to sudo. become: true + become_user: root.
  • How to encrypt a single variable? ansible-vault encrypt_string.
  • What is gather_facts: false good for? Speeds up plays that don’t need system facts.
  • What does any_errors_fatal: true do? Stops the entire play on any host failure.
  • What is the free strategy? Hosts run tasks as fast as they can without waiting for others (vs linear default).

Ansible Automation Platform: A Comprehensive Guide

Ansible Deep Dive — Roles, Collections, Vault, AAP & OCP


1. Roles

What is a Role?

A Role is a structured, reusable unit of automation — it organizes tasks, variables, files, templates, and handlers into a standardized directory layout.

Role Directory Structure
roles/
└── my-webserver/
├── tasks/
│ └── main.yml # Entry point — all tasks
├── handlers/
│ └── main.yml # Handlers (restart, reload)
├── templates/
│ └── nginx.conf.j2 # Jinja2 templates
├── files/
│ └── index.html # Static files to copy
├── vars/
│ └── main.yml # High-priority variables
├── defaults/
│ └── main.yml # Low-priority defaults (overridable)
├── meta/
│ └── main.yml # Role metadata, dependencies
└── README.md
tasks/main.yml
---
- name: Install nginx
ansible.builtin.yum:
name: nginx
state: present
- name: Deploy config from template
ansible.builtin.template:
src: nginx.conf.j2
dest: /etc/nginx/nginx.conf
notify: Restart nginx
- name: Ensure nginx is started
ansible.builtin.service:
name: nginx
state: started
enabled: true
defaults/main.yml
---
nginx_port: 80
nginx_worker_processes: auto
nginx_max_connections: 1024
templates/nginx.conf.j2 (Jinja2)
worker_processes {{ nginx_worker_processes }};
events {
worker_connections {{ nginx_max_connections }};
}
server {
listen {{ nginx_port }};
server_name {{ ansible_hostname }};
}
meta/main.yml — Role Dependencies
---
dependencies:
- role: common
- role: firewall
vars:
open_ports:
- 80
- 443

Using a Role in a Playbook

- name: Configure web servers
hosts: webservers
become: true
roles:
- common
- my-webserver # Simple include
# OR with parameters:
roles:
- role: my-webserver
vars:
nginx_port: 8080
Role Execution Order in a Play
pre_tasks → role dependencies → roles → tasks → post_tasks → handlers

2. Collections

What is a Collection?

A Collection is a distribution format for Ansible content — it packages roles, modules, plugins, playbooks, and docs together under a namespace.

namespace.collection_name
└── ansible.builtin # Core Ansible modules
└── redhat.openshift # OCP-specific modules
└── community.kubernetes # K8s modules
└── amazon.aws # AWS modules
Collection Directory Structure
my_namespace/my_collection/
├── galaxy.yml # Collection metadata
├── README.md
├── roles/
│ └── my_role/
├── plugins/
│ ├── modules/ # Custom modules
│ ├── inventory/ # Dynamic inventory plugins
│ └── filter/ # Custom Jinja2 filters
├── playbooks/
└── docs/
Installing Collections
# From Ansible Galaxy
ansible-galaxy collection install redhat.openshift
# From requirements file (preferred)
ansible-galaxy collection install -r requirements.yml
# From Automation Hub (AAP)
ansible-galaxy collection install redhat.openshift \
--server https://cloud.redhat.com/api/automation-hub/
# Offline / from tarball
ansible-galaxy collection install my_namespace-my_collection-1.0.0.tar.gz
requirements.yml
---
collections:
- name: redhat.openshift
version: ">=2.0.0"
- name: community.kubernetes
version: "2.0.0"
- name: amazon.aws
version: ">=5.0.0"
roles:
- name: geerlingguy.nginx
version: "3.0.0"
Key Collections for OCP/K8s
CollectionUse
redhat.openshiftOCP-specific — routes, builds, operators
kubernetes.coreCore K8s module (k8s, helm, kubectl)
community.kubernetesCommunity K8s modules
redhat.rhel_system_rolesRHEL OS config
ansible.posixPOSIX system modules
amazon.awsAWS provisioning
Using Collection Modules
- name: Deploy OCP application
hosts: localhost
collections:
- redhat.openshift
- kubernetes.core
tasks:
- name: Create namespace
kubernetes.core.k8s:
state: present
definition:
apiVersion: v1
kind: Namespace
metadata:
name: my-app
- name: Deploy from template
kubernetes.core.k8s:
state: present
src: deployment.yaml
- name: Create OCP Route
redhat.openshift.openshift_route:
namespace: my-app
service: my-service
state: present

3. Ansible Vault

What is Vault?

Ansible Vault encrypts sensitive data — passwords, API keys, certificates — at rest in YAML files using AES-256 encryption.

Encrypting Files
# Encrypt an entire file
ansible-vault encrypt secrets.yml
# Decrypt a file
ansible-vault decrypt secrets.yml
# View without decrypting to disk
ansible-vault view secrets.yml
# Edit encrypted file
ansible-vault edit secrets.yml
# Re-key (change password)
ansible-vault rekey secrets.yml
# Encrypt a single string value
ansible-vault encrypt_string 'mysecretpassword' \
--name 'db_password'
Encrypted String Output
# Output of encrypt_string — paste directly into vars file
db_password: !vault |
$ANSIBLE_VAULT;1.1;AES256
61386634653335343832333938343039303664653764323035
64356635333562303937363135626565353537306261363539
...
Vault in Variable Files
# group_vars/all/vault.yml ← encrypted file
vault_db_password: supersecret123
vault_api_key: abc123xyz
# group_vars/all/vars.yml ← plaintext, references vault vars
db_password: "{{ vault_db_password }}"
api_key: "{{ vault_api_key }}"

Running Playbooks with Vault

# Prompt for password
ansible-playbook site.yml --ask-vault-pass
# Use password file
ansible-playbook site.yml --vault-password-file ~/.vault_pass
# Use environment variable
export ANSIBLE_VAULT_PASSWORD_FILE=~/.vault_pass
ansible-playbook site.yml
# Multiple vault IDs (different passwords for different files)
ansible-playbook site.yml \
--vault-id dev@~/.vault_pass_dev \
--vault-id prod@prompt
Multiple Vault IDs
# Encrypt with a specific vault ID label
ansible-vault encrypt_string 'prodpassword' \
--name 'db_password' \
--vault-id prod@~/.vault_pass_prod
# Result:
db_password: !vault |
$ANSIBLE_VAULT;1.2;AES256;prod ← vault ID label
...
Best Practices
  • Never commit plaintext secrets to git
  • Store vault password file outside the repo
  • Use separate vault IDs per environment (dev/staging/prod)
  • Encrypt the vars file, not the whole playbook
  • Use ansible-vault encrypt_string for individual values in otherwise plaintext files

4. Ansible Automation Platform (AAP)

What is AAP?

AAP (formerly Ansible Tower) is the enterprise version of Ansible — adds a Web UI, REST API, RBAC, job scheduling, audit logging, credential management, and execution environments.

AAP Architecture
┌──────────────────────────────────────────────────┐
│ Ansible Automation Platform │
│ │
│ ┌────────────┐ ┌──────────────────────────┐ │
│ │ Automation │ │ Automation Controller │ │
│ │ Hub │ │ (formerly Tower) │ │
│ │(Collections│ │ - Job Templates │ │
│ │ & roles) │ │ - Workflows │ │
│ └────────────┘ │ - Schedules │ │
│ │ - RBAC │ │
│ ┌────────────┐ │ - Credentials │ │
│ │ Event │ └──────────────────────────┘ │
│ │ Driven │ │
│ │ Ansible │ ┌──────────────────────────┐ │
│ │ (EDA) │ │ Execution Environments │ │
│ └────────────┘ │ (containerized runtime) │ │
│ └──────────────────────────┘ │
└──────────────────────────────────────────────────┘

Key AAP Concepts

ConceptDescription
InventoryStatic or dynamic host groups
CredentialStored secrets (SSH, cloud, vault) — never exposed
ProjectSCM-linked (Git) source for playbooks
Job TemplateReusable job definition — playbook + inventory + credential
WorkflowChain multiple job templates with conditions
Execution Environment (EE)Container image with Ansible + dependencies
OrganizationMulti-tenancy unit — groups users, teams, resources
RBACRole-based access — Admin, Execute, Read per resource
NotificationAlerts on job success/failure (Slack, email, webhook)
SurveyRuntime input form for job templates

Execution Environments (EE)
# execution-environment.yml
version: 1
build_arg_defaults:
EE_BASE_IMAGE: registry.redhat.io/ansible-automation-platform/ee-minimal-rhel8
dependencies:
galaxy: requirements.yml
python: requirements.txt
system: bindep.txt
additional_build_steps:
prepend:
- RUN pip3 install boto3
# Build EE with ansible-builder
pip install ansible-builder
ansible-builder build -t my-ee:1.0 -f execution-environment.yml
# Push to registry
podman push my-ee:1.0 registry.example.com/my-ee:1.0

Event-Driven Ansible (EDA)

EDA triggers automation based on events from external sources.

# rulebook.yml
- name: Respond to alerts
hosts: all
sources:
- ansible.eda.webhook: # Listen on webhook
host: 0.0.0.0
port: 5000
- ansible.eda.kafka: # Or Kafka topic
host: kafka.example.com
topic: alerts
rules:
- name: Restart service on alert
condition: event.payload.alertname == "ServiceDown"
action:
run_job_template:
name: Restart Service
organization: Default
- name: Scale up on high CPU
condition: event.payload.metric > 80
action:
run_playbook:
name: scale-up.yml

5. Ansible for OpenShift — Practical Examples

Install OCP Cluster (IPI)
- name: Install OpenShift cluster
hosts: localhost
collections:
- redhat.openshift_installer
tasks:
- name: Generate install-config
template:
src: install-config.yaml.j2
dest: "{{ install_dir }}/install-config.yaml"
- name: Run openshift-install
command: >
openshift-install create cluster
--dir {{ install_dir }}
--log-level info
Day-2 OCP Operations
- name: OCP Day-2 Configuration
hosts: localhost
collections:
- redhat.openshift
- kubernetes.core
tasks:
- name: Create htpasswd identity provider
kubernetes.core.k8s:
state: present
definition: "{{ lookup('template', 'oauth.yaml.j2') }}"
- name: Apply resource quotas
kubernetes.core.k8s:
state: present
definition:
apiVersion: v1
kind: ResourceQuota
metadata:
name: team-quota
namespace: "{{ item }}"
spec:
hard:
pods: "20"
requests.cpu: "4"
requests.memory: 8Gi
loop: "{{ namespaces }}"
- name: Deploy OADP operator
kubernetes.core.k8s:
state: present
src: oadp-subscription.yaml
- name: Wait for operator to be ready
kubernetes.core.k8s_info:
kind: ClusterServiceVersion
namespace: openshift-adp
register: csv
until: >
csv.resources | selectattr('status.phase','equalto','Succeeded') | list | length > 0
retries: 20
delay: 30

Interview Quick-Fire

  • Difference between vars and defaults in a role? defaults lowest priority — easily overridden. vars higher priority — harder to override.
  • What is ansible_facts? Auto-gathered system info (OS, IP, memory) available as variables.
  • How do you run only specific tasks? Use --tags or --skip-tags.
  • What is delegate_to? Run a task on a different host than the current one.
  • Difference between include_tasks and import_tasks? import is static (parsed at startup); include is dynamic (evaluated at runtime — supports loops/conditions).
  • What is an inventory plugin? Dynamically generates inventory from external sources (AWS, Azure, OCP).
  • How do you test Ansible roles? Using Molecule — spins up containers/VMs, runs the role, verifies with testinfra or ansible assertions.

Understanding Ansible: The IT Automation Tool

What is Ansible?

Ansible is an open-source IT automation tool developed by Red Hat that automates provisioning, configuration management, application deployment, and orchestration.


Key Characteristics

Agentless — no software installed on managed nodes; communicates over SSH (Linux) or WinRM (Windows).

Declarative & Procedural — you describe what you want (install nginx, ensure service is running) in YAML-based Playbooks.

Idempotent — running the same playbook multiple times produces the same result without unintended side effects.


Core Components
ComponentDescription
InventoryList of hosts/groups Ansible manages
PlaybookYAML file defining automation tasks
TaskA single unit of work (install package, copy file)
ModuleBuilt-in function that does the actual work (yum, copy, service)
RoleReusable, structured collection of tasks
HandlerTask triggered only when notified (e.g. restart nginx after config change)
VaultEncrypts sensitive data (passwords, keys)
Control NodeMachine where Ansible runs
Managed NodeTarget machine being automated

Simple Playbook Example
- name: Install and start nginx
hosts: webservers
become: true # sudo
tasks:
- name: Install nginx
ansible.builtin.yum:
name: nginx
state: present
- name: Start and enable nginx
ansible.builtin.service:
name: nginx
state: started
enabled: true
- name: Copy config file
ansible.builtin.copy:
src: nginx.conf
dest: /etc/nginx/nginx.conf
notify: Restart nginx # triggers handler only if file changed
handlers:
- name: Restart nginx
ansible.builtin.service:
name: nginx
state: restarted

Common Use Cases
  • Configuration management — enforce consistent state across 100s of servers
  • Application deployment — deploy code, run migrations, restart services
  • Provisioning — spin up cloud VMs, containers, network devices
  • Orchestration — coordinate multi-tier deployments in order
  • Patching — rolling OS updates across a fleet
  • OpenShift/K8s automation — manage OCP clusters, operators, resources via k8s module

Ansible vs Other Tools
AnsiblePuppetChefTerraform
LanguageYAMLDSLRubyHCL
AgentAgentlessAgentAgentAgentless
StyleProcedural + DeclarativeDeclarativeProceduralDeclarative
Best forConfig mgmt + orchestrationConfig mgmtConfig mgmtInfrastructure provisioning

In the OpenShift/Red Hat World

  • Ansible Automation Platform (AAP) — enterprise version with UI, RBAC, scheduling
  • ansible-playbook for OCP — automate cluster installs, day-2 ops, operator config
  • OpenShift + Ansible — often used together; Ansible handles infra, OCP handles workloads
  • Operator SDK — some operators are built using Ansible roles

Optimizing Node Performance with OpenShift’s Node Tuning Operator

Node Tuning Operator (NTO) in OpenShift


What is NTO?

The Node Tuning Operator manages node-level performance tuning on OpenShift by wrapping TuneD — the Linux tuning daemon. It allows you to apply performance profiles to nodes declaratively without manually editing OS-level configs.

MCO vs NTO:

  • MCO — manages OS config (files, systemd, kernel args) — what’s on the node
  • NTO — manages runtime performance tuning (CPU, memory, network, IRQ) — how the node performs

Architecture

┌────────────────────────────────────────────────────────────┐
│ Node Tuning Operator (NTO) │
│ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ Tune-D Operator Controller │ │
│ │ Watches Tuned CRs → renders TuneD profiles │ │
│ └──────────────────────────┬───────────────────────── ┘ │
└───────────────────────────── ┼─────────────────────────────┘
│ DaemonSet on every node
┌──────────────────────────────▼──────────────────────────────┐
│ TuneD Daemon (per node) │
│ │
│ Reads profiles → applies sysctl, CPU governor, │
│ IRQ affinity, huge pages, disk scheduler settings │
└─────────────────────────────────────────────────────────────┘
Components
ComponentRole
NTO ControllerWatches Tuned CRs, renders TuneD profiles on nodes
TuneD DaemonSetRuns on every node, applies active profile
Tuned CRCustom resource defining tuning rules and node matching
PerformanceProfileHigh-level CR for low-latency/HPC workloads (wraps TuneD + MCO)

Core CRDs

CRDPurpose
TunedLow-level TuneD profile definition
PerformanceProfileHigh-level latency/HPC tuning (generates Tuned + MachineConfig)
ProfilePer-node status — shows active TuneD profile on each node

TuneD Profiles — Built-in

# List active profiles on nodes
oc get profile -n openshift-cluster-node-tuning-operator
# Output:
# NAME TUNED APPLIED
# master-0 openshift-control-plane True
# worker-0 openshift-node True
# worker-gpu-1 openshift-node-performance True
Default Built-in Profiles
ProfileUse Case
openshiftBase OCP profile — applied to all nodes
openshift-nodeWorker node baseline
openshift-control-planeControl plane tuning
openshift-node-performanceLow-latency, generated by PerformanceProfile

Tuned CR — Custom Profile

apiVersion: tuned.openshift.io/v1
kind: Tuned
metadata:
name: my-custom-tuning
namespace: openshift-cluster-node-tuning-operator
spec:
profile:
- name: my-worker-profile
data: |
[main]
summary=Custom worker node tuning
[cpu]
governor=performance # CPU frequency scaling
energy_perf_bias=performance
min_perf_pct=100
[vm]
transparent_hugepages=never # Disable THP for databases
[disk]
# Disk I/O scheduler
elevator=none
[net]
nf_conntrack_hashsize=131072
[sysctl]
# Network tuning
net.core.rmem_max=134217728
net.core.wmem_max=134217728
net.ipv4.tcp_rmem=4096 87380 134217728
net.ipv4.tcp_wmem=4096 65536 134217728
net.core.netdev_max_backlog=250000
net.ipv4.tcp_timestamps=0
# Virtual memory
vm.dirty_ratio=10
vm.dirty_background_ratio=3
vm.swappiness=10
# Kernel
kernel.pid_max=1048576
fs.file-max=2097152
fs.inotify.max_user_watches=65536
recommend:
- profile: my-worker-profile
priority: 20 # Lower = higher priority
match:
- label: tuned.openshift.io/elasticsearch # Node label match

Node Matching in recommend

recommend:
# Match by node label
- profile: my-worker-profile
priority: 20
match:
- label: node-role.kubernetes.io/worker
# Match by pod label on the node
- profile: my-latency-profile
priority: 10
match:
- label: app
value: realtime-app
type: pod # Match nodes RUNNING pods with this label
# Match by node name (regex)
- profile: my-gpu-profile
priority: 15
match:
- label: kubernetes.io/hostname
value: gpu-node-.*
type: node
# Fallback — no match condition = applies to all
- profile: openshift-node
priority: 999

PerformanceProfile — High-Level Tuning

PerformanceProfile is the recommended approach for low-latency workloads (NFV, telco, HPC, real-time). It automatically generates:

  • A Tuned profile
  • A MachineConfig (kernel args, realtime kernel)
  • CPU pinning and isolation settings
apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
name: performance-worker
spec:
# --- CPU Configuration ---
cpu:
isolated: "4-31,36-63" # CPUs reserved for workloads (no OS interference)
reserved: "0-3,32-35" # CPUs for OS, kernel, system daemons
# --- NUMA topology ---
numa:
topologyPolicy: single-numa-node # Pods must fit in single NUMA node
# --- Huge Pages ---
hugepages:
defaultHugepagesSize: 1G
pages:
- count: 16
size: 1G
node: 0 # Allocate on NUMA node 0
- count: 1024
size: 2M
# --- Real-time kernel ---
realTimeKernel:
enabled: true # Installs RT kernel via MCO
# --- Additional kernel args ---
additionalKernelArgs:
- nmi_watchdog=0
- audit=0
- mce=off
- processor.max_cstate=1
- idle=poll
- rcupdate.rcu_normal_after_boot=0
# --- Node selection ---
nodeSelector:
node-role.kubernetes.io/worker-rt: ""
# --- NUMA-aware scheduling ---
workloadHints:
highPowerConsumption: true
realTime: true
perPodPowerManagement: false
# --- Machine config pool reference ---
machineConfigPoolSelector:
machineconfiguration.openshift.io/role: worker-rt

What PerformanceProfile Generates

# After applying PerformanceProfile, MCO generates:
oc get mc | grep performance
# 50-performance-worker ← kernel args, hugepages, RT kernel
oc get tuned -n openshift-cluster-node-tuning-operator
# openshift-node-performance-worker ← CPU isolation, IRQ affinity
oc get mcp
# worker-rt ← pool gets the new MachineConfig, nodes reboot

CPU Isolation Deep Dive

Reserved vs Isolated CPUs
┌──────────────────────────────────────────────────┐
│ Node CPUs (0-63) │
│ │
│ ┌─────────────────┐ ┌────────────────────────┐ │
│ │ Reserved CPUs │ │ Isolated CPUs │ │
│ │ (0-3, 32-35) │ │ (4-31, 36-63) │ │
│ │ │ │ │ │
│ │ - OS kernel │ │ - Application pods │ │
│ │ - System daemons│ │ - No OS interrupts │ │
│ │ - OCP infra │ │ - No kernel threads │ │
│ │ - kubelet │ │ - Guaranteed QoS only │ │
│ └─────────────────┘ └────────────────────────┘ │
└──────────────────────────────────────────────────┘
Pod Using Isolated CPUs
apiVersion: v1
kind: Pod
metadata:
name: realtime-pod
annotations:
cpu-load-balancing.crio.io: disable # Disable CPU load balancing
cpu-quota.crio.io: disable # Disable CPU quota
irq-load-balancing.crio.io: disable # Disable IRQ balancing
spec:
runtimeClassName: performance-worker # References PerformanceProfile
containers:
- name: rt-app
image: rt-app:latest
resources:
requests:
cpu: "8"
memory: 4Gi
hugepages-1Gi: 4Gi
limits:
cpu: "8" # requests == limits → Guaranteed QoS
memory: 4Gi
hugepages-1Gi: 4Gi

Isolated CPUs are only available to pods with Guaranteed QoS (requests == limits).


Tuning Categories & Examples

Network Tuning
[sysctl]
net.core.somaxconn=65535
net.core.netdev_max_backlog=250000
net.ipv4.tcp_max_syn_backlog=65535
net.ipv4.ip_local_port_range=1024 65535
net.ipv4.tcp_fin_timeout=15
net.ipv4.tcp_keepalive_time=300
net.ipv4.tcp_tw_reuse=1
Memory / VM Tuning
[sysctl]
vm.swappiness=1
vm.dirty_ratio=15
vm.dirty_background_ratio=5
vm.overcommit_memory=1 # For Redis/containers
kernel.shmmax=68719476736
kernel.shmall=4294967296
[vm]
transparent_hugepages=never # Always disable for databases (MySQL, PostgreSQL, MongoDB)

Storage / Disk Tuning
[disk]
readahead=>4096
[sysctl]
vm.dirty_writeback_centisecs=100
vm.dirty_expire_centisecs=500

vm.dirty_writeback_centisecs=100 vm.dirty_expire_centisecs=500

CPU Tuning
[cpu]
governor=performance
energy_perf_bias=performance
min_perf_pct=100
[sysctl]
kernel.sched_min_granularity_ns=10000000
kernel.sched_wakeup_granularity_ns=15000000

Troubleshooting NTO

Check Profile Application Status
# See active profile per node
oc get profile -n openshift-cluster-node-tuning-operator
# Detailed status of a node's profile
oc describe profile <node-name> \
-n openshift-cluster-node-tuning-operator
Check TuneD Daemon Logs
# Find TuneD pod on specific node
oc get pods -n openshift-cluster-node-tuning-operator \
-l openshift-app=tuned -o wide | grep <node-name>
# Get logs
oc logs -n openshift-cluster-node-tuning-operator \
tuned-<pod-id>
# Live follow
oc logs -n openshift-cluster-node-tuning-operator \
tuned-<pod-id> -f
Verify Sysctl Applied on Node
# Debug shell on node
oc debug node/<node-name>
chroot /host
# Check sysctl values
sysctl net.core.rmem_max
sysctl vm.swappiness
sysctl -a | grep net.ipv4.tcp
# Check TuneD active profile
tuned-adm active
# List available profiles
tuned-adm list
# Check TuneD service
systemctl status tuned
Check PerformanceProfile Status
oc describe performanceprofile performance-worker
# Check generated MachineConfig
oc get mc | grep performance
# Check generated Tuned profile
oc get tuned -n openshift-cluster-node-tuning-operator \
| grep performance
Common Errors & Fixes
ErrorCauseFix
Profile Applied: FalseTuneD syntax error in profileCheck TuneD pod logs for parse error
Profile not matching nodeWrong label selectorVerify node labels match recommend.match
Sysctl not appliedValue exceeds kernel limitCheck dmesg on node for errors
PerformanceProfile stuckMCP not updatingCheck MCP status, look for degraded nodes
hugepages not allocatedNot enough contiguous memoryAllocate hugepages earlier in boot via kernel args
Wrong profile priorityMultiple profiles competingCheck priority values — lower number wins

NTO vs MCO — When to Use Which

RequirementUse
Drop a config file on nodesMCO
Add systemd unitMCO
Change kernel boot argsMCO (or PerformanceProfile)
Tune sysctl at runtimeNTO (Tuned)
CPU isolation / pinningNTO (PerformanceProfile)
Huge pagesNTO (PerformanceProfile)
Real-time kernelNTO (PerformanceProfile) → MCO
CRI-O config changesMCO
IRQ affinityNTO (PerformanceProfile)

Interview Quick-Fire

  • What Linux daemon does NTO wrap? TuneD.
  • What namespace does NTO run in? openshift-cluster-node-tuning-operator.
  • How does NTO select which profile applies to a node? Via recommend rules in the Tuned CR — matched by node label, pod label, or node name. Lowest priority number wins.
  • What is the difference between Tuned CR and PerformanceProfile? Tuned is low-level TuneD config. PerformanceProfile is high-level — generates both a Tuned profile AND a MachineConfig for CPU isolation, hugepages, and RT kernel.
  • Does PerformanceProfile require a node reboot? Yes — it generates a MachineConfig which triggers MCO rolling update and reboot.
  • What QoS class is required for isolated CPU access? Guaranteed (requests == limits).
  • How do you verify which TuneD profile is active on a node? oc get profile -n openshift-cluster-node-tuning-operator or tuned-adm active on the node itself.
  • What does transparent_hugepages=never do and why set it? Disables Transparent Huge Pages — required for databases (MySQL, PostgreSQL, MongoDB, Redis) because THP causes latency spikes during memory compaction.

Understanding Machine Config Operator (MCO) in OpenShift

MCO (Machine Config Operator) in OpenShift


What is MCO?

The Machine Config Operator (MCO) is an OpenShift operator that manages the operating system and runtime configuration of cluster nodes. It treats the underlying RHCOS (Red Hat CoreOS) nodes as immutable infrastructure — all node-level changes go through MCO, never manual SSH edits.

Core idea: Just as Kubernetes manages application workloads declaratively, MCO manages node OS configuration declaratively.


MCO Architecture

┌────────────────────────────────────────────────────────────┐
│ MCO (Control Plane) │
│ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Machine Config Operator (MCO) │ │
│ │ Watches MachineConfig & MachineConfigPool │ │
│ └──────────────────────┬──────────────────────────────┘ │
│ │ │
│ ┌──────────────────────▼──────────────────────────────┐ │
│ │ Machine Config Controller (MCC) │ │
│ │ Renders final config, manages pool updates │ │
│ └──────────────────────┬──────────────────────────────┘ │
│ │ │
│ ┌──────────────────────▼──────────────────────────────┐ │
│ │ Machine Config Server (MCS) │ │
│ │ Serves Ignition configs to new nodes on boot │ │
│ └──────────────────────┬──────────────────────────────┘ │
└─────────────────────────┼──────────────────────────────────┘
│ DaemonSet on every node
┌─────────────────────────▼───────────────────────────────────┐
│ Machine Config Daemon (MCD) │
│ Applies config changes on each node, triggers reboots │
└─────────────────────────────────────────────────────────────┘
Component Responsibilities
ComponentRole
MCOTop-level operator — manages all sub-components
MCC (Controller)Renders MachineConfigs, manages rolling updates per pool
MCS (Server)HTTP server — serves Ignition config to bootstrapping nodes
MCD (Daemon)DaemonSet on every node — applies changes, validates, reboots

Core CRDs

1. MachineConfig (MC)

Defines the desired configuration for a node — files, systemd units, kernel arguments, etc.

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
name: 99-worker-custom-config
labels:
machineconfiguration.openshift.io/role: worker # targets worker pool
spec:
config:
ignition:
version: 3.2.0
# --- Drop files onto nodes ---
storage:
files:
- path: /etc/chrony.conf
mode: 0644
overwrite: true
contents:
source: data:,server%20time.example.com%20iburst%0A # URL-encoded
- path: /etc/sysctl.d/99-custom.conf
mode: 0644
contents:
source: data:,vm.max_map_count%3D262144%0A
# --- Systemd units ---
systemd:
units:
- name: kubelet.service
enabled: true
- name: my-custom-service.service
enabled: true
contents: |
[Unit]
Description=My Custom Service
After=network.target
[Service]
ExecStart=/usr/local/bin/my-script.sh
Restart=always
[Install]
WantedBy=multi-user.target
# --- Kernel arguments ---
kernelArguments:
- hugepages=1024
- intel_iommu=on
# --- Kernel type ---
kernelType: default # default | realtime (for low-latency/NFV)
# --- Extensions (RHCOS layering) ---
extensions:
- usbguard
- kernel-devel

2. MachineConfigPool (MCP)

Defines a group of nodes and which MachineConfigs apply to them. MCO applies configs to all nodes in a pool via rolling update.

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
name: worker
spec:
# Which nodes belong to this pool
machineConfigSelector:
matchExpressions:
- key: machineconfiguration.openshift.io/role
operator: In
values:
- worker
- custom-worker
# Which nodes (by label) are in this pool
nodeSelector:
matchLabels:
node-role.kubernetes.io/worker: ""
# Pause updates (maintenance window)
paused: false
# Max nodes updating simultaneously
maxUnavailable: 1 # Or percentage: "33%"
Built-in Pools
PoolTargetsNotes
masterControl plane nodesChanges here are critical — be careful
workerAll worker nodesDefault pool for most changes
Custom poolSpecific node subsetFor GPU nodes, infra nodes, etc.

3. Rendered MachineConfig

MCO merges all MachineConfigs targeting a pool into a single rendered-<pool>-<hash> config. This is what actually gets applied to nodes.

# View rendered configs
oc get mc | grep rendered
# Example output:
rendered-master-abc123def456 3.2.0 45m
rendered-worker-xyz789ghi012 3.2.0 45m

How MCO Applies Changes — Rolling Update Flow

New/Updated MachineConfig applied
MCC renders new rendered-<pool>-<hash>
MCP detects rendered config changed
MCD on node: drain → apply → reboot → validate
├─ One node at a time (maxUnavailable: 1)
├─ Waits for node Ready before next
└─ Pool goes Updating → Updated
# Watch rolling update progress
oc get mcp
# NAME CONFIG UPDATED UPDATING DEGRADED
# master rendered-master-abc123 True False False
# worker rendered-worker-xyz789 False True False ← updating
# Watch node-by-node
oc get nodes -w

Common MCO Use Cases

1. Add a Custom File to Nodes
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
name: 99-worker-sysctl
labels:
machineconfiguration.openshift.io/role: worker
spec:
config:
ignition:
version: 3.2.0
storage:
files:
- path: /etc/sysctl.d/99-worker-performance.conf
mode: 0644
overwrite: true
contents:
source: data:text/plain;charset=utf-8;base64,dm0ubWF4X21hcF9jb3VudD0yNjIxNDQ=
2. Add SSH Keys to Nodes
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
name: 99-worker-ssh-key
labels:
machineconfiguration.openshift.io/role: worker
spec:
config:
ignition:
version: 3.2.0
passwd:
users:
- name: core
sshAuthorizedKeys:
- ssh-rsa AAAAB3NzaC1yc2E... admin@example.com
3. Configure Container Runtime (CRI-O)
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
name: 99-worker-crio-config
labels:
machineconfiguration.openshift.io/role: worker
spec:
config:
ignition:
version: 3.2.0
storage:
files:
- path: /etc/crio/crio.conf.d/99-custom.conf
mode: 0644
contents:
source: data:text/plain,%5Bcrio.runtime%5D%0Alog_level%20%3D%20%22debug%22
4. Real-Time Kernel for Low-Latency Workloads
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
name: 99-worker-realtime
labels:
machineconfiguration.openshift.io/role: worker
spec:
kernelType: realtime
5. Custom MachineConfigPool (Infra Nodes)
# Step 1: Create the pool
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
name: infra
spec:
machineConfigSelector:
matchExpressions:
- key: machineconfiguration.openshift.io/role
operator: In
values: [worker, infra]
nodeSelector:
matchLabels:
node-role.kubernetes.io/infra: ""
---
# Step 2: Label the nodes
# oc label node infra-node-1 node-role.kubernetes.io/infra=
# Step 3: Create infra-specific MachineConfig
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
name: 99-infra-config
labels:
machineconfiguration.openshift.io/role: infra
spec:
config:
ignition:
version: 3.2.0

Pausing a MachineConfigPool

Use during maintenance windows to prevent automatic node reboots:

# Pause updates to worker pool
oc patch mcp worker --type merge -p '{"spec":{"paused":true}}'
# Apply your MachineConfigs (they queue up, nothing applies yet)
oc apply -f my-machineconfig.yaml
# Resume — all queued changes apply now
oc patch mcp worker --type merge -p '{"spec":{"paused":false}}'

Troubleshooting MCO

Check Pool & Node Status
# Overall pool health
oc get mcp
# Degraded details
oc describe mcp worker | grep -A10 Degraded
# Node MCO status
oc get node -o wide
oc describe node <node> | grep -i machine
Check MCD Logs on a Node
# Find MCD pod on specific node
oc get pod -n openshift-machine-config-operator \
-l k8s-app=machine-config-daemon \
-o wide | grep <node-name>
# Get logs
oc logs -n openshift-machine-config-operator \
machine-config-daemon-<id> -c machine-config-daemon
# Follow live
oc logs -n openshift-machine-config-operator \
machine-config-daemon-<id> -c machine-config-daemon -f
Common Errors & Fixes
ErrorCauseFix
Pool Degraded: TrueConfig failed to apply on a nodeCheck MCD logs on degraded node
Node stuck in NotReady after updateOS/config error post-rebootSSH to node, check journalctl -u machine-config-daemon
rendered config not foundMC label mismatchVerify machineconfiguration.openshift.io/role label
Pool updating indefinitelymaxUnavailable too low or node stuck drainingCheck pod disruption budgets, force-drain if safe
Wrong config appliedMultiple MCs with same file pathCheck MC priority via name prefix (00–99)
Pool paused, changes not applyingIntentional or forgotten pauseoc patch mcp worker --type merge -p '{"spec":{"paused":false}}'
MCD Node Annotation — Useful for Debugging
# Check what config a node currently has vs desired
oc get node <node> -o yaml | grep -A5 machineconfiguration
# Key annotations:
# machineconfiguration.openshift.io/currentConfig ← what's applied
# machineconfiguration.openshift.io/desiredConfig ← what should be applied
# machineconfiguration.openshift.io/state ← Done / Degraded / Working

MCO Interview Quick-Fire

  • Can you SSH into RHCOS nodes and manually edit files? Technically yes, but changes will be overwritten by MCO on next update. All changes must go through MachineConfig.
  • What is the naming convention for MachineConfigs? Prefix 0099 controls merge order; lower numbers apply first. e.g. 00-master, 99-worker-custom.
  • What happens if two MCs write to the same file? The one with the higher numeric prefix (applied last) wins.
  • What triggers a node reboot? Any change to files, systemd units, kernel args, or kernel type managed by MCO.
  • Can you apply MCO changes without a reboot? Only for changes that don’t require it (rare). Most OS-level changes require reboot.
  • What is RHCOS and why does MCO require it? Red Hat CoreOS — immutable, container-optimized OS. MCO is the only supported way to configure it.
  • Where are MCO components running? All in openshift-machine-config-operator namespace.