Optimizing Node Performance with OpenShift’s Node Tuning Operator

Node Tuning Operator (NTO) in OpenShift

What is NTO?

The Node Tuning Operator manages node-level performance tuning on OpenShift by wrapping TuneD — the Linux tuning daemon. It allows you to apply performance profiles to nodes declaratively without manually editing OS-level configs.

MCO vs NTO:

MCO — manages OS config (files, systemd, kernel args) — what’s on the node

NTO — manages runtime performance tuning (CPU, memory, network, IRQ) — how the node performs

Architecture

			
┌────────────────────────────────────────────────────────────┐
│                Node Tuning Operator (NTO)                  │
│                                                            │
│  ┌──────────────────────────────────────────────────────┐  │
│  │              Tune-D Operator Controller              │  │
│  │    Watches Tuned CRs → renders TuneD profiles        │  │
│  └──────────────────────────┬─────────────────────────  ┘  │
└───────────────────────────── ┼─────────────────────────────┘
                               │ DaemonSet on every node
┌──────────────────────────────▼──────────────────────────────┐
│                   TuneD Daemon (per node)                   │
│                                                             │
│  Reads profiles → applies sysctl, CPU governor,             │
│  IRQ affinity, huge pages, disk scheduler settings          │
└─────────────────────────────────────────────────────────────┘

		

Components

Component	Role
NTO Controller	Watches `Tuned` CRs, renders TuneD profiles on nodes
TuneD DaemonSet	Runs on every node, applies active profile
Tuned CR	Custom resource defining tuning rules and node matching
PerformanceProfile	High-level CR for low-latency/HPC workloads (wraps TuneD + MCO)

Core CRDs

CRD	Purpose
`Tuned`	Low-level TuneD profile definition
`PerformanceProfile`	High-level latency/HPC tuning (generates Tuned + MachineConfig)
`Profile`	Per-node status — shows active TuneD profile on each node

TuneD Profiles — Built-in

			
# List active profiles on nodes
oc get profile -n openshift-cluster-node-tuning-operator
# Output:
# NAME                    TUNED                     APPLIED
# master-0                openshift-control-plane   True
# worker-0                openshift-node            True
# worker-gpu-1            openshift-node-performance True

		

Default Built-in Profiles

Profile	Use Case
`openshift`	Base OCP profile — applied to all nodes
`openshift-node`	Worker node baseline
`openshift-control-plane`	Control plane tuning
`openshift-node-performance`	Low-latency, generated by PerformanceProfile

Tuned CR — Custom Profile

			
apiVersion: tuned.openshift.io/v1
kind: Tuned
metadata:
  name: my-custom-tuning
  namespace: openshift-cluster-node-tuning-operator
spec:
  profile:
    - name: my-worker-profile
      data: |
        [main]
        summary=Custom worker node tuning
        [cpu]
        governor=performance              # CPU frequency scaling
        energy_perf_bias=performance
        min_perf_pct=100
        [vm]
        transparent_hugepages=never       # Disable THP for databases
        [disk]
        # Disk I/O scheduler
        elevator=none
        [net]
        nf_conntrack_hashsize=131072
        [sysctl]
        # Network tuning
        net.core.rmem_max=134217728
        net.core.wmem_max=134217728
        net.ipv4.tcp_rmem=4096 87380 134217728
        net.ipv4.tcp_wmem=4096 65536 134217728
        net.core.netdev_max_backlog=250000
        net.ipv4.tcp_timestamps=0
        # Virtual memory
        vm.dirty_ratio=10
        vm.dirty_background_ratio=3
        vm.swappiness=10
        # Kernel
        kernel.pid_max=1048576
        fs.file-max=2097152
        fs.inotify.max_user_watches=65536
  recommend:
    - profile: my-worker-profile
      priority: 20                        # Lower = higher priority
      match:
        - label: tuned.openshift.io/elasticsearch  # Node label match

		

Node Matching in `recommend`

			
recommend:
  # Match by node label
  - profile: my-worker-profile
    priority: 20
    match:
      - label: node-role.kubernetes.io/worker
  # Match by pod label on the node
  - profile: my-latency-profile
    priority: 10
    match:
      - label: app
        value: realtime-app
        type: pod                         # Match nodes RUNNING pods with this label
  # Match by node name (regex)
  - profile: my-gpu-profile
    priority: 15
    match:
      - label: kubernetes.io/hostname
        value: gpu-node-.*
        type: node
  # Fallback — no match condition = applies to all
  - profile: openshift-node
    priority: 999

		

PerformanceProfile — High-Level Tuning

PerformanceProfile is the recommended approach for low-latency workloads (NFV, telco, HPC, real-time). It automatically generates:

A Tuned profile
A MachineConfig (kernel args, realtime kernel)
CPU pinning and isolation settings

			
apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
  name: performance-worker
spec:
  # --- CPU Configuration ---
  cpu:
    isolated: "4-31,36-63"       # CPUs reserved for workloads (no OS interference)
    reserved: "0-3,32-35"        # CPUs for OS, kernel, system daemons
  # --- NUMA topology ---
  numa:
    topologyPolicy: single-numa-node   # Pods must fit in single NUMA node
  # --- Huge Pages ---
  hugepages:
    defaultHugepagesSize: 1G
    pages:
      - count: 16
        size: 1G
        node: 0                   # Allocate on NUMA node 0
      - count: 1024
        size: 2M
  # --- Real-time kernel ---
  realTimeKernel:
    enabled: true                 # Installs RT kernel via MCO
  # --- Additional kernel args ---
  additionalKernelArgs:
    - nmi_watchdog=0
    - audit=0
    - mce=off
    - processor.max_cstate=1
    - idle=poll
    - rcupdate.rcu_normal_after_boot=0
  # --- Node selection ---
  nodeSelector:
    node-role.kubernetes.io/worker-rt: ""
  # --- NUMA-aware scheduling ---
  workloadHints:
    highPowerConsumption: true
    realTime: true
    perPodPowerManagement: false
  # --- Machine config pool reference ---
  machineConfigPoolSelector:
    machineconfiguration.openshift.io/role: worker-rt

		

What PerformanceProfile Generates

			
# After applying PerformanceProfile, MCO generates:
oc get mc | grep performance
# 50-performance-worker   ← kernel args, hugepages, RT kernel
oc get tuned -n openshift-cluster-node-tuning-operator
# openshift-node-performance-worker  ← CPU isolation, IRQ affinity
oc get mcp
# worker-rt   ← pool gets the new MachineConfig, nodes reboot

		

CPU Isolation Deep Dive

Reserved vs Isolated CPUs

			
┌──────────────────────────────────────────────────┐
│                  Node CPUs (0-63)                │
│                                                  │
│  ┌─────────────────┐  ┌────────────────────────┐ │
│  │  Reserved CPUs  │  │    Isolated CPUs       │ │
│  │   (0-3, 32-35)  │  │   (4-31, 36-63)        │ │
│  │                 │  │                        │ │
│  │ - OS kernel     │  │ - Application pods     │ │
│  │ - System daemons│  │ - No OS interrupts     │ │
│  │ - OCP infra     │  │ - No kernel threads    │ │
│  │ - kubelet       │  │ - Guaranteed QoS only  │ │
│  └─────────────────┘  └────────────────────────┘ │
└──────────────────────────────────────────────────┘

		

Pod Using Isolated CPUs

			
apiVersion: v1
kind: Pod
metadata:
  name: realtime-pod
  annotations:
    cpu-load-balancing.crio.io: disable     # Disable CPU load balancing
    cpu-quota.crio.io: disable              # Disable CPU quota
    irq-load-balancing.crio.io: disable     # Disable IRQ balancing
spec:
  runtimeClassName: performance-worker      # References PerformanceProfile
  containers:
    - name: rt-app
      image: rt-app:latest
      resources:
        requests:
          cpu: "8"
          memory: 4Gi
          hugepages-1Gi: 4Gi
        limits:
          cpu: "8"                          # requests == limits → Guaranteed QoS
          memory: 4Gi
          hugepages-1Gi: 4Gi

		

Isolated CPUs are only available to pods with Guaranteed QoS (requests == limits).

Tuning Categories & Examples

Network Tuning

			
[sysctl]
net.core.somaxconn=65535
net.core.netdev_max_backlog=250000
net.ipv4.tcp_max_syn_backlog=65535
net.ipv4.ip_local_port_range=1024 65535
net.ipv4.tcp_fin_timeout=15
net.ipv4.tcp_keepalive_time=300
net.ipv4.tcp_tw_reuse=1

		

Memory / VM Tuning

			
[sysctl]
vm.swappiness=1
vm.dirty_ratio=15
vm.dirty_background_ratio=5
vm.overcommit_memory=1          # For Redis/containers
kernel.shmmax=68719476736
kernel.shmall=4294967296
[vm]
transparent_hugepages=never     # Always disable for databases (MySQL, PostgreSQL, MongoDB)

		

Storage / Disk Tuning

			
[disk]
readahead=>4096
[sysctl]
vm.dirty_writeback_centisecs=100
vm.dirty_expire_centisecs=500

		

vm.dirty_writeback_centisecs=100 vm.dirty_expire_centisecs=500

CPU Tuning

			
[cpu]
governor=performance
energy_perf_bias=performance
min_perf_pct=100
[sysctl]
kernel.sched_min_granularity_ns=10000000
kernel.sched_wakeup_granularity_ns=15000000

		

Troubleshooting NTO

Check Profile Application Status

			
# See active profile per node
oc get profile -n openshift-cluster-node-tuning-operator
# Detailed status of a node's profile
oc describe profile <node-name> \
  -n openshift-cluster-node-tuning-operator

		

Check TuneD Daemon Logs

			
# Find TuneD pod on specific node
oc get pods -n openshift-cluster-node-tuning-operator \
  -l openshift-app=tuned -o wide | grep <node-name>
# Get logs
oc logs -n openshift-cluster-node-tuning-operator \
  tuned-<pod-id>
# Live follow
oc logs -n openshift-cluster-node-tuning-operator \
  tuned-<pod-id> -f

		

Verify Sysctl Applied on Node

			
# Debug shell on node
oc debug node/<node-name>
chroot /host
# Check sysctl values
sysctl net.core.rmem_max
sysctl vm.swappiness
sysctl -a | grep net.ipv4.tcp
# Check TuneD active profile
tuned-adm active
# List available profiles
tuned-adm list
# Check TuneD service
systemctl status tuned

		

Check PerformanceProfile Status

			
oc describe performanceprofile performance-worker
# Check generated MachineConfig
oc get mc | grep performance
# Check generated Tuned profile
oc get tuned -n openshift-cluster-node-tuning-operator \
  | grep performance

		

Common Errors & Fixes

Error	Cause	Fix
Profile `Applied: False`	TuneD syntax error in profile	Check TuneD pod logs for parse error
Profile not matching node	Wrong label selector	Verify node labels match `recommend.match`
Sysctl not applied	Value exceeds kernel limit	Check `dmesg` on node for errors
PerformanceProfile stuck	MCP not updating	Check MCP status, look for degraded nodes
`hugepages` not allocated	Not enough contiguous memory	Allocate hugepages earlier in boot via kernel args
Wrong profile priority	Multiple profiles competing	Check priority values — lower number wins

NTO vs MCO — When to Use Which

Requirement	Use
Drop a config file on nodes	MCO
Add systemd unit	MCO
Change kernel boot args	MCO (or PerformanceProfile)
Tune sysctl at runtime	NTO (Tuned)
CPU isolation / pinning	NTO (PerformanceProfile)
Huge pages	NTO (PerformanceProfile)
Real-time kernel	NTO (PerformanceProfile) → MCO
CRI-O config changes	MCO
IRQ affinity	NTO (PerformanceProfile)

Interview Quick-Fire

What Linux daemon does NTO wrap? TuneD.
What namespace does NTO run in? openshift-cluster-node-tuning-operator.
How does NTO select which profile applies to a node? Via recommend rules in the Tuned CR — matched by node label, pod label, or node name. Lowest priority number wins.
What is the difference between Tuned CR and PerformanceProfile? Tuned is low-level TuneD config. PerformanceProfile is high-level — generates both a Tuned profile AND a MachineConfig for CPU isolation, hugepages, and RT kernel.
Does PerformanceProfile require a node reboot? Yes — it generates a MachineConfig which triggers MCO rolling update and reboot.
What QoS class is required for isolated CPU access? Guaranteed (requests == limits).
How do you verify which TuneD profile is active on a node? oc get profile -n openshift-cluster-node-tuning-operator or tuned-adm active on the node itself.
What does transparent_hugepages=never do and why set it? Disables Transparent Huge Pages — required for databases (MySQL, PostgreSQL, MongoDB, Redis) because THP causes latency spikes during memory compaction.

Infra Cloud Solutions

Optimizing Node Performance with OpenShift’s Node Tuning Operator

Node Tuning Operator (NTO) in OpenShift

What is NTO?

Architecture

Components

Core CRDs

TuneD Profiles — Built-in

Default Built-in Profiles

Tuned CR — Custom Profile

Node Matching in `recommend`

PerformanceProfile — High-Level Tuning

What PerformanceProfile Generates

CPU Isolation Deep Dive

Reserved vs Isolated CPUs

Pod Using Isolated CPUs

Tuning Categories & Examples

Network Tuning

Memory / VM Tuning

Storage / Disk Tuning

CPU Tuning

Troubleshooting NTO

Check Profile Application Status

Check TuneD Daemon Logs

Verify Sysctl Applied on Node

Check PerformanceProfile Status

Common Errors & Fixes

NTO vs MCO — When to Use Which

Interview Quick-Fire

Like this:

Related

Leave a ReplyCancel reply

Node Tuning Operator (NTO) in OpenShift

What is NTO?

Architecture

Components

Core CRDs

TuneD Profiles — Built-in

Default Built-in Profiles

Tuned CR — Custom Profile

Node Matching in recommend

PerformanceProfile — High-Level Tuning

What PerformanceProfile Generates

CPU Isolation Deep Dive

Reserved vs Isolated CPUs

Pod Using Isolated CPUs

Tuning Categories & Examples

Network Tuning

Memory / VM Tuning

Storage / Disk Tuning

CPU Tuning

Troubleshooting NTO

Check Profile Application Status

Check TuneD Daemon Logs

Verify Sysctl Applied on Node

Check PerformanceProfile Status

Common Errors & Fixes

NTO vs MCO — When to Use Which

Interview Quick-Fire

Share this:

Like this:

Related

Leave a ReplyCancel reply

Discover more from Infra Cloud Solutions

Node Matching in `recommend`