linux

Top Linux Interview Questions for All Levels

techhadoop linux

Preparing for a Linux interview can vary wildly depending on whether you are looking at an entry-level helpdesk role, a DevOps position, or a senior Systems Administrator gig.

To give you the best foundation, here is a breakdown of the most frequently asked Linux interview questions, categorized by difficulty and topic.

1. Fundamentals & Concept Questions

Q: What is the difference between a Hard Link and a Soft (Symbolic) Link?
  • Hard Link: A hard link is a direct pointer to the physical data (the inode) on the disk. If you delete the original file, the hard link still works and can access the data. They cannot cross different filesystems or point to directories.
  • Soft Link: Essentially a shortcut. It points to the file name rather than the underlying inode. If you delete the original file, the soft link becomes “broken” (dangling). They can cross filesystems and point to directories.
Q: Can you explain the Linux Boot Sequence?

Interviewers love this because it proves you know how the OS operates under the hood. The core steps are:

  1. BIOS/UEFI: Performs POST (Power-On Self-Test) and loads the MBR/EFI boot sector.
  2. GRUB (Bootloader): Allows you to select the kernel and loads it into memory.
  3. Kernel: Initializes hardware and mounts the root filesystem (/).
  4. systemd / Init: The primary process (PID 1) starts up, managing system services, targets, and daemons.
Q: What is an Inode, and what happens if a disk runs out of them?

An inode (index node) is a data structure that stores metadata about a file (size, owner, permissions, location on disk), but not the file name or actual data.

Interview Tip: If a system runs out of inodes (which you check via df -i), you cannot create new files, even if df -h shows you have gigabytes of free disk space.

2. Practical Troubleshooting Scenario Questions

Q: “The server is running incredibly slow. What are the first three commands you run?”

This tests your live triage methodology. A great baseline answer is:

  • top (or htop): To instantly see CPU/Memory usage and identify rogue, resource-heavy processes.
  • free -m: To check if the system is running out of physical RAM and aggressively using Swap space, which destroys performance.
  • uptime: To look at the system load averages over the last 1, 5, and 15 minutes.
Q: “An application can’t bind to its port, or users can’t connect. How do you troubleshoot?”
  1. Check if the service is actually running and listening: ss -tulpn or netstat -tulnp.
  2. Check for local firewall restrictions: iptables -L or ufw status / firewall-cmd --list-all.
  3. Check security enforcement modules like SELinux or AppArmor: run getenforce to see if it’s blocking the application.
Q: “How do you securely check logs for a specific service failing in systemd?”

Instead of digging around blindly in /var/log/, use:

Bash

journalctl -u nginx.service -n 50 --no-pager

(Replacing nginx.service with the failing service. -n 50 pulls the last 50 lines).

3. High-Frequency Commands Cheat Sheet

Be ready to explain or write out these daily-use commands:

Scenario / ObjectiveCommand to Use
Search text recursively inside a directorygrep -ri "error_pattern" /var/log/
Follow a log file in real-timetail -f /var/log/syslog
Find large files eating up spacedu -sh * | sort -rh
Change file permissions safelychmod 755 filename or chmod u+rwx,go+rx
Check which process is using a filelsof /path/to/file
See detailed block storage layoutlsblk or fdisk -l

4. Advanced / DevOps Linux Questions

Q: What is the difference between a Process and a Thread in Linux?

In Linux, the kernel doesn’t strictly differentiate between them the way other operating systems do. Both are represented by task_struct. However:

  • A Process gets its own isolated memory space.
  • A Thread is a lightweight unit of execution created inside a process that shares the parent process’s memory, file descriptors, and environment.
Q: What is Zombie Process, and how do you kill it?

A zombie process is a process that has completed execution but still has an entry in the process table because its parent process hasn’t read its exit status yet (via wait()).

The Catch: You cannot kill a zombie process using kill -9 PID because it is already technically dead. To get rid of it, you must either kill its parent process or restart the service owning it.

Top Linux Interview Questions for Aspiring Admins

techhadoop linux

Here are key Linux interview questions organized by topic:

Core Concepts

What is the difference between a process and a thread? A process is an independent program with its own memory space; a thread is a lightweight unit within a process sharing the same memory.

What are inodes? Inodes store metadata about files (permissions, ownership, timestamps, disk block locations) but not the filename or file data itself.

Explain the boot process. BIOS/UEFI → Bootloader (GRUB) → Kernel loads → init/systemd starts → runlevel/target reached.

File System

  • Hard link vs soft link — Hard links point to the same inode; symlinks point to a path. Deleting the original breaks a symlink but not a hard link.
  • /proc filesystem — A virtual FS exposing kernel and process info in real time (e.g., /proc/cpuinfo, /proc/meminfo).
  • /etc/fstab — Defines filesystems to mount at boot.

Permissions & Users

  • chmod 755 — Owner: rwx, Group: r-x, Others: r-x
  • setuid bit — Runs executable as the file owner (e.g., passwd runs as root).
  • sudo vs su — sudo runs a single command as root; su switches the entire session to another user.

Processes & Performance

  • ps aux vs top — ps aux is a snapshot; top is real-time.
  • Zombie process — A process that has finished but its parent hasn’t called wait() to read its exit status.
  • nice / renice — Set/change process scheduling priority (-20 highest, 19 lowest).
  • Check memory usage — free -h, vmstat, /proc/meminfo

Networking

  • netstat -tulnp / ss -tulnp — List open ports and listening services.
  • iptables vs firewalld — Both manage the kernel netfilter; firewalld is a dynamic wrapper with zones.
  • Check routing table — ip route or route -n
  • /etc/resolv.conf — Configures DNS nameservers.

Common Commands

TaskCommand
Find files by namefind / -name "file.txt"
Search inside filesgrep -r "pattern" /dir
Disk usagedf -h / du -sh *
Archive & compresstar -czf file.tar.gz dir/
Live log watchingtail -f /var/log/syslog
Schedule taskscrontab -e

systemd

  • systemctl status/start/stop/enable — Manage services.
  • Units — Services (.service), timers (.timer), mounts (.mount), etc.
  • journalctl -u nginx -f — Follow logs for a specific unit.

Shell Scripting

  • $? — Exit code of the last command (0 = success).
  • $@ vs $* — Both hold script arguments; $@ treats each as a separate word, $* treats all as one string.
  • 2>/dev/null — Redirect stderr to discard errors.

Tricky / Advanced Questions

  • What happens when you type a command in the shell? Shell forks a child process → exec() replaces it with the command → parent waits.
  • OOM killer — Kernel kills processes when RAM is exhausted; logs in dmesg.
  • strace — Traces system calls made by a process, useful for debugging.
  • lsof — Lists open files and which processes have them open.

Understanding Log Retention for Prometheus and Docker

techhadoop docker, linux, monitoring

In may setup, “retention” actually applies to two different things: Prometheus metrics (the graphs) and Ubuntu system logs (the text files).

By default, they have very different lifespans.

1. Prometheus Metrics (The Graphs)

If you didn’t specify a retention time when you ran your Prometheus Docker container, it uses the default.

  • Default Duration: 15 days.
  • What happens after? Prometheus uses a “First-In-First-Out” system. Once data hits day 16, the data from day 1 is deleted to make room.
  • How to change it: If you want 30 days of history to show your Director month-over-month trends, you need to add this flag to your docker run command for Prometheus:--storage.tsdb.retention.time=30d

2. Ubuntu System Logs (/var/log)

This is handled by a service called logrotate. It manages things like your mail.log, syslog, and auth.log.

  • Default Duration: Usually 4 weeks (28 days).
  • How it works: It keeps 4 “rotated” files. Every Sunday, it compresses the current log and deletes the oldest one.
  • How to check your specific settings:Bashcat /etc/logrotate.d/rsyslog Look for the number next to rotate. If it says 4, and the interval is weekly, you have 28 days.

3. Docker Container Logs

This is the danger zone. By default, Docker container logs (like the ones for cadvisor or node-exporter) have no limit.

If a container starts throwing thousands of errors, the log file will grow until it fills your entire hard drive. Since we are doing a Pilot Group, you should verify your Docker logging driver.

The “Safe” way to run your containers:

Add these flags to your docker run commands to ensure you only keep 3 files of 10MB each:

Bash

--log-opt max-size=10m --log-opt max-file=3

Summary Table

Data TypeDefault RetentionControlled By
Prometheus Data15 Days--storage.tsdb.retention.time
System Logs~28 Days/etc/logrotate.conf
Docker LogsUnlimited (Until disk is full)Docker Log Driver

Recommendation for your 20 Servers

For an Executive Director’s report, 15 days is usually too short. Most admins prefer 90 days for Prometheus so they can compare “This Quarter vs. Last Quarter.”

To check how much disk space your current Prometheus data is taking:

Bash

du -sh /var/lib/docker/volumes/<prometheus_volume_name>/_data

To update your Prometheus retention to 90 days, you need to restart the container with a specific flag. Since you are likely running this via a docker run command, we will stop the old one and start the new one with the updated storage policy.

1. Update Prometheus to 90-Day Retention

Run these commands on your Central Monitoring Server:

Bash

# 1. Stop and remove the existing Prometheus container
docker stop prometheus
docker rm prometheus
# 2. Start it again with the 90-day retention flag
docker run -d \
  --name=prometheus \
  --restart=always \
  --publish=9090:9090 \
  -v /etc/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml \
  -v prometheus-data:/prometheus \
  prom/prometheus:latest \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/prometheus \
  --storage.tsdb.retention.time=90d

2. Why 90 Days? (The Strategy)

Setting it to 90 days is the “sweet spot” for several reasons:

  • Quarterly Reporting: You can show your Executive Director performance trends across a full 3-month business quarter.
  • Capacity Planning: 90 days of data allows the Predictive AI (the predict_linear we discussed) to be much more accurate because it has a longer history to analyze.
  • Storage Management: Prometheus is very efficient. For 20 servers, 90 days of metrics will likely only consume about 5GB to 10GB of disk space.

3. Verify the Change

Once the container is back up, you can verify that the new setting is active:

  1. Open your browser to http://<CENTRAL_IP>:9090/status.
  2. Look for Command-Line Flags.
  3. Confirm that --storage.tsdb.retention.time=90d is listed there.

4. Pro-Tip: Disk Space Alert

Since you are now keeping 6 times more data than the default, you should add a simple alert in Grafana to monitor the Central Server’s own disk space.

The “Golden Rule” of Monitoring: The monitoring server must be the most stable server in the fleet. If its disk fills up because of long retention, you lose visibility into all other 20 servers.

Next Steps for your Project

Now that your data is safe for the long term:

  1. Check the logs: docker logs prometheus to ensure there are no “permission denied” errors on the data folder.
  2. Snapshot Check: Since you are modifying the central server, it wouldn’t hurt to take a quick vSphere snapshot of this one too, just like you did for the pilot nodes.

Enhance Monitoring with AI: Predictive and Generative Alerting

techhadoop ai, linux, monitoring

To achieve both Predictive Maintenance (knowing when things will fail) and Generative Alerting (getting an AI explanation of the failure), you need to build an “AI Feedback Loop” around your existing Prometheus and Grafana stack.

Here is how you can implement both for your fleet:

1. Predictive: The “Forecasting” Layer

This uses mathematical AI (Linear Regression) to look at your current trends and project them into the future. It’s perfect for preventing “Disk Full” or “Memory Exhaustion” crashes.

How to set it up in Grafana:

  1. Create a new Alert Rule.
  2. Use this formula to predict if a disk will be full in 24 hours based on the last 6 hours of data:$$predict\_linear(node\_filesystem\_free\_bytes{job=”nodes”}[6h], 86400) < 0$$
  3. The Result: Instead of waiting for the disk to hit 95%, the AI alerts you when the trend indicates you are 24 hours away from disaster.

2. Generative: The “Explainable” Layer

This is the most “impressive” part for your Executive Director. It converts technical errors into plain English. Since you have a Mail Server (Postfix) and Grafana, you can use a “Webhook” to send alerts through an AI.

The Workflow:

  1. Trigger: A Pilot Group server crashes.
  2. Webhook: Grafana sends the alert JSON to a simple Python script or an automation tool like n8n or Make.com.
  3. AI Processing: The script sends the error to an LLM (OpenAI or a local Llama model) with this prompt:“I am a Linux admin. I received this alert: [Alert Data]. Explain what happened and give me 3 commands to fix it on Ubuntu.”
  4. Delivery: The AI sends a clean, formatted email through your Postfix server.

3. Integrated Tooling: Netdata

If you want both of these features without writing custom code, I highly recommend installing Netdata on your pilot group.

  • Machine Learning (ML): Netdata has an “Anomaly Advisor” built-in. It trains a model on every single metric (CPU, Disk, Net) every hour.
  • AIOps: It highlights “unusual” behavior in violet on the graphs. If your mail server suddenly starts sending 1,000% more mail than usual, the AI marks it as an anomaly before you even set an alert.

4. Implementation Plan for your 20 Servers

PhaseActionAI Benefit
Step 1Add predict_linear queries to Grafana.Predictive: No more emergency disk-clearing at 2 AM.
Step 2Use Grafana Play-app or n8n to link Alerts to an LLM.Generative: Your team gets “Smart Alerts” with solutions included.
Step 3Install a local AI (like Ollama) on your central server.Privacy: Keep your server data local while still getting AI insights.

How to Present This to Your Director

When you talk to the Executive Director, frame it like this:

“We are moving from Traditional Monitoring to AIOps.

  1. Predictive AI will save us money by preventing downtime before it happens.
  2. Generative AI will act as a ‘Force Multiplier’ for the team, providing instant troubleshooting steps for any system error, reducing our recovery time by 80%.”

Setup Node Exporter for Centralized Monitoring

techhadoop docker, linux

1. Run this on Linux 20 server(s)

I will provide ansible playbook in next post ( when you have a multiple severs, automation is the key)

docker run -d \
  --name=node-exporter \
  --restart=always \
  --net="host" \
  --pid="host" \
  -v "/:/host:ro,rslave" \
  quay.io/prometheus/node-exporter:latest \
  --path.rootfs=/host

I would use :ro,rslave instead of only :ro, because the official Docker example for node_exporter uses bind mounting so the container can correctly see host mount points. Node Exporter is meant to monitor the host system, not just the container. (GitHub)

Check one server:

curl http://localhost:9100/metrics

From central Prometheus server:

curl http://SERVER_IP:9100/metrics

2. Open firewall only from Prometheus server

On each Linux host, allow port 9100 only from your central Prometheus server:

sudo ufw allow from PROMETHEUS_SERVER_IP to any port 9100 proto tcp

Do not expose 9100 publicly.

3. Central Prometheus config

On your central monitoring server, Prometheus scrapes all 20 Node Exporters.

prometheus.yml:

global:
  scrape_interval: 15s
scrape_configs:
  - job_name: "linux_servers"
    static_configs:
      - targets:
          - "10.0.1.11:9100"
          - "10.0.1.12:9100"
          - "10.0.1.13:9100"
          - "10.0.1.14:9100"
          - "10.0.1.15:9100"
          # add all 20 servers here

Prometheus uses scrape_configs and targets to pull metrics from exporters. (Prometheus)

Restart Prometheus:

docker restart prometheus

4. Add Prometheus to Grafana

In Grafana:

Connections → Data sources → Prometheus
URL: http://PROMETHEUS_SERVER_IP:9090
Save & Test

Then import dashboard:

Dashboard ID: 1860

That is the popular Node Exporter Full dashboard. Example of dashboard

Final architecture

20 Linux Servers
   ↓ node-exporter :9100
Central Prometheus
   ↓
Grafana Dashboard

Important: Node Exporter does not send data to Grafana directly.
It exposes metrics, Prometheus pulls them, and Grafana visualizes Prometheus data.

cAdvisor: Your Guide to Container Monitoring

techhadoop docker, linux, Uncategorized

cAdvisor Explained

What is cAdvisor?

cAdvisor (Container Advisor) is an open-source tool by Google that collects, aggregates, and exports resource usage and performance metrics from running containers. It gives you deep visibility into what every container on your host is doing.

┌─────────────────────────────────────────────────────────────┐
│                      LINUX HOST                             │
│                                                             │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐   │
│  │Container │  │Container │  │Container │  │Container │   │
│  │  nginx   │  │  api     │  │ postgres │  │  redis   │   │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘   │
│       │              │              │              │         │
│       └──────────────┴──────────────┴──────────────┘        │
│                              │                              │
│                    ┌─────────▼─────────┐                    │
│                    │    cAdvisor        │                    │
│                    │                   │                    │
│                    │ reads cgroups     │                    │
│                    │ reads /proc       │                    │
│                    │ reads /sys        │                    │
│                    │ reads Docker API  │                    │
│                    └─────────┬─────────┘                    │
│                              │ exposes                      │
│                    ┌─────────▼─────────┐                    │
│                    │  :8080/metrics    │                    │
│                    │  (Prometheus fmt) │                    │
│                    └───────────────────┘                    │
└─────────────────────────────────────────────────────────────┘

How cAdvisor Works

Container Runtime (Docker / containerd)
          │
          │ Docker API / containerd API
          ▼
┌─────────────────────────────────────┐
│           cAdvisor                  │
│                                     │
│  ┌─────────────────────────────┐    │
│  │    Container Discovery      │    │
│  │  polls Docker API every 1s  │    │
│  │  detects start/stop         │    │
│  └──────────────┬──────────────┘    │
│                 │                   │
│  ┌──────────────▼──────────────┐    │
│  │    Metrics Collection       │    │
│  │  /sys/fs/cgroup   (limits)  │    │
│  │  /proc/<pid>/     (usage)   │    │
│  │  /sys/class/net/  (network) │    │
│  └──────────────┬──────────────┘    │
│                 │                   │
│  ┌──────────────▼──────────────┐    │
│  │    In-memory Storage        │    │
│  │  keeps ~2 min of history    │    │
│  └──────────────┬──────────────┘    │
│                 │                   │
│  ┌──────────────▼──────────────┐    │
│  │    Export Endpoints         │    │
│  │  /metrics  (Prometheus)     │    │
│  │  /api/v1.3 (REST API)       │    │
│  │  /containers (Web UI)       │    │
└──┴─────────────────────────────┴────┘

Deploy cAdvisor

Standalone Docker

# docker-compose.yml
cadvisor:
  image: gcr.io/cadvisor/cadvisor:v0.47.2
  container_name: cadvisor
  restart: unless-stopped
  ports:
    - "8080:8080"
  # Required volume mounts — read host filesystem
  volumes:
    - /:/rootfs:ro                          # root filesystem
    - /var/run:/var/run:ro                  # Docker socket dir
    - /var/run/docker.sock:/var/run/docker.sock:ro  # Docker API
    - /sys:/sys:ro                          # kernel/cgroups info
    - /var/lib/docker:/var/lib/docker:ro    # Docker data dir
    - /dev/disk:/dev/disk:ro                # disk info
  # Required for accessing kernel metrics
  privileged: true
  devices:
    - /dev/kmsg                             # kernel message buffer
  # Performance tuning
  command:
    - '--housekeeping_interval=10s'         # collect every 10s
    - '--max_housekeeping_interval=15s'
    - '--event_storage_event_limit=default=0'
    - '--event_storage_age_limit=default=0'
    - '--disable_metrics=percpu,sched,tcp,udp,disk,diskIO,hugetlb,referenced_memory,cpu_topology,resctrl'
    - '--docker_only=true'                  # only Docker containers
    - '--store_container_labels=false'

Kubernetes DaemonSet

# cadvisor runs on every node as a DaemonSet
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: cadvisor
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: cadvisor
  template:
    metadata:
      labels:
        app: cadvisor
    spec:
      hostNetwork: true
      hostPID: true
      containers:
      - name: cadvisor
        image: gcr.io/cadvisor/cadvisor:v0.47.2
        ports:
        - containerPort: 8080
          name: http
        volumeMounts:
        - name: rootfs
          mountPath: /rootfs
          readOnly: true
        - name: var-run
          mountPath: /var/run
          readOnly: true
        - name: sys
          mountPath: /sys
          readOnly: true
        - name: docker
          mountPath: /var/lib/docker
          readOnly: true
        - name: dev-disk
          mountPath: /dev/disk
          readOnly: true
        securityContext:
          privileged: true
        resources:
          requests:
            memory: 200Mi
            cpu: 150m
          limits:
            memory: 400Mi
            cpu: 300m
      volumes:
      - name: rootfs
        hostPath:
          path: /
      - name: var-run
        hostPath:
          path: /var/run
      - name: sys
        hostPath:
          path: /sys
      - name: docker
        hostPath:
          path: /var/lib/docker
      - name: dev-disk
        hostPath:
          path: /dev/disk

cAdvisor Web UI

Access at http://localhost:8080:

http://localhost:8080/containers/   → all containers overview
http://localhost:8080/docker/       → Docker-specific view
http://localhost:8080/metrics       → Prometheus metrics endpoint
Container detail page shows:
├── Isolation (CPU/memory limits set)
├── Usage (real-time CPU/memory charts)
├── Processes (running inside container)
└── Subcontainers (if applicable)

Key Metrics Exposed

cAdvisor exposes hundreds of metrics — here are the most important:

CPU Metrics
# ── Total CPU usage (all cores) ──────────────────────────────
# CPU seconds used — rate gives usage per second
container_cpu_usage_seconds_total{
  name="api",
  cpu="total"
}
# CPU usage % (actual percentage of one core)
rate(container_cpu_usage_seconds_total{
  name="api"
}[5m]) * 100
# CPU throttled time — how long container was throttled
container_cpu_cfs_throttled_seconds_total
# CPU throttle periods — how often throttled
container_cpu_cfs_throttled_periods_total
# CPU limit (from docker run --cpus)
container_spec_cpu_quota       # microseconds
container_spec_cpu_period      # period in microseconds
# CPU limit in cores
container_spec_cpu_quota / container_spec_cpu_period
# CPU usage % relative to limit
rate(container_cpu_usage_seconds_total{name="api"}[5m])
/ (container_spec_cpu_quota{name="api"}
   / container_spec_cpu_period{name="api"})
* 100

Memory Metrics

# ── Memory usage ─────────────────────────────────────────────
# Current memory usage (includes cache)
container_memory_usage_bytes{name="api"}
# Working set memory (excludes reclaimable cache)
# — best metric for actual memory pressure
container_memory_working_set_bytes{name="api"}
# RSS memory (resident set size — actual RAM used by app)
container_memory_rss{name="api"}
# Page cache (filesystem cache — reclaimable)
container_memory_cache{name="api"}
# Memory limit set on container
container_spec_memory_limit_bytes{name="api"}
# Memory usage % relative to limit
container_memory_working_set_bytes{name="api"}
/ container_spec_memory_limit_bytes{name="api"}
* 100
# Memory page faults (minor — no disk I/O)
container_memory_failures_total{
  name="api",
  type="pgfault",
  scope="container"
}
# Memory page faults (major — requires disk read)
container_memory_failures_total{
  name="api",
  type="pgmajfault",
  scope="container"
}

Network Metrics

# ── Network I/O ──────────────────────────────────────────────
# Bytes received per second
rate(container_network_receive_bytes_total{
  name="api"
}[5m])
# Bytes transmitted per second
rate(container_network_transmit_bytes_total{
  name="api"
}[5m])
# Packets received per second
rate(container_network_receive_packets_total{
  name="api"
}[5m])
# Packets transmitted per second
rate(container_network_transmit_packets_total{
  name="api"
}[5m])
# Receive errors
rate(container_network_receive_errors_total{
  name="api"
}[5m])
# Transmit errors
rate(container_network_transmit_errors_total{
  name="api"
}[5m])
# Dropped packets received
rate(container_network_receive_packets_dropped_total{
  name="api"
}[5m])

Disk / Filesystem Metrics

# ── Disk I/O ─────────────────────────────────────────────────
# Bytes read from disk per second
rate(container_fs_reads_bytes_total{
  name="api"
}[5m])
# Bytes written to disk per second
rate(container_fs_writes_bytes_total{
  name="api"
}[5m])
# Read operations per second (IOPS)
rate(container_fs_reads_total{
  name="api"
}[5m])
# Write operations per second (IOPS)
rate(container_fs_writes_total{
  name="api"
}[5m])
# Filesystem space used by container
container_fs_usage_bytes{
  name="api"
}
# Filesystem space limit
container_fs_limit_bytes{
  name="api"
}

Container Lifecycle Metrics

# ── Container state ──────────────────────────────────────────
# Container start time (unix timestamp)
container_start_time_seconds{name="api"}
# Container uptime in seconds
time() - container_start_time_seconds{name="api"}
# Last time container was seen alive
container_last_seen{name="api"}
# Detect container restarts (changes in start time)
changes(container_start_time_seconds{name="api"}[1h])

Important Metric Labels

cAdvisor adds rich labels to every metric:

container_cpu_usage_seconds_total{
  id="/docker/abc123",           # container ID path
  image="nginx:latest",          # image name
  name="my-nginx",               # container name
  container_label_com_docker_compose_project="myapp",
  container_label_com_docker_compose_service="nginx",
  container_label_com_docker_compose_version="2.0",
  cpu="total"
}
LabelValue exampleUse
namemy-nginxFilter by container name
imagenginx:latestFilter by image
id/docker/abc123Unique container ID
container_label_*compose project/serviceFilter by compose labels
interfaceeth0Network interface
device/dev/sdaDisk device

Prometheus Scrape Config for cAdvisor

# prometheus.yml
scrape_configs:
- job_name: 'cadvisor'
  static_configs:
  - targets: ['cadvisor:8080']
  # Drop metrics we don't need (reduce cardinality)
  metric_relabel_configs:
  # Drop pause containers (k8s infrastructure)
  - source_labels: [image]
    regex: 'k8s.gcr.io/pause.*'
    action: drop
  # Drop empty container names
  - source_labels: [name]
    regex: ''
    action: drop
  # Drop high-cardinality metrics not needed
  - source_labels: [__name__]
    regex: 'container_tasks_state|container_memory_failures_total'
    action: drop
  # Keep only Docker containers (not system cgroups)
  - source_labels: [container_label_com_docker_compose_service]
    regex: '.+'
    action: keep

Useful PromQL Queries

# ── Top Consumers ────────────────────────────────────────────
# Top 5 containers by CPU usage
topk(5,
  rate(container_cpu_usage_seconds_total{
    name!="", image!=""
  }[5m]) * 100
)
# Top 5 containers by memory (working set)
topk(5,
  container_memory_working_set_bytes{
    name!="", image!=""
  }
)
# Top 5 containers by network receive
topk(5,
  rate(container_network_receive_bytes_total{
    name!="", image!=""
  }[5m])
)
# Top 5 containers by disk writes
topk(5,
  rate(container_fs_writes_bytes_total{
    name!="", image!=""
  }[5m])
)
# ── Health Checks ────────────────────────────────────────────
# Containers using more than 80% of memory limit
container_memory_working_set_bytes{name!=""}
/ container_spec_memory_limit_bytes{name!=""} > 0.8
# Containers being CPU throttled
rate(container_cpu_cfs_throttled_seconds_total{
  name!=""
}[5m]) > 0
# Throttle % (how much CPU time is throttled)
rate(container_cpu_cfs_throttled_periods_total{
  name!=""
}[5m])
/ rate(container_cpu_cfs_periods_total{
  name!=""
}[5m])
* 100
# Containers that restarted in last hour
changes(container_start_time_seconds{
  name!="", image!=""
}[1h]) > 0
# ── Resource Efficiency ──────────────────────────────────────
# CPU limit utilization per container
rate(container_cpu_usage_seconds_total{name!=""}[5m])
/ (container_spec_cpu_quota{name!=""}
   / container_spec_cpu_period{name!=""})
* 100
# Memory limit utilization per container
container_memory_working_set_bytes{name!=""}
/ container_spec_memory_limit_bytes{name!=""}
* 100
# Containers with no resource limits set
container_spec_memory_limit_bytes == 0

cAdvisor Grafana Dashboard

Import dashboard ID 14282 or build panels manually:

Docker Overview Dashboard
├── Row 1: Summary Stats
│   ├── Total containers running (stat)
│   ├── Total CPU usage % (gauge)
│   ├── Total memory usage (gauge)
│   └── Total network I/O (stat)
│
├── Row 2: CPU
│   ├── CPU usage by container (time series, stacked)
│   ├── CPU throttling % by container (time series)
│   └── CPU limit utilization (bar gauge)
│
├── Row 3: Memory
│   ├── Memory usage by container (time series, stacked)
│   ├── Memory working set by container (time series)
│   ├── Memory limit utilization % (bar gauge)
│   └── OOM events (stat)
│
├── Row 4: Network
│   ├── Network received by container (time series)
│   ├── Network transmitted by container (time series)
│   ├── Network errors (time series)
│   └── Dropped packets (time series)
│
└── Row 5: Disk
    ├── Disk read bytes by container (time series)
    ├── Disk write bytes by container (time series)
    ├── Disk IOPS (time series)
    └── Container filesystem usage (bar gauge)

Alert Rules for cAdvisor

# prometheus/rules/cadvisor_alerts.yml
groups:
- name: cadvisor
  rules:
  # Container down
  - alert: ContainerDown
    expr: |
      time() - container_last_seen{
        name!="",
        image!=""
      } > 60
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "Container down: {{ $labels.name }}"
      description: "Container has not been seen for 60 seconds"
  # High CPU throttling
  - alert: ContainerCPUThrottling
    expr: |
      rate(container_cpu_cfs_throttled_periods_total{name!=""}[5m])
      / rate(container_cpu_cfs_periods_total{name!=""}[5m])
      * 100 > 50
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "CPU throttling: {{ $labels.name }}"
      description: "{{ $value | printf \"%.0f\" }}% of CPU time is throttled"
  # High memory usage
  - alert: ContainerMemoryHigh
    expr: |
      container_memory_working_set_bytes{name!=""}
      / container_spec_memory_limit_bytes{name!=""}
      * 100 > 85
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High memory: {{ $labels.name }}"
      description: "Memory usage is {{ $value | printf \"%.1f\" }}% of limit"
  # Container OOM risk
  - alert: ContainerOOMRisk
    expr: |
      container_memory_working_set_bytes{name!=""}
      / container_spec_memory_limit_bytes{name!=""}
      * 100 > 95
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "OOM risk: {{ $labels.name }}"
      description: "Memory at {{ $value | printf \"%.1f\" }}% — OOM kill imminent"
  # Container restarting
  - alert: ContainerRestarting
    expr: |
      changes(container_start_time_seconds{
        name!="", image!=""
      }[30m]) > 3
    for: 0m
    labels:
      severity: warning
    annotations:
      summary: "Container restarting: {{ $labels.name }}"
      description: "Restarted {{ $value }} times in last 30 minutes"
  # No CPU limit set
  - alert: ContainerNoCPULimit
    expr: |
      container_spec_cpu_quota{name!="", image!=""} == -1
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "No CPU limit: {{ $labels.name }}"
      description: "Container has no CPU limit — can consume all host CPU"
  # No memory limit set
  - alert: ContainerNoMemoryLimit
    expr: |
      container_spec_memory_limit_bytes{
        name!="", image!=""
      } == 0
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "No memory limit: {{ $labels.name }}"
      description: "Container has no memory limit — OOM kill risk to host"

cAdvisor vs Node Exporter

They are complementary — not alternatives:

Node ExportercAdvisor
ScopeHost / OS levelContainer level
CPU metricsPer core, per modePer container
MemoryHost RAM breakdownPer container + limits
NetworkPer NIC, host-levelPer container
DiskPer device, per mountPer container writes
ProcessesHost process countContainer processes
LimitsN/ACPU/memory limits & usage
Best forIs the server healthy?Which container is the problem?
Debugging workflow:
                                    
Node Exporter → "Host CPU is 95%"
                      ↓
cAdvisor → "api container using 80% of host CPU"
                      ↓
App metrics → "api processing 10k req/s, 50ms p99"
                      ↓
Root cause found

cAdvisor Limitations

LimitationWorkaround
Only ~2 min in-memory historyUse Prometheus for long-term storage
High metric cardinality with many containersDrop unused metrics via relabeling
No application-level metricsAdd app-specific exporters
No log collectionUse Loki + Promtail alongside
No alertingUse Prometheus Alertmanager
Resource overhead on busy hostsTune --housekeeping_interval
No cross-host aggregationPrometheus federation or Thanos

Performance Tuning

# Reduce cAdvisor overhead on busy hosts
command:
  # Increase collection interval (default 1s)
  - '--housekeeping_interval=10s'
  # Disable metrics you don't need
  - '--disable_metrics=percpu,sched,tcp,udp,hugetlb,referenced_memory,cpu_topology,resctrl'
  # Only monitor Docker (not all cgroups)
  - '--docker_only=true'
  # Don't store container labels (reduce cardinality)
  - '--store_container_labels=false'
  # Allowlist only needed labels
  - '--allowlisted_container_labels=com.docker.compose.service,com.docker.compose.project'
  # Reduce in-memory storage
  - '--memory_storage_duration=1m'

cAdvisor is the standard tool for container-level observability — it answers the question “what is this specific container doing?” and is the foundation of container monitoring in both Docker and Kubernetes environments. Paired with Node Exporter for host metrics and Prometheus for storage, it gives you complete visibility from hardware up to individual container processes.

Monitor Linux Servers with Node Exporter Full

techhadoop linux

Node Exporter Full Dashboard Explained

What is Node Exporter Full?

Node Exporter Full is the most popular Grafana dashboard (ID: 1860) for Linux server monitoring. It provides a comprehensive view of every hardware and OS metric collected by Node Exporter — over 30 panels covering CPU, memory, disk, network, and system metrics.

Dashboard Overview

┌─────────────────────────────────────────────────────────────┐
│              NODE EXPORTER FULL — DASHBOARD LAYOUT          │
├─────────────────────────────────────────────────────────────┤
│  [Server selector] [Time range] [Refresh interval]          │
├──────────┬──────────┬──────────┬──────────┬────────────────┤
│  Uptime  │CPU Cores │  RAM     │  SWAP    │  Root FS       │
│  (stat)  │ (stat)   │ (stat)   │  (stat)  │  (stat)        │
├──────────┴──────────┴──────────┴──────────┴────────────────┤
│                    CPU USAGE (time series)                   │
├─────────────────────────────┬───────────────────────────────┤
│   CPU Basic (gauge)         │   CPU Busy (time series)      │
├─────────────────────────────┼───────────────────────────────┤
│   Memory Basic (gauge)      │   Memory Usage (time series)  │
├─────────────────────────────┴───────────────────────────────┤
│                    DISK I/O (time series)                    │
├─────────────────────────────────────────────────────────────┤
│                    NETWORK TRAFFIC (time series)             │
├─────────────────────────────┬───────────────────────────────┤
│   Disk Space (bar gauge)    │   Network Errors (time series)│
└─────────────────────────────┴───────────────────────────────┘

Section 1 — Quick Stats Row (Top)

The top row shows current snapshot values at a glance:

┌──────────┬──────────┬──────────┬──────────┬──────────────┐
│  Uptime  │CPU Cores │   RAM    │  SWAP    │  Root FS     │
│  45 days │    8     │ 31.2 GB  │ 2.0 GB   │  234 GB      │
└──────────┴──────────┴──────────┴──────────┴──────────────┘
Uptime
# How long the server has been running
(time() - node_boot_time_seconds{instance="$node", job="$job"})

Shows days, hours, minutes — quick health check. Server rebooted unexpectedly? Uptime drops.

CPU Cores
# Total logical CPU count
count(
  count by(cpu) (
    node_cpu_seconds_total{
      instance="$node",
      job="$job"
    }
  )
)
Total RAM
# Physical RAM in bytes
node_memory_MemTotal_bytes{
  instance="$node",
  job="$job"
}
SWAP Total
# Total swap space
node_memory_SwapTotal_bytes{
  instance="$node",
  job="$job"
}

High swap usage = memory pressure — app may be swapping pages to disk.

Root Filesystem
# Total size of root partition
node_filesystem_size_bytes{
  instance="$node",
  job="$job",
  mountpoint="/",
  fstype!="rootfs"
}

Section 2 — CPU Panels

CPU Basic (Gauge)
┌─────────────────────────────┐
│          CPU Basic          │
│                             │
│           67%               │
│    ████████████░░░░░        │
│   0%        [67%]      100% │
│   Green < 50  Yellow < 80   │
│              Red > 80       │
└─────────────────────────────┘
# Current overall CPU busy %
(1 - avg by(instance) (
  rate(node_cpu_seconds_total{
    instance="$node",
    job="$job",
    mode="idle"
  }[$__rate_interval])
)) * 100

Thresholds typically set at:

  • 🟢 Green: 0-50%
  • 🟡 Yellow: 50-80%
  • 🔴 Red: 80-100%
CPU Usage Time Series

The most detailed CPU panel — shows how CPU time is being spent broken down by mode:

100% ┤                                    ┐
     │  ██ steal                          │
 80% ┤  ██ iowait                         │
     │  ██ irq/softirq                    │
 60% ┤  ██ system                         │
     │  ████████ user                     │
 40% ┤  ████████████████                  │
     │  ████████████████████ idle         │
  0% ┤────────────────────────────────────┘
     12:00      13:00      14:00
# CPU user time (app code)
avg by(instance) (
  rate(node_cpu_seconds_total{
    instance="$node",
    job="$job",
    mode="user"
  }[$__rate_interval])
) * 100
# CPU system time (kernel)
avg by(instance) (
  rate(node_cpu_seconds_total{
    instance="$node",
    job="$job",
    mode="system"
  }[$__rate_interval])
) * 100
# CPU iowait (waiting for disk I/O)
avg by(instance) (
  rate(node_cpu_seconds_total{
    instance="$node",
    job="$job",
    mode="iowait"
  }[$__rate_interval])
) * 100
# CPU steal (hypervisor stealing CPU from VM)
avg by(instance) (
  rate(node_cpu_seconds_total{
    instance="$node",
    job="$job",
    mode="steal"
  }[$__rate_interval])
) * 100
# CPU softirq (network interrupts, timers)
avg by(instance) (
  rate(node_cpu_seconds_total{
    instance="$node",
    job="$job",
    mode="softirq"
  }[$__rate_interval])
) * 100
Understanding CPU Modes
ModeMeaningHigh value means
userApp/process code runningHigh app activity — normal
systemKernel code runningMany syscalls, context switches
iowaitCPU idle waiting for I/ODisk bottleneck
stealHypervisor stealing CPUVM is being throttled by host
irqHardware interrupt handlingNetwork/disk interrupt storm
softirqSoftware interrupt handlingHigh network traffic
idleCPU doing nothingPlenty of headroom
niceLow-priority user processesBackground tasks running
CPU Busy by Core

Shows individual core utilization — detects uneven load distribution:

# Per-core CPU usage
(1 - rate(node_cpu_seconds_total{
  instance="$node",
  job="$job",
  mode="idle"
}[$__rate_interval])) * 100
Core 0: ████████████████ 78%
Core 1: ████ 22%
Core 2: ████████████████████ 95%  ← hot core
Core 3: ██ 10%

One hot core = single-threaded bottleneck. All cores high = genuinely compute-bound.

System Load Average
┌────────────────────────────────────────┐
│           System Load / CPU            │
│                                        │
│  1.2 ┤      ╭──╮                       │
│  0.8 ┤   ╭──╯  ╰──╮    1min load      │
│  0.4 ┤───╯         ╰── 5min load      │
│  0.0 ┤              ── 15min load     │
└────────────────────────────────────────┘
# Load average normalized per CPU core
node_load1{instance="$node", job="$job"}
  / count by(instance)(
    node_cpu_seconds_total{
      instance="$node",
      job="$job",
      mode="idle"
    }
  )
node_load5{instance="$node", job="$job"}
  / count by(instance)(...)
node_load15{instance="$node", job="$job"}
  / count by(instance)(...)

Interpreting load average:

< 1.0 per core  = plenty of headroom
= 1.0 per core  = fully utilized, no queue
> 1.0 per core  = processes waiting for CPU
> 2.0 per core  = severe overload
Context Switches and Interrupts
# Context switches per second
rate(node_context_switches_total{
  instance="$node",
  job="$job"
}[$__rate_interval])
# Hardware interrupts per second
rate(node_intr_total{
  instance="$node",
  job="$job"
}[$__rate_interval])

High context switches = many processes competing for CPU, or lots of I/O-bound processes sleeping and waking.

Section 3 — Memory Panels

Memory Basic (Gauge)
┌──────────────────────────────┐
│         Memory Basic         │
│                              │
│            82%               │
│    ████████████████░░        │
│   0%          [82%]     100% │
└──────────────────────────────┘
# RAM usage %
(1 - (
  node_memory_MemAvailable_bytes{instance="$node", job="$job"}
  / node_memory_MemTotal_bytes{instance="$node", job="$job"}
)) * 100

Note: Uses MemAvailable not MemFree — available includes reclaimable cache, which is a better measure of actual free memory.

Memory Usage Breakdown (Time Series)
32GB ┤ ████████ Used (apps)
     │ ████ Buffers
     │ ██████████ Cached (filesystem cache)
     │ ████ Free
  0  ┤─────────────────────────────────────
# Actually used by apps (exclude cache/buffers)
node_memory_MemTotal_bytes{instance="$node", job="$job"}
- node_memory_MemFree_bytes{instance="$node", job="$job"}
- node_memory_Buffers_bytes{instance="$node", job="$job"}
- node_memory_Cached_bytes{instance="$node", job="$job"}
# Filesystem cache (reclaimable)
node_memory_Cached_bytes{instance="$node", job="$job"}
# Buffer cache (reclaimable)
node_memory_Buffers_bytes{instance="$node", job="$job"}
# Truly free
node_memory_MemFree_bytes{instance="$node", job="$job"}
# SWAP used
node_memory_SwapTotal_bytes{instance="$node", job="$job"}
- node_memory_SwapFree_bytes{instance="$node", job="$job"}
Understanding Memory Types
Memory TypeDescriptionConcern level
UsedActive app memory🔴 High if > 80% of total
CachedLinux filesystem cache🟢 Normal — reclaimable
BuffersDisk write buffers🟢 Normal — reclaimable
FreeCompletely unused🟢 Low is OK if cache is high
AvailableFree + reclaimable✅ Best indicator of real free
SWAP UsedMemory paged to disk🔴 Any non-zero is a warning
SWAP Activity
# Swap pages swapped in per second (bad — reading from disk)
rate(node_vmstat_pswpin{instance="$node", job="$job"}[$__rate_interval])
# Swap pages swapped out per second (bad — writing to disk)
rate(node_vmstat_pswpout{instance="$node", job="$job"}[$__rate_interval])

Any swap activity means the system is memory constrained — apps are being paged to disk, causing severe performance degradation.

Memory Pages
# Page faults per second
rate(node_vmstat_pgfault{
  instance="$node",
  job="$job"
}[$__rate_interval])
# Major page faults (require disk I/O — worse)
rate(node_vmstat_pgmajfault{
  instance="$node",
  job="$job"
}[$__rate_interval])

Section 4 — Disk Panels

Disk Space Used (Bar Gauge)

Shows all mounted filesystems and their usage:

/          ████████████████░░░░  78%   (234 GB / 300 GB)
/data      ████████░░░░░░░░░░░░  42%   (420 GB / 1 TB)
/var/log   ████████████████████  96%   ← critical!
/boot      ████░░░░░░░░░░░░░░░░  18%   (180 MB / 1 GB)
# Disk usage % per mountpoint
(1 - node_filesystem_avail_bytes{
  instance="$node",
  job="$job",
  fstype!~"tmpfs|fuse.lxcfs|squashfs|vfat"
} / node_filesystem_size_bytes{
  instance="$node",
  job="$job",
  fstype!~"tmpfs|fuse.lxcfs|squashfs|vfat"
}) * 100
Disk I/O Time Series
200MB/s ┤         ╭──╮
        │  reads  │  │ writes
100MB/s ┤──────╮──╯  ╰──────╮
        │      │             │
   0    ┤──────╯             ╰───
        12:00       13:00
# Disk read throughput (bytes/sec)
rate(node_disk_read_bytes_total{
  instance="$node",
  job="$job",
  device=~"$disk"
}[$__rate_interval])
# Disk write throughput (bytes/sec)
rate(node_disk_written_bytes_total{
  instance="$node",
  job="$job",
  device=~"$disk"
}[$__rate_interval])
# Read IOPS (operations per second)
rate(node_disk_reads_completed_total{
  instance="$node",
  job="$job",
  device=~"$disk"
}[$__rate_interval])
# Write IOPS
rate(node_disk_writes_completed_total{
  instance="$node",
  job="$job",
  device=~"$disk"
}[$__rate_interval])
Disk I/O Utilization (Saturation)
# % of time disk was busy (saturation)
rate(node_disk_io_time_seconds_total{
  instance="$node",
  job="$job",
  device=~"$disk"
}[$__rate_interval]) * 100
Disk utilization interpretation:
0-40%   = disk has plenty of headroom
40-80%  = moderately busy
80-100% = disk is saturated — I/O bottleneck
> 100%  = queue building up (very slow disk)
Disk I/O Wait Time
# Average read wait time (milliseconds)
rate(node_disk_read_time_seconds_total{
  instance="$node",
  job="$job"
}[$__rate_interval])
/ rate(node_disk_reads_completed_total{
  instance="$node",
  job="$job"
}[$__rate_interval])
* 1000
# Average write wait time (milliseconds)
rate(node_disk_write_time_seconds_total{
  instance="$node",
  job="$job"
}[$__rate_interval])
/ rate(node_disk_writes_completed_total{
  instance="$node",
  job="$job"
}[$__rate_interval])
* 1000
LatencyDisk typeConcern
< 1msNVMe SSD✅ Excellent
1-5msSSD✅ Good
5-20msSSD under load🟡 Acceptable
20-100msHDD or slow SSD🔴 Poor
> 100msSeverely overloaded🔴 Critical

Section 5 — Network Panels

Network Traffic (Time Series)
1 GB/s ┤              ╭──╮
       │  ▲ received  │  │
500MB/s┤──╮────────╮──╯  ╰──╮
       │  │  ▼ sent│         │
   0   ┤──╯        ╰─────────╯
       12:00          13:00
# Network received bytes/sec (per interface)
rate(node_network_receive_bytes_total{
  instance="$node",
  job="$job",
  device=~"$nic"
}[$__rate_interval])
# Network transmitted bytes/sec
rate(node_network_transmit_bytes_total{
  instance="$node",
  job="$job",
  device=~"$nic"
}[$__rate_interval])
# Convert to bits (×8) for bandwidth comparison
rate(node_network_receive_bytes_total{...}[$__rate_interval]) * 8
Network Errors and Drops
# Receive errors per second
rate(node_network_receive_errs_total{
  instance="$node",
  job="$job",
  device=~"$nic"
}[$__rate_interval])
# Transmit errors per second
rate(node_network_transmit_errs_total{
  instance="$node",
  job="$job",
  device=~"$nic"
}[$__rate_interval])
# Receive packet drops (buffer overflow)
rate(node_network_receive_drop_total{
  instance="$node",
  job="$job",
  device=~"$nic"
}[$__rate_interval])
# Transmit packet drops
rate(node_network_transmit_drop_total{
  instance="$node",
  job="$job",
  device=~"$nic"
}[$__rate_interval])

Any non-zero errors or drops indicate:

  • Receive drops — NIC buffer overflow, CPU can’t process packets fast enough
  • Transmit errors — bad cable, network congestion, NIC issue
  • Errors — hardware problem, duplex mismatch
Network Packets
# Packets received per second
rate(node_network_receive_packets_total{
  instance="$node",
  job="$job",
  device=~"$nic"
}[$__rate_interval])
# Packets transmitted per second
rate(node_network_transmit_packets_total{
  instance="$node",
  job="$job",
  device=~"$nic"
}[$__rate_interval])

Section 6 — System Panels

Open File Descriptors
# Current open file descriptors
node_filefd_allocated{
  instance="$node",
  job="$job"
}
# System limit
node_filefd_maximum{
  instance="$node",
  job="$job"
}
# Usage percentage
node_filefd_allocated{instance="$node", job="$job"}
/ node_filefd_maximum{instance="$node", job="$job"}
* 100

Running out of file descriptors = apps fail to open files or sockets. Common with high-connection services like web servers, databases.

Processes

# Currently running processes (on CPU)
node_procs_running{instance="$node", job="$job"}
# Processes in uninterruptible sleep (D state — waiting for I/O)
node_procs_blocked{instance="$node", job="$job"}
# Total processes
node_procs_running + node_procs_blocked

High blocked processes = disk I/O bottleneck — processes stuck waiting for disk.

Systemd Failed Services

# Count of failed systemd services
count by(instance) (
  node_systemd_unit_state{
    instance="$node",
    job="$job",
    state="failed"
  } == 1
)
# Which services failed
node_systemd_unit_state{
  instance="$node",
  job="$job",
  state="failed"
} == 1

Dashboard Variables (Dropdowns)

The dashboard uses template variables so you can switch between servers:

Variable: $node
Query:    label_values(node_uname_info, instance)
→ Dropdown shows all connected servers
Variable: $job
Query:    label_values(node_uname_info{instance="$node"}, job)
→ Filters by job name
Variable: $disk
Query:    label_values(node_disk_io_time_seconds_total
            {instance="$node"}, device)
→ Dropdown of all disks (sda, sdb, nvme0n1 etc)
Variable: $nic
Query:    label_values(node_network_info
            {instance="$node"}, device)
→ Dropdown of all NICs (eth0, ens3 etc)
            excluding: lo, docker, veth, br

Reading the Dashboard — What to Look For

Scenario 1: High CPU, low I/O wait
─────────────────────────────────
user%  ████████████ 85%   → App is CPU-bound
system ██ 10%
iowait ░ 2%
→ Scale up CPU or optimize app code
Scenario 2: High iowait, low user CPU
──────────────────────────────────────
user%  ███ 20%
iowait ████████████ 70%   → Disk is bottleneck
→ Check disk latency panel
→ Upgrade to SSD or optimize DB queries
Scenario 3: High memory, swap activity
────────────────────────────────────────
RAM used  ███████████████ 95%
Swap used ████████ 40%    → OOM risk
Swap I/O  ██ active       → Severe performance hit
→ Add RAM or reduce app memory usage
Scenario 4: Network drops increasing
──────────────────────────────────────
RX drops ████ increasing  → NIC buffer overflow
→ Tune net.core.rmem_max
→ Check if CPU can keep up with IRQs
Scenario 5: Load > 1.0 per core, low CPU%
───────────────────────────────────────────
Load/core  1.8            → Processes queued
CPU user%  30%            → Not CPU-bound
iowait%    60%            → I/O queue is the bottleneck
→ Disk or network I/O is causing the queue

Import the Dashboard

# Method 1 — Via Grafana UI
# Go to Dashboards → Import → Enter ID: 1860 → Load
# Select Prometheus datasource → Import
# Method 2 — Via API
curl -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "dashboard": {"id": null},
    "inputs": [{
      "name": "DS_PROMETHEUS",
      "pluginId": "prometheus",
      "type": "datasource",
      "value": "Prometheus"
    }],
    "overwrite": true
  }' \
  http://admin:password@localhost:3000/api/dashboards/import
# Method 3 — Provision via file (GitOps)
# Download dashboard JSON
curl https://grafana.com/api/dashboards/1860/revisions/latest/download \
  -o grafana/dashboards/node-exporter-full.json
# Grafana picks it up automatically via provisioning config

Node Exporter Full is the single most useful starting point for Linux server monitoring — it gives you complete visibility into every layer of system performance from a single dashboard, with enough detail to diagnose almost any server issue without SSH-ing into the box.

Monitor Linux and Docker with Grafana & Prometheus

techhadoop docker, linux

Monitor Linux Server and Docker with Grafana and Prometheus

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                     LINUX SERVER                             │
│                                                              │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐   │
│  │ Node Exporter│  │cAdvisor      │  │ Docker Engine    │   │
│  │              │  │              │  │ (metrics endpoint│   │
│  │ CPU/RAM/Disk │  │ Container    │  │  optional)       │   │
│  │ Network/FS   │  │ metrics      │  │                  │   │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────────┘   │
│         │                 │                  │               │
│         └─────────────────┴──────────────────┘               │
│                           │ scrape                           │
│                    ┌──────▼───────┐                          │
│                    │  Prometheus  │                          │
│                    │              │                          │
│                    │  stores      │                          │
│                    │  metrics     │                          │
│                    └──────┬───────┘                          │
│                           │ query                           │
│                    ┌──────▼───────┐                          │
│                    │   Grafana    │                          │
│                    │              │                          │
│                    │  dashboards  │                          │
│                    │  alerts      │                          │
│                    └──────────────┘                          │
└─────────────────────────────────────────────────────────────┘

Project Structure

monitoring/
├── docker-compose.yml
├── prometheus/
│   ├── prometheus.yml
│   └── rules/
│       ├── linux_alerts.yml
│       └── docker_alerts.yml
├── grafana/
│   ├── provisioning/
│   │   ├── datasources/
│   │   │   └── prometheus.yml
│   │   └── dashboards/
│   │       └── dashboard.yml
│   └── dashboards/
│       ├── linux-server.json
│       └── docker.json
└── alertmanager/
    └── alertmanager.yml

Step 1 — Docker Compose Stack

# docker-compose.yml
version: '3.8'
networks:
  monitoring:
    driver: bridge
volumes:
  prometheus_data: {}
  grafana_data: {}
services:
  # ── Prometheus ───────────────────────────────────────────
  prometheus:
    image: prom/prometheus:v2.49.0
    container_name: prometheus
    restart: unless-stopped
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus/rules:/etc/prometheus/rules
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=30d'    # keep 30 days
      - '--storage.tsdb.retention.size=10GB'
      - '--web.enable-lifecycle'               # hot reload config
      - '--web.enable-admin-api'
    networks:
      - monitoring
  # ── Grafana ──────────────────────────────────────────────
  grafana:
    image: grafana/grafana:10.2.0
    container_name: grafana
    restart: unless-stopped
    ports:
      - "3000:3000"
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning
      - ./grafana/dashboards:/var/lib/grafana/dashboards
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=SecurePass123
      - GF_USERS_ALLOW_SIGN_UP=false
      - GF_SERVER_DOMAIN=grafana.yourdomain.com
      - GF_SMTP_ENABLED=true
      - GF_SMTP_HOST=smtp.gmail.com:587
      - GF_SMTP_USER=alerts@yourdomain.com
      - GF_SMTP_PASSWORD=your-smtp-password
      - GF_SMTP_FROM_ADDRESS=alerts@yourdomain.com
    networks:
      - monitoring
    depends_on:
      - prometheus
  # ── Node Exporter (Linux metrics) ────────────────────────
  node-exporter:
    image: prom/node-exporter:v1.7.0
    container_name: node-exporter
    restart: unless-stopped
    ports:
      - "9100:9100"
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.rootfs=/rootfs'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
      - '--collector.systemd'          # systemd service metrics
      - '--collector.processes'        # process metrics
    pid: host                          # see host processes
    network_mode: host                 # see host network stats
    cap_add:
      - SYS_TIME
  # ── cAdvisor (Docker container metrics) ──────────────────
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.47.2
    container_name: cadvisor
    restart: unless-stopped
    ports:
      - "8080:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker:/var/lib/docker:ro
      - /dev/disk:/dev/disk:ro
    privileged: true
    devices:
      - /dev/kmsg
    networks:
      - monitoring
  # ── Alertmanager ─────────────────────────────────────────
  alertmanager:
    image: prom/alertmanager:v0.26.0
    container_name: alertmanager
    restart: unless-stopped
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml
    command:
      - '--config.file=/etc/alertmanager/alertmanager.yml'
      - '--storage.path=/alertmanager'
      - '--web.external-url=http://alertmanager.yourdomain.com'
    networks:
      - monitoring

Step 2 — Prometheus Configuration

# prometheus/prometheus.yml
global:
  scrape_interval: 15s          # collect metrics every 15s
  evaluation_interval: 15s      # evaluate rules every 15s
  scrape_timeout: 10s
  external_labels:
    cluster: 'production'
    environment: 'prod'
# Alertmanager connection
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - alertmanager:9093
# Load alert rules
rule_files:
  - /etc/prometheus/rules/linux_alerts.yml
  - /etc/prometheus/rules/docker_alerts.yml
scrape_configs:
  # ── Prometheus self-monitoring ────────────────────────────
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']
    metrics_path: /metrics
  # ── Linux Server (Node Exporter) ─────────────────────────
  - job_name: 'node-exporter'
    static_configs:
    - targets: ['node-exporter:9100']
      labels:
        server: 'linux-prod-01'
        env: 'production'
  # Multiple servers
  - job_name: 'linux-servers'
    static_configs:
    - targets:
      - '10.0.1.10:9100'
      - '10.0.1.11:9100'
      - '10.0.1.12:9100'
      labels:
        env: 'production'
    relabel_configs:
    - source_labels: [__address__]
      target_label: instance
      regex: '([^:]+):.*'
      replacement: '$1'
  # ── Docker Containers (cAdvisor) ─────────────────────────
  - job_name: 'cadvisor'
    static_configs:
    - targets: ['cadvisor:8080']
      labels:
        server: 'linux-prod-01'
    metric_relabel_configs:
    # Drop high-cardinality metrics we don't need
    - source_labels: [__name__]
      regex: 'container_tasks_state|container_memory_failures_total'
      action: drop
    # Keep only running containers
    - source_labels: [container_label_com_docker_compose_service]
      regex: '.+'
      action: keep
  # ── Docker Engine metrics (optional) ─────────────────────
  - job_name: 'docker-engine'
    static_configs:
    - targets: ['host.docker.internal:9323']
  # ── Grafana self-monitoring ───────────────────────────────
  - job_name: 'grafana'
    static_configs:
    - targets: ['grafana:3000']
    metrics_path: /metrics

Step 3 — Alert Rules

# prometheus/rules/linux_alerts.yml
groups:
- name: linux.server
  interval: 30s
  rules:
  # ── CPU Alerts ───────────────────────────────────────────
  - alert: HighCPUUsage
    expr: |
      100 - (avg by(instance) (
        rate(node_cpu_seconds_total{mode="idle"}[5m])
      ) * 100) > 85
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High CPU usage on {{ $labels.instance }}"
      description: "CPU usage is {{ $value | printf \"%.1f\" }}% (threshold: 85%)"
  - alert: CriticalCPUUsage
    expr: |
      100 - (avg by(instance) (
        rate(node_cpu_seconds_total{mode="idle"}[5m])
      ) * 100) > 95
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "Critical CPU usage on {{ $labels.instance }}"
      description: "CPU usage is {{ $value | printf \"%.1f\" }}%"
  # ── Memory Alerts ─────────────────────────────────────────
  - alert: HighMemoryUsage
    expr: |
      (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 85
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High memory usage on {{ $labels.instance }}"
      description: "Memory usage is {{ $value | printf \"%.1f\" }}%"
  - alert: CriticalMemoryUsage
    expr: |
      (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 95
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "Critical memory usage on {{ $labels.instance }}"
      description: "Memory usage is {{ $value | printf \"%.1f\" }}%"
  # ── Disk Alerts ───────────────────────────────────────────
  - alert: DiskSpaceLow
    expr: |
      (node_filesystem_avail_bytes{mountpoint="/",fstype!="tmpfs"} /
       node_filesystem_size_bytes{mountpoint="/",fstype!="tmpfs"}) * 100 < 20
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Low disk space on {{ $labels.instance }}"
      description: "Disk {{ $labels.mountpoint }} has {{ $value | printf \"%.1f\" }}% free"
  - alert: DiskSpaceCritical
    expr: |
      (node_filesystem_avail_bytes{mountpoint="/",fstype!="tmpfs"} /
       node_filesystem_size_bytes{mountpoint="/",fstype!="tmpfs"}) * 100 < 10
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "Critical disk space on {{ $labels.instance }}"
      description: "Only {{ $value | printf \"%.1f\" }}% disk space remaining"
  - alert: DiskWillFillIn24h
    expr: |
      predict_linear(
        node_filesystem_avail_bytes{fstype!="tmpfs"}[6h], 24 * 3600
      ) < 0
    for: 1h
    labels:
      severity: warning
    annotations:
      summary: "Disk will fill in 24h on {{ $labels.instance }}"
  # ── Network Alerts ────────────────────────────────────────
  - alert: HighNetworkErrors
    expr: |
      rate(node_network_receive_errs_total[5m]) > 10
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High network errors on {{ $labels.instance }}"
      description: "{{ $value | printf \"%.0f\" }} errors/sec on {{ $labels.device }}"
  # ── System Alerts ─────────────────────────────────────────
  - alert: ServerDown
    expr: up{job="node-exporter"} == 0
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "Server DOWN: {{ $labels.instance }}"
      description: "Node exporter is not reachable"
  - alert: HighLoadAverage
    expr: |
      node_load15 / count by(instance)(
        node_cpu_seconds_total{mode="idle"}
      ) > 0.9
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "High load average on {{ $labels.instance }}"
      description: "15min load average is {{ $value | printf \"%.2f\" }} per core"
  - alert: SystemdServiceFailed
    expr: |
      node_systemd_unit_state{state="failed"} == 1
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Systemd service failed on {{ $labels.instance }}"
      description: "Service {{ $labels.name }} is in failed state"
  - alert: ClockSkewDetected
    expr: |
      abs(node_timex_offset_seconds) > 0.05
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Clock skew on {{ $labels.instance }}"
      description: "Clock offset is {{ $value }}s"
# prometheus/rules/docker_alerts.yml
groups:
- name: docker.containers
  rules:
  # ── Container Status Alerts ───────────────────────────────
  - alert: ContainerDown
    expr: |
      absent(container_last_seen{
        name!="",
        name!~".*_tmp.*"
      })
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "Container down: {{ $labels.name }}"
  - alert: ContainerRestarting
    expr: |
      rate(container_last_seen{name!=""}[5m]) == 0
      and on(name)
      changes(container_last_seen{name!=""}[10m]) > 2
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Container restarting: {{ $labels.name }}"
  # ── Container CPU Alerts ──────────────────────────────────
  - alert: ContainerHighCPU
    expr: |
      (rate(container_cpu_usage_seconds_total{
        name!="",
        image!=""
      }[5m]) * 100) > 80
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High CPU in container {{ $labels.name }}"
      description: "Container CPU is {{ $value | printf \"%.1f\" }}%"
  # ── Container Memory Alerts ───────────────────────────────
  - alert: ContainerHighMemory
    expr: |
      (container_memory_usage_bytes{name!="", image!=""} /
       container_spec_memory_limit_bytes{name!="", image!=""} * 100) > 85
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High memory in container {{ $labels.name }}"
      description: "Memory usage is {{ $value | printf \"%.1f\" }}% of limit"
  - alert: ContainerOOMKilled
    expr: |
      kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1
      or
      container_oom_events_total > 0
    for: 0m
    labels:
      severity: critical
    annotations:
      summary: "Container OOM killed: {{ $labels.name }}"
  # ── Container Disk Alerts ─────────────────────────────────
  - alert: ContainerHighDiskWrite
    expr: |
      rate(container_fs_writes_bytes_total{
        name!="",
        image!=""
      }[5m]) > 50000000    # 50MB/s
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High disk writes in container {{ $labels.name }}"
      description: "Writing {{ $value | humanize }}B/s"

Step 4 — Alertmanager Configuration

# alertmanager/alertmanager.yml
global:
  resolve_timeout: 5m
  slack_api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'
  smtp_smarthost: 'smtp.gmail.com:587'
  smtp_from: 'alerts@yourdomain.com'
  smtp_auth_username: 'alerts@yourdomain.com'
  smtp_auth_password: 'your-app-password'
# Route tree
route:
  group_by: ['alertname', 'instance']
  group_wait: 30s          # wait before sending first alert
  group_interval: 5m       # wait between alert groups
  repeat_interval: 4h      # resend if still firing
  receiver: 'slack-default'
  routes:
  # Critical → PagerDuty + Slack immediately
  - match:
      severity: critical
    receiver: 'pagerduty-critical'
    group_wait: 10s
    repeat_interval: 1h
    continue: true         # also send to default
  # Warning → Slack only
  - match:
      severity: warning
    receiver: 'slack-warnings'
    repeat_interval: 4h
  # Server down → immediate notification
  - match:
      alertname: ServerDown
    receiver: 'pagerduty-critical'
    group_wait: 0s         # no delay for server down
receivers:
# Default Slack channel
- name: 'slack-default'
  slack_configs:
  - channel: '#alerts'
    title: '{{ template "slack.title" . }}'
    text: '{{ template "slack.text" . }}'
    color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}'
    send_resolved: true
# Warnings channel
- name: 'slack-warnings'
  slack_configs:
  - channel: '#alerts-warning'
    title: '⚠️ {{ .GroupLabels.alertname }}'
    text: |
      {{ range .Alerts }}
      *Instance:* {{ .Labels.instance }}
      *Description:* {{ .Annotations.description }}
      {{ end }}
    send_resolved: true
# PagerDuty for critical
- name: 'pagerduty-critical'
  pagerduty_configs:
  - routing_key: 'YOUR_PAGERDUTY_ROUTING_KEY'
    description: '{{ .GroupLabels.alertname }}: {{ .CommonAnnotations.summary }}'
    severity: critical
# Email notifications
- name: 'email-alerts'
  email_configs:
  - to: 'devops-team@yourdomain.com'
    subject: '[{{ .Status | toUpper }}] {{ .GroupLabels.alertname }}'
    html: |
      <h2>{{ .GroupLabels.alertname }}</h2>
      {{ range .Alerts }}
      <p><b>Instance:</b> {{ .Labels.instance }}</p>
      <p><b>Description:</b> {{ .Annotations.description }}</p>
      {{ end }}
    send_resolved: true
inhibit_rules:
# If server is down, suppress all other alerts for that server
- source_match:
    alertname: ServerDown
  target_match_re:
    alertname: '.+'
  equal: ['instance']

Step 5 — Grafana Provisioning

# grafana/provisioning/datasources/prometheus.yml
apiVersion: 1
datasources:
- name: Prometheus
  type: prometheus
  access: proxy
  url: http://prometheus:9090
  isDefault: true
  editable: false
  jsonData:
    timeInterval: "15s"
    queryTimeout: "60s"
    httpMethod: POST
# grafana/provisioning/dashboards/dashboard.yml
apiVersion: 1
providers:
- name: 'default'
  orgId: 1
  folder: 'Server Monitoring'
  type: file
  disableDeletion: false
  editable: true
  updateIntervalSeconds: 30
  options:
    path: /var/lib/grafana/dashboards
    foldersFromFilesStructure: true

Step 6 — Key PromQL Queries

# ── CPU ────────────────────────────────────────────────────
# CPU usage % per server
100 - (avg by(instance) (
  rate(node_cpu_seconds_total{mode="idle"}[5m])
) * 100)
# CPU by mode (user, system, iowait, steal)
avg by(instance, mode) (
  rate(node_cpu_seconds_total{mode!="idle"}[5m])
) * 100
# Top 5 CPU-consuming containers
topk(5,
  rate(container_cpu_usage_seconds_total{
    name!="", image!=""
  }[5m]) * 100
)
# ── Memory ─────────────────────────────────────────────────
# Memory usage %
(1 - (node_memory_MemAvailable_bytes /
      node_memory_MemTotal_bytes)) * 100
# Memory breakdown (used, cached, buffered, free)
node_memory_MemTotal_bytes - node_memory_MemFree_bytes
  - node_memory_Buffers_bytes - node_memory_Cached_bytes
# Container memory usage vs limit
container_memory_usage_bytes{name!=""}
  / container_spec_memory_limit_bytes{name!=""} * 100
# ── Disk ───────────────────────────────────────────────────
# Disk usage % per mount
(1 - node_filesystem_avail_bytes{fstype!="tmpfs"} /
     node_filesystem_size_bytes{fstype!="tmpfs"}) * 100
# Disk I/O read/write bytes per second
rate(node_disk_read_bytes_total[5m])
rate(node_disk_written_bytes_total[5m])
# Disk I/O wait time (saturation)
rate(node_disk_io_time_seconds_total[5m])
# Predict disk full in hours
predict_linear(
  node_filesystem_avail_bytes{mountpoint="/"}[6h],
  3600
) / 1024 / 1024 / 1024   # convert to GB
# ── Network ────────────────────────────────────────────────
# Network bandwidth in/out per interface
rate(node_network_receive_bytes_total{
  device!~"lo|docker.*|veth.*"
}[5m]) * 8    # convert to bits
rate(node_network_transmit_bytes_total{
  device!~"lo|docker.*|veth.*"
}[5m]) * 8
# Network errors
rate(node_network_receive_errs_total[5m])
rate(node_network_transmit_errs_total[5m])
# ── Docker / Containers ────────────────────────────────────
# Running containers count
count(container_last_seen{name!="", image!=""})
# Container CPU usage %
rate(container_cpu_usage_seconds_total{
  name!="", image!=""
}[5m]) * 100
# Container network traffic
rate(container_network_receive_bytes_total{name!=""}[5m])
rate(container_network_transmit_bytes_total{name!=""}[5m])
# Container restart count
changes(container_start_time_seconds{name!=""}[1h])
# ── System ─────────────────────────────────────────────────
# System load per CPU core
node_load1 / count by(instance)(
  node_cpu_seconds_total{mode="idle"}
)
# Open file descriptors
node_filefd_allocated / node_filefd_maximum * 100
# System uptime in days
(time() - node_boot_time_seconds) / 86400
# Number of processes
node_procs_running
node_procs_blocked

Step 7 — Deploy and Manage

# Start the monitoring stack
docker-compose up -d
# Check all services running
docker-compose ps
# NAME            STATUS          PORTS
# prometheus      Up              0.0.0.0:9090->9090/tcp
# grafana         Up              0.0.0.0:3000->3000/tcp
# node-exporter   Up              0.0.0.0:9100->9100/tcp
# cadvisor        Up              0.0.0.0:8080->8080/tcp
# alertmanager    Up              0.0.0.0:9093->9093/tcp
# Check Prometheus targets (all should be UP)
curl http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'
# Reload Prometheus config (without restart)
curl -X POST http://localhost:9090/-/reload
# Check Prometheus rules loaded
curl http://localhost:9090/api/v1/rules | jq '.data.groups[].name'
# View Grafana logs
docker-compose logs grafana -f
# Check alertmanager config is valid
docker run --rm \
  -v $(pwd)/alertmanager:/config \
  prom/alertmanager:v0.26.0 \
  --config.file=/config/alertmanager.yml \
  check-config
# Backup Prometheus data
docker run --rm \
  -v prometheus_data:/data \
  -v $(pwd)/backup:/backup \
  alpine tar czf /backup/prometheus-$(date +%Y%m%d).tar.gz /data
# Update stack
docker-compose pull
docker-compose up -d

Step 8 — Install Node Exporter on Bare Metal

If Prometheus runs separately from the monitored server:

# Download and install node exporter
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar xvf node_exporter-1.7.0.linux-amd64.tar.gz
sudo cp node_exporter-1.7.0.linux-amd64/node_exporter /usr/local/bin/
# Create systemd service
sudo tee /etc/systemd/system/node_exporter.service << EOF
[Unit]
Description=Node Exporter
After=network.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter \
  --collector.systemd \
  --collector.processes \
  --web.listen-address=:9100
[Install]
WantedBy=multi-user.target
EOF
# Create user and start service
sudo useradd -rs /bin/false node_exporter
sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter
# Verify
curl http://localhost:9100/metrics | head -20

Grafana Dashboard Setup

# Access Grafana
open http://localhost:3000
# Login: admin / SecurePass123
# Import community dashboards via ID:
# Node Exporter Full:     ID 1860
# Docker monitoring:      ID 893
# cAdvisor:               ID 14282
# Prometheus stats:       ID 3662
# Import via CLI
curl -X POST \
  http://admin:SecurePass123@localhost:3000/api/dashboards/import \
  -H 'Content-Type: application/json' \
  -d '{
    "dashboard": {"id": null, "uid": null},
    "folderId": 0,
    "inputs": [{"name": "DS_PROMETHEUS", "type": "datasource",
                "pluginId": "prometheus", "value": "Prometheus"}],
    "overwrite": false,
    "path": "1860"
  }'

Useful Grafana Panel Examples

Linux Server Dashboard panels:
├── CPU Usage gauge          (0-100%, threshold at 80/95)
├── Memory Usage gauge       (0-100%, threshold at 80/95)
├── Disk Usage per mount     (bar gauge)
├── CPU Usage over time      (time series, stacked by mode)
├── Memory breakdown         (time series, stacked)
├── Network bandwidth        (time series, in/out)
├── Disk I/O                 (time series, read/write)
├── System Load              (time series, 1/5/15 min)
├── Top processes            (table)
└── Uptime stat              (stat panel)
Docker Dashboard panels:
├── Running containers       (stat)
├── Container CPU top 10     (bar chart)
├── Container Memory top 10  (bar chart)
├── Container restarts       (table with alert)
├── Network I/O per container(time series)
├── Container disk I/O       (time series)
└── Container logs           (logs panel via Loki)

Security Hardening

# Add basic auth to Prometheus
# Use nginx reverse proxy in front
nginx:
  image: nginx:alpine
  ports:
    - "9090:80"
  volumes:
    - ./nginx/prometheus.conf:/etc/nginx/conf.d/default.conf
    - ./nginx/.htpasswd:/etc/nginx/.htpasswd
# nginx/prometheus.conf
server {
  listen 80;
  location / {
    auth_basic "Prometheus";
    auth_basic_user_file /etc/nginx/.htpasswd;
    proxy_pass http://prometheus:9090;
  }
}
# Generate password file
htpasswd -c ./nginx/.htpasswd admin
# Restrict firewall — only allow internal access
ufw allow from 10.0.0.0/8 to any port 9090
ufw allow from 10.0.0.0/8 to any port 9100
ufw allow from 10.0.0.0/8 to any port 8080
ufw allow from anywhere to any port 3000   # Grafana only public

Quick Access URLs

Grafana:       http://localhost:3000      admin/SecurePass123
Prometheus:    http://localhost:9090
Alertmanager:  http://localhost:9093
Node Exporter: http://localhost:9100/metrics
cAdvisor:      http://localhost:8080

This gives you a production-ready monitoring stack — Linux server metrics via Node Exporter, Docker container metrics via cAdvisor, intelligent alerting via Alertmanager, and beautiful dashboards via Grafana, all wired together with Prometheus as the metrics backbone.

Linux network

techhadoop linux , , , ,

Linux network issues can come from a lot of places—misconfiguration, hardware problems, DNS failures, firewall rules, or even upstream network outages. The trick is to diagnose step-by-step instead of guessing. Here’s a practical workflow you can follow.

1. Check Basic Connectivity

Start simple—make sure the system is even connected.

ip a
  • Look for an IP address (not just 127.0.0.1)
  • Check if the interface is UP
ip link
  • Verify your interface (e.g., eth0, wlan0) is not DOWN

If it’s down:

sudo ip link set eth0 up

2. Test Network Reachability

Try pinging known addresses:

ping -c 4 8.8.8.8
  • If this fails → no internet or routing issue

Then test DNS:

ping -c 4 google.com
  • If IP ping works but domain fails → DNS problem

3. Check Routing Table

ip route

Look for a default route:

default via 192.168.1.1 dev eth0

If missing, add it:

sudo ip route add default via <gateway-ip>

4. Verify DNS Configuration

Check:

cat /etc/resolv.conf

You should see something like:

nameserver 8.8.8.8

If not, temporarily add:

sudo nano /etc/resolv.conf

Or if using systemd:

systemd-resolve --status

5. Check Network Manager / Services

Depending on your system:

Using NetworkManager

nmcli device status
nmcli connection show

Restart:

sudo systemctl restart NetworkManager

Using systemd-networkd

networkctl status
sudo systemctl restart systemd-networkd

6. Inspect Firewall Rules

Firewall can silently block traffic.

sudo iptables -L -n

Or if using UFW:

sudo ufw status

Temporarily disable (for testing):

sudo ufw disable

7. Check Open Ports

ss -tulnp

This shows:

  • Listening services
  • Ports in use

8. Test with traceroute

See where packets stop:

traceroute 8.8.8.8

If it stops early → local network issue
If it goes far but fails → ISP or remote issue

9. Check Logs

Logs often reveal the real problem:

journalctl -u NetworkManager
journalctl -xe
dmesg | grep -i network

10. Wireless-Specific Checks

If using Wi-Fi:

iwconfig

Scan networks:

iw dev wlan0 scan

Reconnect:

nmcli device wifi connect "SSID" password "PASSWORD"

11. Hardware / Driver Issues

Check drivers:

lspci | grep -i network
lsmod | grep <driver>

If missing drivers, you may need to install firmware packages.

Common Problems & Quick Fixes

  • No IP address → DHCP issue → restart network service
  • Can ping IP but not domain → DNS misconfigured
  • Interface down → enable with ip link set
  • Connected but no internet → gateway missing
  • Wi-Fi connected but slow → interference or driver issue

On Ubuntu, networking is usually managed by NetworkManager, so troubleshooting is a bit more standardized. Let’s narrow this down quickly with a focused Ubuntu workflow.

1. Check if NetworkManager is running

sudo systemctl status NetworkManager

If it’s not active:

sudo systemctl restart NetworkManager

2. See connection status

nmcli device status

You’ll see something like:

  • connected → good
  • disconnected → problem
  • unmanaged → config issue

To reconnect:

nmcli connection up <connection-name>

3. Check IP address

ip a

Look for something like:

inet 192.168.x.x

❌ No IP? Try renewing DHCP:

sudo dhclient -v

4. Test connectivity

ping -c 4 8.8.8.8
  • ❌ fails → network/gateway issue
  • ✅ works → move to DNS test:
ping -c 4 google.com
  • ❌ fails → DNS problem

5. Check default gateway

ip route

You should see:

default via 192.168.x.1

If missing:

sudo ip route add default via 192.168.1.1

6. Fix DNS (very common on Ubuntu)

Check:

cat /etc/resolv.conf

If it’s broken or empty, try:

sudo nano /etc/systemd/resolved.conf

Set:

DNS=8.8.8.8 1.1.1.1

Then restart:

sudo systemctl restart systemd-resolved

7. Wi-Fi specific (if applicable)

Check:

nmcli device wifi list

Reconnect:

nmcli device wifi connect "SSID" password "PASSWORD"

If Wi-Fi adapter missing:

lspci | grep -i network

8. Firewall check

Ubuntu often uses UFW:

sudo ufw status

Temporarily disable:

sudo ufw disable

9. Look at logs (very useful)

journalctl -u NetworkManager --no-pager | tail -50

Fast “reset everything” fix (works surprisingly often)

sudo systemctl restart NetworkManager
sudo systemctl restart systemd-resolved
sudo dhclient

Docker Swarm: Simplified Container Management

techhadoop docker, linux

Docker Swarm is Docker’s built-in container orchestration platform. It allows you to manage multiple Docker hosts as a single cluster and deploy containers across them with high availability and load balancing.

How Docker Swarm works

Components
Manager Nodes
  • Maintain cluster state
  • Schedule containers
  • Handle Swarm commands
  • Manage services and networking
Worker Nodes
  • Run application containers
  • Receive tasks from managers
  • Execute workloads
Services

A service defines:

  • Container image
  • Number of replicas
  • Network settings
  • Resource limits

Example:

docker service create \
  --name nginx \
  --replicas 3 \
  nginx:latest

Swarm automatically runs 3 nginx containers across available nodes.

Example Architecture

                    Manager Node
                  +-------------+
                  | Docker API  |
                  | Scheduler   |
                  +-------------+
                         |
      ------------------------------------
      |                |                |
 Worker 1         Worker 2         Worker 3
 nginx-1          nginx-2          nginx-3

Users connect through a Swarm VIP and traffic is automatically load-balanced.

Create a Swarm Cluster

Initialize Manager
docker swarm init

Output:

docker swarm join --token SWMTKN-xxx \
192.168.1.10:2377
Join Worker

Run the generated command on worker nodes:

docker swarm join \
--token SWMTKN-xxx \
192.168.1.10:2377
Verify
docker node ls

Example:

ID        HOSTNAME     STATUS
abc123    manager1     Ready
def456    worker1      Ready
ghi789    worker2      Ready

Deploy a Service

docker service create \
--name web \
--replicas 3 \
-p 80:80 \
nginx

Check:

docker service ls

Tasks:

docker service ps web

Rolling Updates

Update image without downtime:

docker service update \
--image nginx:1.29 \
web

Rollback:

docker service rollback web

Overlay Networking

Create a cluster-wide network:

docker network create \
--driver overlay \
app-net

Containers on different nodes can communicate using service names.

Example:

frontend ---> backend

without knowing IP addresses.

Scaling

Scale from 3 to 10 replicas:

docker service scale web=10

Swarm automatically distributes containers.

High Availability

Recommended:

3 Manager Nodes
2+ Worker Nodes

Example:

Manager1
Manager2
Manager3
Worker1
Worker2
Worker3
Worker4

Managers use a Raft consensus database.

Swarm vs Kubernetes

FeatureDocker SwarmKubernetes
InstallationEasyComplex
Learning CurveLowHigh
ScalingGoodExcellent
EcosystemSmallHuge
Enterprise FeaturesBasicAdvanced
Auto-healingYesYes
Rolling UpdatesYesYes
Service DiscoveryYesYes
Market AdoptionLowVery High

When to Use Swarm

Good for:

  • Small environments
  • Home labs
  • Development clusters
  • 5–20 servers
  • Simple HA applications

Example:

3 Docker hosts
Prometheus
Grafana
Loki
cAdvisor
Node Exporter

Swarm is very easy to deploy for this use case.

When to Use Kubernetes/OpenShift

Given your background with OpenShift and Kubernetes, use Kubernetes when you need:

  • Hundreds of containers
  • Enterprise RBAC
  • GitOps (ArgoCD/Flux)
  • Service Mesh
  • Advanced networking
  • Multi-cluster management
  • Production-grade platform engineering

For a small monitoring stack (Prometheus + Grafana + cAdvisor + Node Exporter across 10–20 Linux servers), Docker Swarm is often simpler and faster to operate than a full Kubernetes cluster.