Alertmanager is the specialized component of the Prometheus ecosystem that handles alerts.
While Prometheus is responsible for recording metrics and triggering an alert when something goes wrong (like a server running out of RAM), Prometheus itself has no idea how to send an email, slack message, or page an engineer. It hands that problem off to Alertmanager.
The Workflow: How an Alert Moves
[ Your Servers ] ──(Metrics)──► [ Prometheus ]
│
▼ (Metrics hit a threshold)
[ Alertmanager ]
│
┌──────────────────────┼──────────────────────┐
▼ ▼ ▼
[ Email ] [ Slack ] [ PagerDuty ]
Prometheus constantly evaluates your data against rules you write (e.g., Is CPU usage > 90% for more than 5 minutes?). If yes, Prometheus fires the alert and shoots it over to Alertmanager.
Alertmanager takes that raw alert and figures out how to handle it intelligently.
The Receiver (Slack, Email, PagerDuty, Discord, etc.) actually delivers the message to your team.
Why do we need a separate tool just for alerts?
You might wonder why Prometheus doesn’t just send emails directly. Alertmanager exists because in a large infrastructure (like your 20+ servers), raw alerts can quickly turn into a nightmare without three critical features:
1. Grouping (Preventing “Alert Fatigue”)
Imagine a network switch dies, cutting off 15 of your Docker hosts at the exact same moment. If Prometheus sent alerts directly, you would get 15 individual emails or Slack pings within 10 seconds, blowing up your phone.
Alertmanager groups them. It sees that 15 servers failed for the same reason, bundles them into one single notification: “Alert: 15 hosts are currently unreachable.”
2. Inhibition (Muting Dependent Alerts)
If a physical server completely crashes, both the Host Down alert and the 10 Containers on this Host are Down alerts will trigger. You don’t need to be told the containers are down if you already know the whole machine is dead. Alertmanager allows you to mute (inhibit) the container alerts if the main host alert is already active.
3. Silencing
If you are doing planned maintenance on server-05 on a Tuesday morning, you can log into Alertmanager’s web interface and click Silence for that specific server for 2 hours. Prometheus will keep tracking the metrics, but Alertmanager will block any notifications from waking up your team while you work on it.
A Quick Look at How It Connects to Your Stack
Since you are running your central monitoring stack via Portainer, you would add an Alertmanager service to your existing Docker Compose file right alongside Prometheus and Grafana:
Note that Grafana can also send alerts based on your dashboards. However, in professional setups using Prometheus, engineers prefer Alertmanager because it can process math across your entire fleet instantly at the database level, rather than waiting for a Grafana graph to load and trigger.
To install the Prometheus Node Exporter directly onto a bare-metal Linux server without using Docker, you want to set it up as a native system service. This ensures it starts automatically when the server reboots and runs quietly in the background.
Here is the step-by-step production setup guide. You can do this manually on one server, or drop these steps into your Ansible workflow.
Step 1: Create a Dedicated System User
For security, Node Exporter should never run as root. Create a system user with no login shells or home directory.
Save and exit the file (in Nano, press Ctrl+O, Enter, then Ctrl+X).
Step 5: Start and Enable Node Exporter
Reload the systemd manager to recognize your new service configuration, start Node Exporter, and enable it so it boots up automatically with the server.
Bash
sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter
Verify it’s running:
Check the status of the service to ensure there are no errors:
Bash
sudo systemctl status node_exporter
You can also verify that it is successfully broadcasting metrics by hitting its port (9100) via curl:
Bash
curl http://localhost:9100/metrics
If you see a wall of text containing metrics starting with node_..., the installation was a complete success.
Step 6: Update Your Central Prometheus Server
Just like you did with your Docker containers, you must now tell your central Prometheus server to scrape this physical machine. Add the bare metal server’s IP address under your node-host-os job configuration in your central prometheus.yml file:
YAML
- job_name: 'node-host-os'
static_configs:
- targets: ['192.168.1.50:9100'] # Your existing Docker hosts
- targets: ['192.168.1.85:9100'] # Your new bare metal server IP
labels:
instance: 'bare-metal-01'
Reload or restart your central Prometheus container, and the bare-metal server will instantly feed into your existing Grafana Node Exporter Full dashboard!
A complete Prometheus + Grafana monitoring solution for 20 Linux servers + 20 Windows servers is absolutely achievable with a single, scalable monitoring stack. Below is a clear architecture, required components, and step‑by‑step setup based on authoritative sources.
Concise Takeaway
You will deploy Prometheus on a central Linux server, install node_exporter on all 20 Linux hosts, install windows_exporter on all 20 Windows hosts, and visualize everything in Grafana. Prometheus scrapes all 40 servers on a schedule and Grafana queries Prometheus for dashboards. Grafana Labsmylinux.work
Recommended Architecture
Prometheus server (Linux) — scrapes metrics from all exporters.
Node Exporter on each Linux server — exposes CPU, RAM, disk, network metrics.
Windows Exporter on each Windows server — exposes Windows OS metrics on port 9182. devopsofworld.com
Grafana server — dashboards + alert visualization.
Alertmanager (optional but recommended) — email/Slack/Teams alerts. mylinux.work
1. Install Exporters on All Servers
Linux (20 servers)
Install node_exporter on each Linux host:
Prometheus documentation recommends Node Exporter for Linux metrics. Grafana Labs
Metrics exposed on port 9100.
Windows (20 servers)
Install windows_exporter:
Download windows_exporter and run it as a service (NSSM recommended).
Here is a clear, production‑grade monitoring architecture diagram for your Prometheus + Grafana setup covering 20 Linux + 20 Windows servers. I’m giving you both a visual ASCII diagram and a conceptual architecture so you can use it in documentation or design reviews.
To install the Prometheus Node Exporter directly onto a bare-metal Linux server without using Docker, you want to set it up as a native system service. This ensures it starts automatically when the server reboots and runs quietly in the background.
Here is the step-by-step production setup guide. You can do this manually on one server, or drop these steps into your Ansible workflow.
Step 1: Create a Dedicated System User
For security, Node Exporter should never run as root. Create a system user with no login shells or home directory.
Save and exit the file (in Nano, press Ctrl+O, Enter, then Ctrl+X).
Step 5: Start and Enable Node Exporter
Reload the systemd manager to recognize your new service configuration, start Node Exporter, and enable it so it boots up automatically with the server.
Bash
sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter
Verify it’s running:
Check the status of the service to ensure there are no errors:
Bash
sudo systemctl status node_exporter
You can also verify that it is successfully broadcasting metrics by hitting its port (9100) via curl:
Bash
curl http://localhost:9100/metrics
If you see a wall of text containing metrics starting with node_..., the installation was a complete success.
Step 6: Update Your Central Prometheus Server
Just like you did with your Docker containers, you must now tell your central Prometheus server to scrape this physical machine. Add the bare metal server’s IP address under your node-host-os job configuration in your central prometheus.yml file:
YAML
- job_name: 'node-host-os'
static_configs:
- targets: ['192.168.1.50:9100'] # Your existing Docker hosts
- targets: ['192.168.1.85:9100'] # Your new bare metal server IP
labels:
instance: 'bare-metal-01'
Reload or restart your central Prometheus container, and the bare-metal server will instantly feed into your existing Grafana Node Exporter Full dashboard!
Both commands are designed to spin up the Prometheus Node Exporter to monitor your host machine’s physical hardware. While they look nearly identical, the first command is production-ready, while the second command has a hidden trap that can eventually crash your server.
Here is the exact breakdown of the two major differences between them.
Difference 1: Log Rotation (The Crucial Difference)
The primary difference lies in these three lines present only in the first command:
Bash
--log-driver json-file \
--log-opt max-size=10m \
--log-opt max-file=45 \
The First Command (Safe & Controlled)
This explicitly configures Docker’s logging mechanisms. It tells Docker:
Treat container logs as standard JSON files.
max-size=10m: Once Node Exporter’s log file reaches 10 Megabytes, roll it over and start a new one.
max-file=45: Keep a maximum of 45 old log files.
This sets a strict upper bound on how much space Node Exporter’s logs can take up on your host: $45 \times 10\text{ MB} = 450\text{ MB}$ maximum.
The Second Command (The Hidden Danger)
Because it omits these lines, it uses your system’s default Docker logging behavior. By default, Docker does not limit log file sizes.
If Node Exporter encounters a continuous warning or error (for example, if a specific disk partition becomes unreadable), it will spam logs rapidly. Without a cap, that single log file will grow to 10 GB, 50 GB, or more, until it completely consumes your host’s hard drive and locks up the server.
Difference 2: Readability and Syntax
The second difference comes down to formatting and execution safety.
First Command: Uses the backslash (\) at the end of every line. This tells the Linux terminal, “This command isn’t finished yet, continue reading on the next line.” It makes the script highly readable and easy to modify.
Second Command: Missing the backslashes. If you attempt to copy and paste the second block exactly as it is typed into a standard Linux terminal, it will break. The terminal will try to execute docker run -d by itself on line 1, fail because it has no image name, and then treat the remaining lines as separate, unrecognized commands.
Summary Matrix
Feature
Command 1 (With Log Options)
Command 2 (Without Log Options)
Maximum Disk Used by Logs
Capped at 450 MB
Infinite (Will grow until disk is full)
Log Files Auto-Cleaned?
Yes, old logs are purged automatically.
No, requires manual intervention or a global Docker daemon config.
Terminal Copy-Paste Ready
Yes (Clean line breaks via \).
No (Will fail unless flattened into a single line).
Which one should you use?
Always use the first command. When you are managing + 20 different production servers, you cannot afford to have a single chatty container quietly fill up a local hard drive. The first command ensures predictable storage safety.
Explain docker stats output
docker run command is 100% correct for a production Node Exporter setup (--net="host", --pid="host", etc. are all perfectly placed).
The reason it showed up with a 3.824GiB limit in docker stats is a harmless quirk of how DDEV or Docker calculates default fallbacks when using host networking.
Here is exactly why it happened and why you don’t need to worry about it:
The “Host Network” Illusion
When you pass --net="host" to a container, you are telling it: “Do not create an isolated network sandbox. Use the host’s actual network stack directly.”
Because of this, when you run docker stats, the Docker daemon gets a bit confused about the boundaries of that container. In certain environments (especially management setups like DDEV or Dokploy where default resource profiles or parent cgroups are enforced across the docker daemon), Docker will simply mirror the memory limit of the host or the surrounding project cluster onto any container running in host network mode.
How to Verify It’s actually fine
You can prove that your Node Exporter isn’t actually being restricted or grouped into a restricted application pool by running this command on your host:
You will see "Memory": 0. A value of 0 in Docker means unlimited—it has full access to the bare-metal server’s resource pool.
Verdict
Your command is great. The 3.824GiB limit you saw in docker stats is just a visual reporting artifact because of --net="host". You can safely leave it exactly as it is!
The Agents (Remote Hosts): Every one of your 20 servers runs just a single, lightweight cAdvisor container. They expose their metrics on port 8080.
The Core (Central Host): One designated management server runs Prometheus and Grafana. Prometheus is configured to reach out over your network to all 20 servers to grab their metrics.
Step 1: Deploy cAdvisor on all 20 Remote Hosts
You need to spin up cAdvisor on every Docker server. If you use an automation tool like Ansible, this is a one-click playbook. Otherwise, run this docker run command on each machine:
Bash
docker run \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:ro \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--volume=/dev/disk/:/dev/disk:ro \
--publish=8080:8080 \
--detach=true \
--name=cadvisor \
--privileged \
--device=/dev/kmsg \
--restart=always \
gcr.io/cadvisor/cadvisor:latest
Security Note: By publishing 8080:8080, cAdvisor metrics are public to anyone who can reach that IP. Ensure your internal firewall/security groups only allow traffic to port 8080 from your Central Prometheus Server IP.
Step 2: Configure the Central Prometheus Server
On your central monitoring server, create a prometheus.yml file. Instead of pointing to localhost, you will list your 20 servers using Prometheus targets and labels. Labels are crucial here because they allow you to filter your Grafana dashboard by specific servers.
YAML
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'docker-swarm-nodes'
static_configs:
- targets: ['192.168.1.50:8080']
labels:
instance: 'prod-web-01'
env: 'production'
- targets: ['192.168.1.51:8080']
labels:
instance: 'prod-db-01'
env: 'production'
- targets: ['192.168.1.70:8080']
labels:
instance: 'stage-app-01'
env: 'staging'
# ... add the remaining 17 servers here
Start Prometheus and Grafana on this central machine using a simplified version of the Docker Compose template from earlier (removing the local cAdvisor block from it).
Step 3: Multi-Host Grafana Dashboard
Once Prometheus is scraping all 20 hosts, you need a Grafana dashboard that lets you switch between them smoothly.
Go to Grafana -> Dashboards -> Import.
Use Dashboard ID: 14282 or 10619 (both are heavily optimized for multi-host setups).
These dashboards automatically generate a dropdown menu at the top left labeled “Instance” or “Host” based on the labels you defined in your prometheus.yml.
You can now view your entire fleet’s aggregated resource consumption, or drill down into a specific container running on prod-db-01.
Pro-Tips for Managing 20+ Hosts
Node Exporter: cAdvisor only monitors container metrics. If you want to monitor the host Linux OS itself (host disk space, total RAM usage, bare-metal CPU temperature), deploy Prometheus Node Exporter alongside cAdvisor on all 20 nodes, exposing it on port 9100.
Scale Warning: Scraping 20 hosts every 15 seconds will generate a decent chunk of data. Ensure your Central Prometheus server has a reasonable storage retention policy set (e.g., --storage.tsdb.retention.time=15d to keep data for 15 days) so it doesn’t quietly fill up the server’s hard drive.
Combining Grafana and cAdvisor (Container Advisor) is the standard open-source recipe for monitoring Docker and Kubernetes container metrics (like CPU, memory, network, and disk usage).
Because cAdvisor only keeps a tiny buffer of real-time data in memory, you need a time-series database (almost always Prometheus or Grafana Alloy) to scrape that data and hand it off to Grafana for visualization.
Here is a breakdown of how the architecture works, how to set it up, and how to get a dashboard running.
The Monitoring Pipeline
cAdvisor: Sits on the host machine, hooks into the Linux kernel cgroups, and collects resource usage from all running containers. It exposes these raw numbers at a /metrics endpoint.
Prometheus: Periodically “scrapes” (pulls) the data from cAdvisor’s /metrics endpoint and stores it as historical time-series data.
Grafana: Queries Prometheus using PromQL and plots the data onto clean, interactive dashboards.
Quick Setup: Docker Compose Example
The easiest way to spin up cAdvisor, Prometheus, and Grafana all at once to monitor your local Docker containers is by using a docker-compose.yml file.
YAML
version: '3.8'
services:
cadvisor:
image: gcr.io/cadvisor/cadvisor:latest
container_name: cadvisor
ports:
- "8080:8080"
volumes:
- /:/rootfs:ro
- /var/run:/var/run:ro
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
- /dev/disk/:/dev/disk:ro
devices:
- /dev/kmsg
restart: unless-stopped
prometheus:
image: prom/prometheus:latest
container_name: prometheus
ports:
- "9090:9090"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
restart: unless-stopped
grafana:
image: grafana/grafana:latest
container_name: grafana
ports:
- "3000:3000"
restart: unless-stopped
The Prometheus Config (prometheus.yml)
To tell Prometheus to scrape your cAdvisor container, create a prometheus.yml file in the same directory:
YAML
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'cadvisor'
static_configs:
- targets: ['cadvisor:8080']
Run docker compose up -d, and your basic infrastructure is live!
Visualizing with Grafana Dashboards
Instead of building your container monitoring dashboards from scratch, you can import highly optimized community templates.
Recommended Pre-built Dashboards
Log into your Grafana instance (usually http://localhost:3000 — default credentials are admin / admin).
Add Prometheus as your data source (Connections > Data sources > Add data source).
Go to Dashboards > New > Import.
Paste one of these popular community Dashboard IDs:
19908 (cAdvisor Docker Insights – clean, official-feeling modern dashboard)
14282 (Cadvisor Exporter – great for focused container-by-container metrics)
19792 (Advanced cAdvisor dashboard with support for Docker Compose projects)
Key cAdvisor Metrics to Know
When creating your own panels or alerts, look out for these fundamental metric names:
CPU Usage:container_cpu_usage_seconds_total (usually paired with rate() to calculate CPU percentage like sum(rate(container_cpu_usage_seconds_total[5m])) by (name))
Memory Usage:container_memory_usage_bytes (tells you the exact RAM consumption)
Network Traffic:container_network_receive_bytes_total and container_network_transmit_bytes_total
Disk I/O:container_fs_reads_bytes_total and container_fs_writes_bytes_total
Moving into a SysAdmin or DevOps-focused Linux interview means the questions shift from “Do you know this command?” to “How do you automate, scale, troubleshoot, and secure this infrastructure?” At this level, interviewers want to see that you think in terms of reliability, automation, and minimizing downtime.
1. Advanced Storage & File Systems
Q: Can you explain how LVM (Logical Volume Manager) works and why we use it?
In an enterprise environment, partitioning a physical disk directly is risky because resizing it later is difficult. LVM adds an abstraction layer between the physical storage and the operating system.
PV (Physical Volumes): The actual raw hard drives or RAID arrays (e.g., /dev/sdb).
VG (Volume Groups): A pool created by combining multiple PVs. Think of it as a giant virtual hard drive.
LV (Logical Volumes): The actual partitions carved out of the VG, which you format with a filesystem (like ext4 or XFS) and mount.
Why use it? It allows you to dynamically resize disks on a live production server without unmounting them or causing downtime.
Q: A disk is 100% full. You delete a massive 50GB log file, but df -h still shows the disk is 100% full. What is happening, and how do you fix it?
The Cause: A running process (like Nginx or an app daemon) still has an open file descriptor pointing to that deleted file. In Linux, space isn’t reclaimed until both the directory entry is gone and all process references to the file are closed.
How to find it: Run lsof +L1 (list open files with a link count less than 1). This will show you the process ID (PID) holding onto the deleted file.
How to fix without a reboot: Instead of killing the process abruptly, zero out the file descriptor dynamically:echo > /proc/<PID>/fd/<FD_NUMBER> # Or safely gracefully restart/reload the service: systemctl reload nginx
2. Networking, Performance & Kernel Tuning
Q: What is the “OOM Killer” (Out of Memory Killer), and how can you tune or prevent it from killing a critical process like a database?
When the Linux kernel completely runs out of physical memory and swap, it invokes the OOM Killer to sacrifice processes to save the OS from crashing. It assigns an oom_score to processes based on how much memory they use relative to how long they’ve been running.
To protect a critical process (like MySQL or a core container):
Immediate adjustment: Write a negative value to the process’s score adjustment file: echo -1000 > /proc/<PID>/oom_score_adj
DevOps Best Practice: Instead of just adjusting scores, implement proper cgroup/container memory limits, optimize application heap sizes, or configure a healthy amount of swap space as a buffer.
Q: How do you track down network latency or dropped packets on a specific Linux server?
sar -n DEV 1 5: To look at network interface statistics in real-time to check for bandwidth saturation or packet drops at the NIC level.
ss -s: Gives a summary of socket statistics. If you see thousands of connections in TIME_WAIT, the server might be running out of ephemeral ports.
tcpdump -i eth0 port 80 -w capture.pcap: To capture raw traffic for deep analysis in Wireshark if packet corruption or asymmetric routing is suspected.
mtr backend-service.internal: (My Traceroute) Combine ping and traceroute to see exactly which network hop is introducing latency or packet loss.
3. Automation, Infrastructure & CI/CD Linux Concepts
Q: How do Linux Namespaces and Cgroups form the foundation of Docker containers?
A Docker container isn’t a virtual machine; it’s just a standard Linux process wrapped in two kernel features:
Namespaces (Isolate what you can SEE): It isolates the process environment. pid namespace hides other processes; net namespace provides a private routing table; mnt namespace isolates filesystem mount points.
Cgroups / Control Groups (Limit what you can USE): It enforces resource constraints. It dictates exactly how much CPU, memory, network bandwidth, and disk I/O a process group is allowed to consume.
Q: You need to deploy a configuration change or a hotfix to 500 Linux servers simultaneously. How do you approach this?
The Wrong Answer: Writing a manual bash script wrapped in a for loop using SSH. It doesn’t scale, has no error handling, and isn’t idempotent.
The DevOps Answer: Use a Configuration Management tool like Ansible, SaltStack, or Puppet.
You write an Ansible Playbook defining the desired state of the target file.
Execute it using forks (ansible-playbook -f 50 playbook.yml) to apply the change concurrently across infrastructure.
Ensure logging and monitoring (like Datadog or Prometheus) are watched during rollout to trigger an automated rollback if error rates spike.
4. Scenario-Based Design & Architecture
Q: “Our web application goes down every day at 3:00 AM for exactly 5 minutes. No deployments are happening. How do you investigate?”
This tests your infrastructure intuition. A systematic approach looks like this:
Check Cron Jobs & Systemd Timers: Look inside /etc/crontab, /etc/cron.d/, and systemctl list-timers. A heavy backup script, database optimization job, or log rotation (logrotate) is likely running at 3 AM.
Correlate System Logs: Check /var/log/syslog or journalctl --since "02:55" --until "03:10". Look for out-of-memory errors, service restarts, or high CPU spikes during that window.
Check External Dependencies: If the local logs are clean, check if a network switch backup or an upstream cloud infrastructure snapshot is choking I/O operations at that time.
