linux

Install Clair on Linux Using Docker: A Step-by-Step Guide

techhadoop docker, linux, security

The easiest and most reliable way to install Clair (the open-source static container vulnerability scanner) on a Linux host is by using Docker.

Because Clair is a stateless API service, it relies on a PostgreSQL database to store its vulnerability definitions and indexing data.

Follow this step-by-step guide to set up a PostgreSQL database and run Clair on your Linux system.

Prerequisites

Ensure your Linux system has Docker and curl installed:

Bash

sudo apt update && sudo apt install -y docker.io curl   # Debian/Ubuntu
# OR
sudo dnf install -y docker curl                       # RHEL/Rocky Linux/Fedora

Step 1: Start the PostgreSQL Database

Clair requires PostgreSQL (version 13 or newer). Start a database container and create the database for Clair:

Bash

docker run -d \
  --name clair-db \
  -p 5432:5432 \
  -e POSTGRES_DB=clair \
  -e POSTGRES_USER=postgres \
  -e POSTGRES_PASSWORD=clair_password_123 \
  postgres:15

Step 2: Create the Clair Configuration

Clair needs a config.yaml file to tell it how to connect to PostgreSQL and run its updaters.

  1. Create a directory for the config:mkdir -p ./clair_config
  2. Generate a basic config.yaml using this content (ensure your database connection string matches the password you used in Step 1):cat <<EOF > ./clair_config/config.yaml http_listen_addr: ":6060" introspection_addr: ":6061" log_level: "info" database: type: "pgsql" options: # Replace with your host IP if Docker cannot resolve localhost back to the host source: "host=host.docker.internal port=5432 user=postgres password=clair_password_123 dbname=clair sslmode=disable" migrations: true updater: interval: "12h" EOF

Step 3: Run the Clair Container

Deploy the official Clair v4 image from Red Hat’s Quay registry:

Bash

docker run -d \
  --name clair \
  -p 6060:6060 \
  -p 6061:6061 \
  --add-host=host.docker.internal:host-gateway \
  -v $(pwd)/clair_config:/config \
  -e CLAIR_CONF=/config/config.yaml \
  -e CLAIR_MODE=combo \
  quay.io/projectquay/clair:latest
  • CLAIR_MODE=combo: Instructs Clair to run the indexer, matcher, and updater all within a single process (standard for single-host deployments).
  • --add-host...: Allows the Clair container to reach the Postgres container running on the host’s localhost port.

Step 4: Verify the Installation

Verify that Clair is running and healthy:

Bash

curl http://localhost:6061/health

You should receive an HTTP/1.1 200 OK or a JSON status indicating that the application is healthy.

Note on Initial Setup: On its first boot, Clair’s updater will immediately begin downloading massive vulnerability feeds (CVEs, Red Hat, Ubuntu, Alpine lists). This synchronization can take anywhere from 15 minutes to an hour depending on your network speed.

Step 5: Interacting with Clair (clairctl)

To submit local or remote container images to Clair for scanning, install clairctl (the official command-line interface):

Bash

# Download the latest Linux amd64 binary
wget https://github.com/quay/clair/releases/latest/download/clairctl-linux-amd64
# Make it executable and move it to your PATH
chmod +x clairctl-linux-amd64
sudo mv clairctl-linux-amd64 /usr/local/bin/clairctl

To scan an image (such as ubuntu:focal), run:

Bash

clairctl --config ./clair_config/config.yaml report ubuntu:focal

Analyzing RAM Usage in Linux with Grafana

techhadoop Grafana, linux, monitoring

The Grafana Node Exporter Full dashboard memory breakdown above showing how Linux RAM is being used over time.

Let’s interpret it as a Linux administrator or SRE would.

Total Memory Usage Breakdown
CategoryMean
Applications717 MiB
Cache5.46 GiB
Buffers614 MiB
Slab380 MiB
Unused623 MiB
Everything elseSmall

The first thing to notice:

Most of the memory is being used for Cache and Buffers, not applications.

This is normal Linux behavior.

Apps – Memory used by user-space applications

Min: 632 MiB
Mean: 717 MiB
Max: 1.37 GiB

This is the memory actually consumed by running processes:

Examples:

  • Java
  • Nginx
  • Apache
  • PostgreSQL
  • Docker containers
  • Tomcat
  • Kong Gateway

Your server’s applications are only using about:

~717 MB average

which is very low.

Cache

Mean: 5.46 GiB

Linux uses free RAM to cache file contents.

Examples:

  • Recently read files
  • Database files
  • Log files
  • Container images

Linux philosophy:

Free RAM is wasted RAM.

So Linux fills unused memory with cache.

The good news:

5.46 GiB Cache

can be reclaimed instantly if applications need RAM.

This is NOT memory pressure.

Buffers

Mean: 614 MiB

Buffers are cache for block devices.

Examples:

  • Disk metadata
  • Filesystem operations
  • Read/write operations

This is normal.

Slab

Mean: 380 MiB

Kernel memory used for internal structures:

Examples:

  • inode cache
  • dentry cache
  • network structures
  • process descriptors

For example:

slabtop

might show:

dentry
inode_cache
kmalloc

380 MB is reasonable.

PageTables

Mean: 9.09 MiB

Used to translate:

Virtual Address
     ↓
Physical RAM Address

Every process needs page tables.

9 MB is tiny.

No concern.

Swap

Mean: 2.48 MiB

Almost no swap usage.

This is excellent.

Typically:

0 MB - 100 MB

is negligible.

If you saw:

2 GB
5 GB
10 GB

then we’d investigate memory pressure.

SwapCache

Mean: 131 KiB

Tracks swapped pages that may still exist in RAM.

Tiny amount.

Ignore.

Unused

Mean: 623 MiB
Max: 891 MiB

Actually free RAM.

Linux tries to keep this low because it prefers using RAM for caching.

Having only 623 MB free is not necessarily bad because:

Cache = 5.46 GB

can be reclaimed instantly.

Hardware Corrupted

0 B

No RAM errors detected by the kernel.

Good.

What would I conclude?

If I were reviewing this server:

Memory Health

No memory pressure

No excessive swap usage

Applications use very little RAM

Plenty of reclaimable cache

Kernel memory usage is normal

No hardware memory issues

Interview Answer

If an interviewer asks:

“Memory usage is 90%. Should I worry?”

You could answer:

Not necessarily. Linux aggressively uses free RAM for filesystem cache and buffers. I would look at “Available Memory” and swap activity rather than “Used Memory.” In this example, applications consume less than 1 GB while more than 5 GB is file cache, which can be reclaimed when needed. The negligible swap usage indicates there is no memory pressure.

Useful Commands to Verify
free -h

Look especially at:

available

rather than:

used
vmstat 1

Check:

si
so

(Swap In / Swap Out)

top

or

htop

to identify memory-consuming processes.

From these numbers, the server appears healthy and has significant RAM available through reclaimable cache, even though the “used memory” percentage may look high in Grafana.

Install Prometheus Alertmanager on Ubuntu: Step-by-Step Guide

techhadoop linux, monitoring, Prometheus

Here’s a complete step-by-step guide to install and configure Prometheus Alertmanager on Ubuntu.

Step 1: Update System

sudo apt update && sudo apt upgrade -y

Step 2: Create a Dedicated User

sudo useradd --no-create-home --shell /bin/false alertmanager

Step 3: Download Alertmanager

Download the latest release from GitHub:

cd /tmp
wget https://github.com/prometheus/alertmanager/releases/latest/download/alertmanager-0.27.0.linux-amd64.tar.gz
tar -xvf alertmanager-0.27.0.linux-amd64.tar.gz
cd alertmanager-0.27.0.linux-amd64

Check https://github.com/prometheus/alertmanager/releases for the latest version and substitute accordingly.

Step 4: Install Binaries

Move the binaries to /usr/local/bin/:

sudo mv alertmanager /usr/local/bin/
sudo mv amtool /usr/local/bin/

Step 5: Create Directories

Create config and data directories and set ownership:

sudo mkdir /etc/alertmanager
sudo mkdir /var/lib/alertmanager
sudo chown alertmanager:alertmanager /etc/alertmanager
sudo chown alertmanager:alertmanager /var/lib/alertmanager

Step 6: Create the Config File

sudo nano /etc/alertmanager/alertmanager.yml

Paste a basic configuration (example with email):

global:
  smtp_smarthost: 'smtp.gmail.com:587'
  smtp_from: 'alertmanager@example.com'
  smtp_auth_username: 'your@gmail.com'
  smtp_auth_password: 'your-app-password'
route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
  receiver: 'email-alerts'
receivers:
  - name: 'email-alerts'
    email_configs:
      - to: 'you@example.com'
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

Set proper ownership:

sudo chown alertmanager:alertmanager /etc/alertmanager/alertmanager.yml

Step 7: Create a systemd Service

Create the service file:

sudo nano /etc/systemd/system/alertmanager.service
[Unit]
Description=Prometheus Alertmanager
Wants=network-online.target
After=network-online.target
[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --storage.path=/var/lib/alertmanager
[Install]
WantedBy=multi-user.target

Step 8: Start and Enable the Service

sudo systemctl daemon-reload
sudo systemctl start alertmanager
sudo systemctl enable alertmanager
sudo systemctl status alertmanager

Alertmanager will be available at http://<your-server-ip>:9093.

Step 9: Integrate with Prometheus

Add this to your prometheus.yml to wire Alertmanager in:

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['localhost:9093']

Optionally, also scrape Alertmanager’s own metrics:

scrape_configs:
  - job_name: alertmanager
    static_configs:
      - targets: ['localhost:9093']

Then restart Prometheus:

sudo systemctl restart prometheus

Step 10: (Optional) Firewall Rules

If Alertmanager should only be accessible locally:

sudo ufw allow from 127.0.0.1 to any port 9093
sudo ufw deny 9093

Verify & Troubleshoot

# Check logs
sudo journalctl -u alertmanager -f
# Validate config file
amtool check-config /etc/alertmanager/alertmanager.yml
Common Receivers Beyond Email
ReceiverKey Config Field
Slackslack_configs with api_url
PagerDutypagerduty_configs with service_key
Webhookwebhook_configs with url
OpsGenieopsgenie_configs with api_key

Here’s the complete email configuration for Alertmanager:

Update /etc/alertmanager/alertmanager.yml

sudo nano /etc/alertmanager/alertmanager.yml
global:
  smtp_smarthost: 'smtp.gmail.com:587'
  smtp_from: 'alertmanager@yourdomain.com'
  smtp_auth_username: 'your@gmail.com'
  smtp_auth_password: 'your-app-password'   # Gmail App Password, NOT your login password
  smtp_require_tls: true
route:
  receiver: 'email-alerts'
  group_by: ['alertname', 'severity']
  group_wait: 30s        # Wait before sending first alert in a group
  group_interval: 5m     # Wait before sending new alerts for an existing group
  repeat_interval: 4h    # Resend if alert is still firing after this duration
  # Optional: route critical alerts separately
  routes:
    - match:
        severity: critical
      receiver: 'email-critical'
      repeat_interval: 1h
receivers:
  - name: 'email-alerts'
    email_configs:
      - to: 'your-team@example.com'
        send_resolved: true    # Also notify when alert resolves
  - name: 'email-critical'
    email_configs:
      - to: 'oncall@example.com'
        send_resolved: true
        headers:
          Subject: '[CRITICAL] {{ .GroupLabels.alertname }}'
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'instance']

Gmail Setup (Important)

Gmail blocks plain passwords — you need an App Password:

  1. Go to https://myaccount.google.com/security
  2. Enable 2-Step Verification (required)
  3. Go to App Passwords → select “Mail” → generate
  4. Use that 16-character password in smtp_auth_password

For other providers, swap out the SMTP settings:

Providersmtp_smarthost
Gmailsmtp.gmail.com:587
Outlook/Office365smtp.office365.com:587
SendGridsmtp.sendgrid.net:587
AWS SESemail-smtp.<region>.amazonaws.com:587

Apply & Test

# Validate config before restarting
amtool check-config /etc/alertmanager/alertmanager.yml
# Restart to apply
sudo systemctl restart alertmanager
# Send a test alert manually
amtool alert add alertname="TestAlert" severity="warning" \
  --alertmanager.url=http://localhost:9093
# Check it appears
amtool alert --alertmanager.url=http://localhost:9093

Check Logs if Email Fails

sudo journalctl -u alertmanager -f

Common issues to look for:

  • authentication failed → wrong App Password
  • connection refused → check SMTP host/port
  • tls: no supported versions → set smtp_require_tls: false for port 25

Once you confirm email is working, you can layer in Slack or PagerDuty alongside it in the same config.

Understanding Alertmanager in the Prometheus Ecosystem

techhadoop Grafana, linux, monitoring, Prometheus

Alertmanager is the specialized component of the Prometheus ecosystem that handles alerts.

While Prometheus is responsible for recording metrics and triggering an alert when something goes wrong (like a server running out of RAM), Prometheus itself has no idea how to send an email, slack message, or page an engineer. It hands that problem off to Alertmanager.

The Workflow: How an Alert Moves

[ Your Servers ] ──(Metrics)──► [ Prometheus ]
                                       │
                                       ▼ (Metrics hit a threshold)
                                [ Alertmanager ]
                                       │
                ┌──────────────────────┼──────────────────────┐
                ▼                      ▼                      ▼
           [ Email ]               [ Slack ]             [ PagerDuty ]
  1. Prometheus constantly evaluates your data against rules you write (e.g., Is CPU usage > 90% for more than 5 minutes?). If yes, Prometheus fires the alert and shoots it over to Alertmanager.
  2. Alertmanager takes that raw alert and figures out how to handle it intelligently.
  3. The Receiver (Slack, Email, PagerDuty, Discord, etc.) actually delivers the message to your team.

Why do we need a separate tool just for alerts?

You might wonder why Prometheus doesn’t just send emails directly. Alertmanager exists because in a large infrastructure (like your 20+ servers), raw alerts can quickly turn into a nightmare without three critical features:

1. Grouping (Preventing “Alert Fatigue”)

Imagine a network switch dies, cutting off 15 of your Docker hosts at the exact same moment. If Prometheus sent alerts directly, you would get 15 individual emails or Slack pings within 10 seconds, blowing up your phone.

Alertmanager groups them. It sees that 15 servers failed for the same reason, bundles them into one single notification: “Alert: 15 hosts are currently unreachable.”

2. Inhibition (Muting Dependent Alerts)

If a physical server completely crashes, both the Host Down alert and the 10 Containers on this Host are Down alerts will trigger. You don’t need to be told the containers are down if you already know the whole machine is dead. Alertmanager allows you to mute (inhibit) the container alerts if the main host alert is already active.

3. Silencing

If you are doing planned maintenance on server-05 on a Tuesday morning, you can log into Alertmanager’s web interface and click Silence for that specific server for 2 hours. Prometheus will keep tracking the metrics, but Alertmanager will block any notifications from waking up your team while you work on it.

A Quick Look at How It Connects to Your Stack

Since you are running your central monitoring stack via Portainer, you would add an Alertmanager service to your existing Docker Compose file right alongside Prometheus and Grafana:

YAML

  alertmanager:
    image: prom/alertmanager:latest
    container_name: central-alertmanager
    ports:
      - "9093:9093"
    volumes:
      - /opt/alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml
    restart: unless-stopped

In your alertmanager.yml file, you define your routing. For example, telling it to route critical alerts to Slack and warning alerts to Email:

YAML

route:
  receiver: 'slack-notifications'
receivers:
- name: 'slack-notifications'
  slack_configs:
  - api_url: 'https://hooks.slack.com/services/T000/B000/XXXXXX'
    channel: '#ops-alerts'
Alertmanager vs. Grafana Alerting

Note that Grafana can also send alerts based on your dashboards. However, in professional setups using Prometheus, engineers prefer Alertmanager because it can process math across your entire fleet instantly at the database level, rather than waiting for a Grafana graph to load and trigger.

Install Prometheus Node Exporter on Bare-metal Linux

techhadoop linux, monitoring, Prometheus

To install the Prometheus Node Exporter directly onto a bare-metal Linux server without using Docker, you want to set it up as a native system service. This ensures it starts automatically when the server reboots and runs quietly in the background.

Here is the step-by-step production setup guide. You can do this manually on one server, or drop these steps into your Ansible workflow.

Step 1: Create a Dedicated System User

For security, Node Exporter should never run as root. Create a system user with no login shells or home directory.

Bash

sudo useradd --no-create-home --shell /bin/false node_exporter

Step 2: Download and Extract the Binary

Navigate to the /tmp folder, download the latest stable release of Node Exporter, and extract it.

Bash

cd /tmp
wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
tar -xvf node_exporter-1.8.2.linux-amd64.tar.gz
install last version

(Note: You can check the Prometheus Downloads page to see if a newer version is available and change the version number accordingly.)

This is the last version:

wget https://github.com/prometheus/node_exporter/releases/download/v1.11.1/node_exporter-1.11.1.linux-amd64.tar.gz

Step 3: Move the Binary into System Path

Move the executable binary to /usr/local/bin so it can be executed system-wide, and assign ownership to the user you created in Step 1.

Bash

sudo mv node_exporter-1.8.2.linux-amd64/node_exporter /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

Clean up the remaining installation files you no longer need:

Bash

rm -rf node_exporter-1.8.2.linux-amd64*

Step 4: Create a Systemd Service File

To manage Node Exporter like any other Linux service (e.g., systemctl start), create a systemd service definition file.

Open a new file using your preferred text editor:

Bash

sudo nano /etc/systemd/system/node_exporter.service

Paste the following configuration into the file:

Ini, TOML

[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target

Save and exit the file (in Nano, press Ctrl+O, Enter, then Ctrl+X).

Step 5: Start and Enable Node Exporter

Reload the systemd manager to recognize your new service configuration, start Node Exporter, and enable it so it boots up automatically with the server.

Bash

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter

Verify it’s running:

Check the status of the service to ensure there are no errors:

Bash

sudo systemctl status node_exporter

You can also verify that it is successfully broadcasting metrics by hitting its port (9100) via curl:

Bash

curl http://localhost:9100/metrics

If you see a wall of text containing metrics starting with node_..., the installation was a complete success.

Step 6: Update Your Central Prometheus Server

Just like you did with your Docker containers, you must now tell your central Prometheus server to scrape this physical machine. Add the bare metal server’s IP address under your node-host-os job configuration in your central prometheus.yml file:

YAML

  - job_name: 'node-host-os'
    static_configs:
      - targets: ['192.168.1.50:9100'] # Your existing Docker hosts
      - targets: ['192.168.1.85:9100'] # Your new bare metal server IP
        labels: 
          instance: 'bare-metal-01'

Reload or restart your central Prometheus container, and the bare-metal server will instantly feed into your existing Grafana Node Exporter Full dashboard!

Effective Prometheus and Grafana Setup for Linux and Windows Servers

techhadoop Grafana, linux, monitoring, Prometheus

A complete Prometheus + Grafana monitoring solution for 20 Linux servers + 20 Windows servers is absolutely achievable with a single, scalable monitoring stack. Below is a clear architecture, required components, and step‑by‑step setup based on authoritative sources.

Concise Takeaway

You will deploy Prometheus on a central Linux server, install node_exporter on all 20 Linux hosts, install windows_exporter on all 20 Windows hosts, and visualize everything in Grafana. Prometheus scrapes all 40 servers on a schedule and Grafana queries Prometheus for dashboards. Grafana Labs mylinux.work

Recommended Architecture

  • Prometheus server (Linux) — scrapes metrics from all exporters.
  • Node Exporter on each Linux server — exposes CPU, RAM, disk, network metrics.
  • Windows Exporter on each Windows server — exposes Windows OS metrics on port 9182. devopsofworld.com
  • Grafana server — dashboards + alert visualization.
  • Alertmanager (optional but recommended) — email/Slack/Teams alerts. mylinux.work

1. Install Exporters on All Servers

Linux (20 servers)

Install node_exporter on each Linux host:

  • Prometheus documentation recommends Node Exporter for Linux metrics. Grafana Labs
  • Metrics exposed on port 9100.

Windows (20 servers)

Install windows_exporter:

Example Windows service install (from source):

nssm install Windows_exporter C:\path\windows_exporter.exe --collectors.enabled="cpu,logical_disk,net,os,service,system"

2. Prometheus Configuration

Prometheus scrapes all 40 servers.
Example prometheus.yml:

scrape_configs:
  - job_name: 'linux'
    static_configs:
      - targets:
        - 'linux01:9100'
        - 'linux02:9100'
        # ... all 20 Linux servers
  - job_name: 'windows'
    static_configs:
      - targets:
        - 'win01:9182'
        - 'win02:9182'
        # ... all 20 Windows servers

Windows scrape configuration is consistent with documented examples. mylinux.work

3. Grafana Setup

Grafana connects to Prometheus as a data source.

Recommended dashboards:

  • Node Exporter Full (Linux)
  • Windows Exporter Dashboard (Windows) — available from Grafana community. devopsofworld.com

4. Alerting (Optional but Recommended)

Use Alertmanager for notifications:

  • Email
  • Slack / Teams
  • Webhooks

Alertmanager is part of the standard Linux monitoring stack. mylinux.work

5. Hardware Requirements (for 40 servers)

For a single Prometheus + Grafana node:

ComponentRecommended
CPU4 vCPUs
RAM8–16 GB
Storage100–200 GB SSD
OSLinux (Ubuntu recommended)

This is inferred from typical Prometheus deployments for ~50 nodes.

6. Deployment Model Options

Option A — Single Monitoring Server (simplest)

  • Prometheus + Grafana on one VM
  • Good for up to ~100 nodes

Option B — Distributed (Prometheus + Grafana separate)

  • Prometheus on one VM
  • Grafana on another
  • More scalable; recommended by LinuxConfig architecture. LinuxConfig.org

7. What You Will Monitor

  • CPU, RAM, Disk, Network
  • Windows services
  • Windows event metrics
  • Linux system metrics
  • Optional: Docker, cAdvisor, SQL Server, IIS, etc. kx.cloudingenium.com

Next step

  • Prometheus configuration
  • Deployment diagram
  • Step-by-step guide

Here is a clear, production‑grade monitoring architecture diagram for your Prometheus + Grafana setup covering 20 Linux + 20 Windows servers.
I’m giving you both a visual ASCII diagram and a conceptual architecture so you can use it in documentation or design reviews.

Monitoring Architecture Diagram (Prometheus + Grafana)

                           ┌──────────────────────────────┐
                           │        Grafana Server         │
                           │  - Dashboards & Visualization │
                           │  - Connects to Prometheus     │
                           └───────────────▲──────────────┘
                                           │
                                           │  Queries
                                           │
                           ┌───────────────┴──────────────┐
                           │       Prometheus Server      │
                           │  - Scrapes all exporters     │
                           │  - Stores time‑series data   │
                           │  - Pushes alerts to AM       │
                           └───────────────▲──────────────┘
                                           │
                                           │  Scrape (HTTP)
                                           │
        ┌──────────────────────────┬───────┴───────────────┬──────────────────────────┐
        │                          │                       │                          │
        │                          │                       │                          │
┌───────────────┐        ┌────────────────┐       ┌────────────────┐        ┌───────────────┐
│ 20 Linux       │       │ 20 Windows     │       │ Alertmanager   │        │ Optional Tools │
│ Servers        │       │ Servers        │       │ (Email/Slack)  │        │  - cAdvisor    │
│ node_exporter  │       │ windows_exporter│      │ Notifications  │        │  - Blackbox    │
│ Port 9100      │       │ Port 9182       │      │ Routing        │        │  - SNMP Export │
└───────────────┘        └────────────────┘       └────────────────┘        └───────────────┘

How the Architecture Works

  • Prometheus scrapes metrics from:
    • node_exporter on each Linux server
    • windows_exporter on each Windows server
  • Grafana queries Prometheus to build dashboards.
  • Alertmanager receives alerts from Prometheus and sends notifications.
  • Exporters expose metrics over HTTP; Prometheus pulls them on a schedule.

Component Breakdown

  • Prometheus Server
    Central collector + time‑series database.
  • Grafana Server
    Dashboards, visualization, alerting UI.
  • Node Exporter
    Linux OS metrics (CPU, RAM, disk, network).
  • Windows Exporter
    Windows OS metrics (services, CPU, memory, disks).
  • Alertmanager
    Email, Slack, Teams, webhook alerts.

Optional Add‑Ons

  • Blackbox Exporter — HTTP/TCP/ICMP checks
  • cAdvisor — container metrics
  • SNMP Exporter — network devices

Install Prometheus Node Exporter on Bare-Metal Linux

techhadoop linux, monitoring

To install the Prometheus Node Exporter directly onto a bare-metal Linux server without using Docker, you want to set it up as a native system service. This ensures it starts automatically when the server reboots and runs quietly in the background.

Here is the step-by-step production setup guide. You can do this manually on one server, or drop these steps into your Ansible workflow.

Step 1: Create a Dedicated System User

For security, Node Exporter should never run as root. Create a system user with no login shells or home directory.

Bash

sudo useradd --no-create-home --shell /bin/false node_exporter

Step 2: Download and Extract the Binary

Navigate to the /tmp folder, download the latest stable release of Node Exporter, and extract it.

Bash

cd /tmp
wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
tar -xvf node_exporter-1.8.2.linux-amd64.tar.gz

(Note: You can check the Prometheus Downloads page to see if a newer version is available and change the version number accordingly.)

Step 3: Move the Binary into System Path

Move the executable binary to /usr/local/bin so it can be executed system-wide, and assign ownership to the user you created in Step 1.

Bash

sudo mv node_exporter-1.8.2.linux-amd64/node_exporter /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

Clean up the remaining installation files you no longer need:

Bash

rm -rf node_exporter-1.8.2.linux-amd64*

Step 4: Create a Systemd Service File

To manage Node Exporter like any other Linux service (e.g., systemctl start), create a systemd service definition file.

Open a new file using your preferred text editor:

Bash

sudo nano /etc/systemd/system/node_exporter.service

Paste the following configuration into the file:

Ini, TOML

[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target

Save and exit the file (in Nano, press Ctrl+O, Enter, then Ctrl+X).

Step 5: Start and Enable Node Exporter

Reload the systemd manager to recognize your new service configuration, start Node Exporter, and enable it so it boots up automatically with the server.

Bash

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter
Verify it’s running:

Check the status of the service to ensure there are no errors:

Bash

sudo systemctl status node_exporter

You can also verify that it is successfully broadcasting metrics by hitting its port (9100) via curl:

Bash

curl http://localhost:9100/metrics

If you see a wall of text containing metrics starting with node_..., the installation was a complete success.

Step 6: Update Your Central Prometheus Server

Just like you did with your Docker containers, you must now tell your central Prometheus server to scrape this physical machine. Add the bare metal server’s IP address under your node-host-os job configuration in your central prometheus.yml file:

YAML

  - job_name: 'node-host-os'
    static_configs:
      - targets: ['192.168.1.50:9100'] # Your existing Docker hosts
      - targets: ['192.168.1.85:9100'] # Your new bare metal server IP
        labels: 
          instance: 'bare-metal-01'

Reload or restart your central Prometheus container, and the bare-metal server will instantly feed into your existing Grafana Node Exporter Full dashboard!

Building a Grafana Dashboard for Multi-Host Metrics

techhadoop docker, linux, monitoring, Prometheus

Centralized monitoring architecture.

One central Grafana and Prometheus server that pulls metrics from 20 lightweight cAdvisor agents running across your network.

Here is the cleanest way to architect and deploy this.

Centralized Architecture Overview

[ Central Monitor Server ]               [ 20x Remote Docker Hosts ]
┌────────────────────────┐               ┌─────────────────────────┐
│        Grafana         │               │  Docker Host 01         │
│           ▲            │               │  └─ cAdvisor (Port 8080)│
│           │ (Queries)  │               └─────────────────────────┘
│       Prometheus       │◄─────────────┐             ▲
└────────────────────────┘  (Scrapes    │             │
                             over HTTP) ├─────────────┤
                                        │             ▼
                                        │┌─────────────────────────┐
                                        ││  Docker Host 20         │
                                        └┤  └─ cAdvisor (Port 8080)│
                                         └─────────────────────────┘
  • The Agents (Remote Hosts): Every one of your 20 servers runs just a single, lightweight cAdvisor container. They expose their metrics on port 8080.
  • The Core (Central Host): One designated management server runs Prometheus and Grafana. Prometheus is configured to reach out over your network to all 20 servers to grab their metrics.

Step 1: Deploy cAdvisor on all 20 Remote Hosts

You need to spin up cAdvisor on every Docker server. If you use an automation tool like Ansible, this is a one-click playbook. Otherwise, run this docker run command on each machine:

Bash

docker run \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:ro \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --volume=/dev/disk/:/dev/disk:ro \
  --publish=8080:8080 \
  --detach=true \
  --name=cadvisor \
  --privileged \
  --device=/dev/kmsg \
  --restart=always \
  gcr.io/cadvisor/cadvisor:latest

Security Note: By publishing 8080:8080, cAdvisor metrics are public to anyone who can reach that IP. Ensure your internal firewall/security groups only allow traffic to port 8080 from your Central Prometheus Server IP.

Step 2: Configure the Central Prometheus Server

On your central monitoring server, create a prometheus.yml file. Instead of pointing to localhost, you will list your 20 servers using Prometheus targets and labels. Labels are crucial here because they allow you to filter your Grafana dashboard by specific servers.

YAML

global:
  scrape_interval: 15s
scrape_configs:
  - job_name: 'docker-swarm-nodes'
    static_configs:
      - targets: ['192.168.1.50:8080']
        labels:
          instance: 'prod-web-01'
          env: 'production'
      - targets: ['192.168.1.51:8080']
        labels:
          instance: 'prod-db-01'
          env: 'production'
      - targets: ['192.168.1.70:8080']
        labels:
          instance: 'stage-app-01'
          env: 'staging'
          
      # ... add the remaining 17 servers here

Start Prometheus and Grafana on this central machine using a simplified version of the Docker Compose template from earlier (removing the local cAdvisor block from it).

Step 3: Multi-Host Grafana Dashboard

Once Prometheus is scraping all 20 hosts, you need a Grafana dashboard that lets you switch between them smoothly.

  1. Go to Grafana -> Dashboards -> Import.
  2. Use Dashboard ID: 14282 or 10619 (both are heavily optimized for multi-host setups).
  3. These dashboards automatically generate a dropdown menu at the top left labeled “Instance” or “Host” based on the labels you defined in your prometheus.yml.

You can now view your entire fleet’s aggregated resource consumption, or drill down into a specific container running on prod-db-01.

Pro-Tips for Managing 20+ Hosts

  • Node Exporter: cAdvisor only monitors container metrics. If you want to monitor the host Linux OS itself (host disk space, total RAM usage, bare-metal CPU temperature), deploy Prometheus Node Exporter alongside cAdvisor on all 20 nodes, exposing it on port 9100.
  • Scale Warning: Scraping 20 hosts every 15 seconds will generate a decent chunk of data. Ensure your Central Prometheus server has a reasonable storage retention policy set (e.g., --storage.tsdb.retention.time=15d to keep data for 15 days) so it doesn’t quietly fill up the server’s hard drive.

Monitoring Docker with Grafana and cAdvisor

techhadoop docker, linux, monitoring

Combining Grafana and cAdvisor (Container Advisor) is the standard open-source recipe for monitoring Docker and Kubernetes container metrics (like CPU, memory, network, and disk usage).

Because cAdvisor only keeps a tiny buffer of real-time data in memory, you need a time-series database (almost always Prometheus or Grafana Alloy) to scrape that data and hand it off to Grafana for visualization.

Here is a breakdown of how the architecture works, how to set it up, and how to get a dashboard running.

The Monitoring Pipeline

  1. cAdvisor: Sits on the host machine, hooks into the Linux kernel cgroups, and collects resource usage from all running containers. It exposes these raw numbers at a /metrics endpoint.
  2. Prometheus: Periodically “scrapes” (pulls) the data from cAdvisor’s /metrics endpoint and stores it as historical time-series data.
  3. Grafana: Queries Prometheus using PromQL and plots the data onto clean, interactive dashboards.

Quick Setup: Docker Compose Example

The easiest way to spin up cAdvisor, Prometheus, and Grafana all at once to monitor your local Docker containers is by using a docker-compose.yml file.

YAML

version: '3.8'
services:
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    ports:
      - "8080:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    devices:
      - /dev/kmsg
    restart: unless-stopped
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    restart: unless-stopped
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    restart: unless-stopped
The Prometheus Config (prometheus.yml)

To tell Prometheus to scrape your cAdvisor container, create a prometheus.yml file in the same directory:

YAML

global:
  scrape_interval: 15s
scrape_configs:
  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']

Run docker compose up -d, and your basic infrastructure is live!

Visualizing with Grafana Dashboards

Instead of building your container monitoring dashboards from scratch, you can import highly optimized community templates.

Recommended Pre-built Dashboards
  1. Log into your Grafana instance (usually http://localhost:3000 — default credentials are admin / admin).
  2. Add Prometheus as your data source (Connections > Data sources > Add data source).
  3. Go to Dashboards > New > Import.
  4. Paste one of these popular community Dashboard IDs:
    • 19908 (cAdvisor Docker Insights – clean, official-feeling modern dashboard)
    • 14282 (Cadvisor Exporter – great for focused container-by-container metrics)
    • 19792 (Advanced cAdvisor dashboard with support for Docker Compose projects)

Key cAdvisor Metrics to Know

When creating your own panels or alerts, look out for these fundamental metric names:

  • CPU Usage: container_cpu_usage_seconds_total (usually paired with rate() to calculate CPU percentage like sum(rate(container_cpu_usage_seconds_total[5m])) by (name))
  • Memory Usage: container_memory_usage_bytes (tells you the exact RAM consumption)
  • Network Traffic: container_network_receive_bytes_total and container_network_transmit_bytes_total
  • Disk I/O: container_fs_reads_bytes_total and container_fs_writes_bytes_total

Mastering Linux Interviews: Automation & Reliability Questions

techhadoop linux

Moving into a SysAdmin or DevOps-focused Linux interview means the questions shift from “Do you know this command?” to “How do you automate, scale, troubleshoot, and secure this infrastructure?” At this level, interviewers want to see that you think in terms of reliability, automation, and minimizing downtime.

1. Advanced Storage & File Systems

Q: Can you explain how LVM (Logical Volume Manager) works and why we use it?

In an enterprise environment, partitioning a physical disk directly is risky because resizing it later is difficult. LVM adds an abstraction layer between the physical storage and the operating system.

  • PV (Physical Volumes): The actual raw hard drives or RAID arrays (e.g., /dev/sdb).
  • VG (Volume Groups): A pool created by combining multiple PVs. Think of it as a giant virtual hard drive.
  • LV (Logical Volumes): The actual partitions carved out of the VG, which you format with a filesystem (like ext4 or XFS) and mount.
  • Why use it? It allows you to dynamically resize disks on a live production server without unmounting them or causing downtime.
Q: A disk is 100% full. You delete a massive 50GB log file, but df -h still shows the disk is 100% full. What is happening, and how do you fix it?
  • The Cause: A running process (like Nginx or an app daemon) still has an open file descriptor pointing to that deleted file. In Linux, space isn’t reclaimed until both the directory entry is gone and all process references to the file are closed.
  • How to find it: Run lsof +L1 (list open files with a link count less than 1). This will show you the process ID (PID) holding onto the deleted file.
  • How to fix without a reboot: Instead of killing the process abruptly, zero out the file descriptor dynamically:echo > /proc/<PID>/fd/<FD_NUMBER> # Or safely gracefully restart/reload the service: systemctl reload nginx

2. Networking, Performance & Kernel Tuning

Q: What is the “OOM Killer” (Out of Memory Killer), and how can you tune or prevent it from killing a critical process like a database?

When the Linux kernel completely runs out of physical memory and swap, it invokes the OOM Killer to sacrifice processes to save the OS from crashing. It assigns an oom_score to processes based on how much memory they use relative to how long they’ve been running.

To protect a critical process (like MySQL or a core container):

  • Immediate adjustment: Write a negative value to the process’s score adjustment file: echo -1000 > /proc/<PID>/oom_score_adj
  • DevOps Best Practice: Instead of just adjusting scores, implement proper cgroup/container memory limits, optimize application heap sizes, or configure a healthy amount of swap space as a buffer.
Q: How do you track down network latency or dropped packets on a specific Linux server?
  1. sar -n DEV 1 5: To look at network interface statistics in real-time to check for bandwidth saturation or packet drops at the NIC level.
  2. ss -s: Gives a summary of socket statistics. If you see thousands of connections in TIME_WAIT, the server might be running out of ephemeral ports.
  3. tcpdump -i eth0 port 80 -w capture.pcap: To capture raw traffic for deep analysis in Wireshark if packet corruption or asymmetric routing is suspected.
  4. mtr backend-service.internal: (My Traceroute) Combine ping and traceroute to see exactly which network hop is introducing latency or packet loss.

3. Automation, Infrastructure & CI/CD Linux Concepts

Q: How do Linux Namespaces and Cgroups form the foundation of Docker containers?

A Docker container isn’t a virtual machine; it’s just a standard Linux process wrapped in two kernel features:

  • Namespaces (Isolate what you can SEE): It isolates the process environment. pid namespace hides other processes; net namespace provides a private routing table; mnt namespace isolates filesystem mount points.
  • Cgroups / Control Groups (Limit what you can USE): It enforces resource constraints. It dictates exactly how much CPU, memory, network bandwidth, and disk I/O a process group is allowed to consume.
Q: You need to deploy a configuration change or a hotfix to 500 Linux servers simultaneously. How do you approach this?
  • The Wrong Answer: Writing a manual bash script wrapped in a for loop using SSH. It doesn’t scale, has no error handling, and isn’t idempotent.
  • The DevOps Answer: Use a Configuration Management tool like Ansible, SaltStack, or Puppet.
    • You write an Ansible Playbook defining the desired state of the target file.
    • Execute it using forks (ansible-playbook -f 50 playbook.yml) to apply the change concurrently across infrastructure.
    • Ensure logging and monitoring (like Datadog or Prometheus) are watched during rollout to trigger an automated rollback if error rates spike.

4. Scenario-Based Design & Architecture

Q: “Our web application goes down every day at 3:00 AM for exactly 5 minutes. No deployments are happening. How do you investigate?”

This tests your infrastructure intuition. A systematic approach looks like this:

  1. Check Cron Jobs & Systemd Timers: Look inside /etc/crontab, /etc/cron.d/, and systemctl list-timers. A heavy backup script, database optimization job, or log rotation (logrotate) is likely running at 3 AM.
  2. Correlate System Logs: Check /var/log/syslog or journalctl --since "02:55" --until "03:10". Look for out-of-memory errors, service restarts, or high CPU spikes during that window.
  3. Check External Dependencies: If the local logs are clean, check if a network switch backup or an upstream cloud infrastructure snapshot is choking I/O operations at that time.