A comprehensive security audit for AWS Linux servers

A comprehensive security audit for AWS Linux servers requires checking two distinct layers: the AWS Cloud Infrastructure Level (EC2, IAM, VPC) and the OS / Host Level (Linux instance internals).

Phase 1: AWS Cloud Infrastructure Audit

1. Identity & Access Management (IAM)

  • Instance Profiles: Verify EC2 instances use IAM roles instead of hardcoded AWS access keys (.aws/credentials or environment variables).
  • Least Privilege: Ensure attached IAM policies grant only necessary permissions (e.g., restrict S3 write access to specific buckets).
  • AWS Systems Manager (SSM): Audit whether instances are attached to SSM Session Manager to eliminate the need for open SSH ports (22) and static SSH keys.

2. Network Security (VPC & Security Groups)

  • SSH Access: Verify port 22 is never open to 0.0.0.0/0. Restrict access to specific corporate IP ranges or access via SSM / Bastion host.
  • Public IP Exposure: Ensure backend servers reside in Private Subnets without public IPv4/IPv6 addresses attached.
  • Unused Security Groups:Audit and delete unattached security groups to avoid accidental misconfigurations.

3. Storage & Backups (EBS & S3)

  • EBS Encryption: Confirm all attached EBS volumes (and default account EBS configuration) have KMS encryption enabled at rest.
  • Backup Recovery: Check AWS Backup or Lifecycle Manager policies for regular snapshots and cross-region replication for critical instances.

Phase 2: OS-Level Linux Server Audit

1. SSH Hardening (/etc/ssh/sshd_config)

Run a quick audit of key SSH settings:

Bash

egrep -i "^(PermitRootLogin|PasswordAuthentication|PubkeyAuthentication|X11Forwarding)" /etc/ssh/sshd_config
  • PermitRootLogin: Set to no or prohibit-password.
  • PasswordAuthentication: Set to no (enforce key-based or SSM authentication).
  • X11Forwarding: Set to no.

2. Account & User Privilege Audit

# Check users with superuser (UID 0) access
awk -F: '($3 == "0") {print $1}' /etc/passwd
# Check for unauthorized sudoers rules
cat /etc/sudoers /etc/sudoers.d/* | grep -v '^#' | grep -v '^$'
# Find active user SSH authorized keys
find /home /root -name "authorized_keys" -exec ls -la {} \;

3. Patching & Package Management

  • Verify system packages are updated and automated security patches are active:
    • Amazon Linux / RHEL / CentOS:
      dnf updateinfo list security
    • Ubuntu / Debian:
      apt update && apt list --upgradable

4. Listening Ports & Network Services

# Inspect all active open listening ports
ss -tulpn
  • Ensure non-essential services (e.g., RPC, telnet, test databases) are stopped and disabled (systemctl disable --now <service>).

5. File System & Storage Permissions

# Check world-writable files across the system
find / -maxdepth 3 -type f -perm -0002 -ls 2>/dev/null
  • Verify system critical directories (/etc/shadow, /etc/passwd, /etc/sudoers) have strict 0600 or 0440 permissions.

Phase 3: Automated Audit Tools

Instead of manually running commands across every server, run these industry-standard open-source audit frameworks:

  1. Lynis (Host OS Security Scanner):
    sudo apt install lynis -y # Ubuntu/Debian # OR: sudo dnf install lynis -y # Amazon Linux/RHEL sudo lynis audit system
  2. Prowler (AWS Cloud Infrastructure Scanner):
    pip install prowler prowler aws --services ec2 vpc iam
  3. AWS Inspector: Enable AWS Inspector in the AWS Console for continuous automated vulnerability assessments on EC2 instances.

Here is the implementation for all three security audit requests: the automated Lynis script, the hardened SSH configuration, and the Prowler setup for AWS IAM and EC2.

1. Automated Lynis Setup & Reporting Script

Save this script as run_lynis_audit.sh, make it executable with chmod +x run_lynis_audit.sh, and run it with sudo ./run_lynis_audit.sh.

Bash

#!/bin/bash
# Description: Automated Lynis installation, audit execution, and report generation
set -e
REPORT_DIR="/var/log/lynis-reports"
TIMESTAMP=$(date +"%Y%m%d_%H%M%S")
REPORT_FILE="${REPORT_DIR}/audit_report_${TIMESTAMP}.txt"
echo "[+] Preparing report directory: ${REPORT_DIR}"
mkdir -p "${REPORT_DIR}"
chmod 700 "${REPORT_DIR}"
# Detect Package Manager and Install Lynis if missing
if ! command -v lynis &> /dev/null; then
echo "[+] Lynis not found. Installing..."
if command -v dnf &> /dev/null; then
dnf install -y lynis
elif command -v apt-get &> /dev/null; then
apt-get update && apt-get install -y lynis
else
echo "[-] Supported package manager (apt/dnf) not found. Exiting."
exit 1
fi
fi
echo "[+] Running Lynis system audit..."
# --quick skips waiting for keypresses; --warnings-only keeps summary focused
lynis audit system --quick | tee "${REPORT_FILE}"
echo "[+] Verification:"
if [ -f "${REPORT_FILE}" ]; then
echo " Success! Audit log saved to: ${REPORT_FILE}"
echo " Lynis report data written to: /var/log/lynis.log"
echo " Lynis report details written to: /var/log/lynis-report.dat"
else
echo " Audit failed to generate report file."
fi

2. AWS EC2 Hardened SSH Configuration (/etc/ssh/sshd_config)

Replace or update /etc/ssh/sshd_config with the following hardened settings.

Ini, TOML

# /etc/ssh/sshd_config - Hardened AWS EC2 Configuration
Port 22
Protocol 2
# Authentication Settings
PermitRootLogin no
MaxAuthTries 3
PubkeyAuthentication yes
PasswordAuthentication no
PermitEmptyPasswords no
HostbasedAuthentication no
# Security & Session Controls
X11Forwarding no
AllowTcpForwarding no
AllowAgentForwarding no
ClientAliveInterval 300
ClientAliveCountMax 2
LoginGraceTime 30
# Modern Safe Ciphers & MACs (Disables weak algorithms)
KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com
# Logging
SyslogFacility AUTH
LogLevel VERBOSE

Testing & Applying the Configuration:

Bash

# 1. Verify syntax before restarting (MUST pass without errors)
sudo sshd -t
# 2. Restart SSH daemon
sudo systemctl restart sshd

Verification: Open a new terminal window and attempt to connect via SSH using a password or root account to confirm access is rejected, while standard key-based access works. Do not close your existing SSH session until verified.

3. Scan AWS IAM & EC2 using Prowler

To scan your cloud infrastructure layer with Prowler from a local terminal or admin server:

# Step 1: Install Python virtual environment and Prowler
python3 -m venv prowler-env
source prowler-env/bin/activate
pip install --upgrade pip
pip install prowler
# Step 2: Configure AWS credentials or ensure active IAM profile
# Must have SecurityAudit / ViewOnlyAccess permissions
aws configure
# Step 3: Run Prowler targeted scan for EC2, VPC, and IAM services
prowler aws --services ec2 vpc iam

Generating HTML Reports:

prowler aws --services ec2 vpc iam -M html

Verification: Look for generated report files in the output/ folder (e.g., output/prowler-output-*.html). Open the HTML file in a browser to review failed checks and remediation guidelines.

Install Clair on Linux Using Docker: A Step-by-Step Guide

The easiest and most reliable way to install Clair (the open-source static container vulnerability scanner) on a Linux host is by using Docker.

Because Clair is a stateless API service, it relies on a PostgreSQL database to store its vulnerability definitions and indexing data.

Follow this step-by-step guide to set up a PostgreSQL database and run Clair on your Linux system.

Prerequisites

Ensure your Linux system has Docker and curl installed:

Bash

sudo apt update && sudo apt install -y docker.io curl # Debian/Ubuntu
# OR
sudo dnf install -y docker curl # RHEL/Rocky Linux/Fedora

Step 1: Start the PostgreSQL Database

Clair requires PostgreSQL (version 13 or newer). Start a database container and create the database for Clair:

Bash

docker run -d \
--name clair-db \
-p 5432:5432 \
-e POSTGRES_DB=clair \
-e POSTGRES_USER=postgres \
-e POSTGRES_PASSWORD=clair_password_123 \
postgres:15

Step 2: Create the Clair Configuration

Clair needs a config.yaml file to tell it how to connect to PostgreSQL and run its updaters.

  1. Create a directory for the config:mkdir -p ./clair_config
  2. Generate a basic config.yaml using this content (ensure your database connection string matches the password you used in Step 1):cat <<EOF > ./clair_config/config.yaml http_listen_addr: ":6060" introspection_addr: ":6061" log_level: "info" database: type: "pgsql" options: # Replace with your host IP if Docker cannot resolve localhost back to the host source: "host=host.docker.internal port=5432 user=postgres password=clair_password_123 dbname=clair sslmode=disable" migrations: true updater: interval: "12h" EOF

Step 3: Run the Clair Container

Deploy the official Clair v4 image from Red Hat’s Quay registry:

Bash

docker run -d \
--name clair \
-p 6060:6060 \
-p 6061:6061 \
--add-host=host.docker.internal:host-gateway \
-v $(pwd)/clair_config:/config \
-e CLAIR_CONF=/config/config.yaml \
-e CLAIR_MODE=combo \
quay.io/projectquay/clair:latest
  • CLAIR_MODE=combo: Instructs Clair to run the indexer, matcher, and updater all within a single process (standard for single-host deployments).
  • --add-host...: Allows the Clair container to reach the Postgres container running on the host’s localhost port.

Step 4: Verify the Installation

Verify that Clair is running and healthy:

Bash

curl http://localhost:6061/health

You should receive an HTTP/1.1 200 OK or a JSON status indicating that the application is healthy.

Note on Initial Setup: On its first boot, Clair’s updater will immediately begin downloading massive vulnerability feeds (CVEs, Red Hat, Ubuntu, Alpine lists). This synchronization can take anywhere from 15 minutes to an hour depending on your network speed.

Step 5: Interacting with Clair (clairctl)

To submit local or remote container images to Clair for scanning, install clairctl (the official command-line interface):

Bash

# Download the latest Linux amd64 binary
wget https://github.com/quay/clair/releases/latest/download/clairctl-linux-amd64
# Make it executable and move it to your PATH
chmod +x clairctl-linux-amd64
sudo mv clairctl-linux-amd64 /usr/local/bin/clairctl

To scan an image (such as ubuntu:focal), run:

Bash

clairctl --config ./clair_config/config.yaml report ubuntu:focal

Analyzing RAM Usage in Linux with Grafana

The Grafana Node Exporter Full dashboard memory breakdown above showing how Linux RAM is being used over time.

Let’s interpret it as a Linux administrator or SRE would.

Total Memory Usage Breakdown
CategoryMean
Applications717 MiB
Cache5.46 GiB
Buffers614 MiB
Slab380 MiB
Unused623 MiB
Everything elseSmall

The first thing to notice:

Most of the memory is being used for Cache and Buffers, not applications.

This is normal Linux behavior.


Apps – Memory used by user-space applications

Min: 632 MiB
Mean: 717 MiB
Max: 1.37 GiB

This is the memory actually consumed by running processes:

Examples:

  • Java
  • Nginx
  • Apache
  • PostgreSQL
  • Docker containers
  • Tomcat
  • Kong Gateway

Your server’s applications are only using about:

~717 MB average

which is very low.


Cache

Mean: 5.46 GiB

Linux uses free RAM to cache file contents.

Examples:

  • Recently read files
  • Database files
  • Log files
  • Container images

Linux philosophy:

Free RAM is wasted RAM.

So Linux fills unused memory with cache.

The good news:

5.46 GiB Cache

can be reclaimed instantly if applications need RAM.

This is NOT memory pressure.


Buffers

Mean: 614 MiB

Buffers are cache for block devices.

Examples:

  • Disk metadata
  • Filesystem operations
  • Read/write operations

This is normal.


Slab

Mean: 380 MiB

Kernel memory used for internal structures:

Examples:

  • inode cache
  • dentry cache
  • network structures
  • process descriptors

For example:

slabtop

might show:

dentry
inode_cache
kmalloc

380 MB is reasonable.


PageTables

Mean: 9.09 MiB

Used to translate:

Virtual Address
↓
Physical RAM Address

Every process needs page tables.

9 MB is tiny.

No concern.


Swap

Mean: 2.48 MiB

Almost no swap usage.

This is excellent.

Typically:

0 MB - 100 MB

is negligible.

If you saw:

2 GB
5 GB
10 GB

then we’d investigate memory pressure.


SwapCache

Mean: 131 KiB

Tracks swapped pages that may still exist in RAM.

Tiny amount.

Ignore.


Unused

Mean: 623 MiB
Max: 891 MiB

Actually free RAM.

Linux tries to keep this low because it prefers using RAM for caching.

Having only 623 MB free is not necessarily bad because:

Cache = 5.46 GB

can be reclaimed instantly.


Hardware Corrupted

0 B

No RAM errors detected by the kernel.

Good.


What would I conclude?

If I were reviewing this server:

Memory Health

No memory pressure

No excessive swap usage

Applications use very little RAM

Plenty of reclaimable cache

Kernel memory usage is normal

No hardware memory issues


Interview Answer

If an interviewer asks:

“Memory usage is 90%. Should I worry?”

You could answer:

Not necessarily. Linux aggressively uses free RAM for filesystem cache and buffers. I would look at “Available Memory” and swap activity rather than “Used Memory.” In this example, applications consume less than 1 GB while more than 5 GB is file cache, which can be reclaimed when needed. The negligible swap usage indicates there is no memory pressure.

Useful Commands to Verify
free -h

Look especially at:

available

rather than:

used
vmstat 1

Check:

si
so

(Swap In / Swap Out)

top

or

htop

to identify memory-consuming processes.

From these numbers, the server appears healthy and has significant RAM available through reclaimable cache, even though the “used memory” percentage may look high in Grafana.

Install Prometheus Alertmanager on Ubuntu: Step-by-Step Guide

Here’s a complete step-by-step guide to install and configure Prometheus Alertmanager on Ubuntu.


Step 1: Update System

sudo apt update && sudo apt upgrade -y

Step 2: Create a Dedicated User

sudo useradd --no-create-home --shell /bin/false alertmanager

Step 3: Download Alertmanager

Download the latest release from GitHub:

cd /tmp
wget https://github.com/prometheus/alertmanager/releases/latest/download/alertmanager-0.27.0.linux-amd64.tar.gz
tar -xvf alertmanager-0.27.0.linux-amd64.tar.gz
cd alertmanager-0.27.0.linux-amd64

Check https://github.com/prometheus/alertmanager/releases for the latest version and substitute accordingly.


Step 4: Install Binaries

Move the binaries to /usr/local/bin/:

sudo mv alertmanager /usr/local/bin/
sudo mv amtool /usr/local/bin/

Step 5: Create Directories

Create config and data directories and set ownership:

sudo mkdir /etc/alertmanager
sudo mkdir /var/lib/alertmanager
sudo chown alertmanager:alertmanager /etc/alertmanager
sudo chown alertmanager:alertmanager /var/lib/alertmanager

Step 6: Create the Config File

sudo nano /etc/alertmanager/alertmanager.yml

Paste a basic configuration (example with email):

global:
smtp_smarthost: 'smtp.gmail.com:587'
smtp_from: 'alertmanager@example.com'
smtp_auth_username: 'your@gmail.com'
smtp_auth_password: 'your-app-password'
route:
group_by: ['alertname']
group_wait: 30s
group_interval: 5m
repeat_interval: 1h
receiver: 'email-alerts'
receivers:
- name: 'email-alerts'
email_configs:
- to: 'you@example.com'
inhibit_rules:
- source_match:
severity: 'critical'
target_match:
severity: 'warning'
equal: ['alertname', 'dev', 'instance']

Set proper ownership:

sudo chown alertmanager:alertmanager /etc/alertmanager/alertmanager.yml

Step 7: Create a systemd Service

Create the service file:

sudo nano /etc/systemd/system/alertmanager.service
[Unit]
Description=Prometheus Alertmanager
Wants=network-online.target
After=network-online.target
[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager \
--config.file=/etc/alertmanager/alertmanager.yml \
--storage.path=/var/lib/alertmanager
[Install]
WantedBy=multi-user.target

Step 8: Start and Enable the Service

sudo systemctl daemon-reload
sudo systemctl start alertmanager
sudo systemctl enable alertmanager
sudo systemctl status alertmanager

Alertmanager will be available at http://<your-server-ip>:9093.


Step 9: Integrate with Prometheus

Add this to your prometheus.yml to wire Alertmanager in:

alerting:
alertmanagers:
- static_configs:
- targets: ['localhost:9093']

Optionally, also scrape Alertmanager’s own metrics:

scrape_configs:
- job_name: alertmanager
static_configs:
- targets: ['localhost:9093']

Then restart Prometheus:

sudo systemctl restart prometheus

Step 10: (Optional) Firewall Rules

If Alertmanager should only be accessible locally:

sudo ufw allow from 127.0.0.1 to any port 9093
sudo ufw deny 9093

Verify & Troubleshoot

# Check logs
sudo journalctl -u alertmanager -f
# Validate config file
amtool check-config /etc/alertmanager/alertmanager.yml

Common Receivers Beyond Email
ReceiverKey Config Field
Slackslack_configs with api_url
PagerDutypagerduty_configs with service_key
Webhookwebhook_configs with url
OpsGenieopsgenie_configs with api_key

Here’s the complete email configuration for Alertmanager:

Update /etc/alertmanager/alertmanager.yml

sudo nano /etc/alertmanager/alertmanager.yml
global:
smtp_smarthost: 'smtp.gmail.com:587'
smtp_from: 'alertmanager@yourdomain.com'
smtp_auth_username: 'your@gmail.com'
smtp_auth_password: 'your-app-password' # Gmail App Password, NOT your login password
smtp_require_tls: true
route:
receiver: 'email-alerts'
group_by: ['alertname', 'severity']
group_wait: 30s # Wait before sending first alert in a group
group_interval: 5m # Wait before sending new alerts for an existing group
repeat_interval: 4h # Resend if alert is still firing after this duration
# Optional: route critical alerts separately
routes:
- match:
severity: critical
receiver: 'email-critical'
repeat_interval: 1h
receivers:
- name: 'email-alerts'
email_configs:
- to: 'your-team@example.com'
send_resolved: true # Also notify when alert resolves
- name: 'email-critical'
email_configs:
- to: 'oncall@example.com'
send_resolved: true
headers:
Subject: '[CRITICAL] {{ .GroupLabels.alertname }}'
inhibit_rules:
- source_match:
severity: 'critical'
target_match:
severity: 'warning'
equal: ['alertname', 'instance']

Gmail Setup (Important)

Gmail blocks plain passwords — you need an App Password:

  1. Go to https://myaccount.google.com/security
  2. Enable 2-Step Verification (required)
  3. Go to App Passwords → select “Mail” → generate
  4. Use that 16-character password in smtp_auth_password

For other providers, swap out the SMTP settings:

Providersmtp_smarthost
Gmailsmtp.gmail.com:587
Outlook/Office365smtp.office365.com:587
SendGridsmtp.sendgrid.net:587
AWS SESemail-smtp.<region>.amazonaws.com:587

Apply & Test

# Validate config before restarting
amtool check-config /etc/alertmanager/alertmanager.yml
# Restart to apply
sudo systemctl restart alertmanager
# Send a test alert manually
amtool alert add alertname="TestAlert" severity="warning" \
--alertmanager.url=http://localhost:9093
# Check it appears
amtool alert --alertmanager.url=http://localhost:9093

Check Logs if Email Fails

sudo journalctl -u alertmanager -f

Common issues to look for:

  • authentication failed → wrong App Password
  • connection refused → check SMTP host/port
  • tls: no supported versions → set smtp_require_tls: false for port 25

Once you confirm email is working, you can layer in Slack or PagerDuty alongside it in the same config.

Understanding Alertmanager in the Prometheus Ecosystem

Alertmanager is the specialized component of the Prometheus ecosystem that handles alerts.

While Prometheus is responsible for recording metrics and triggering an alert when something goes wrong (like a server running out of RAM), Prometheus itself has no idea how to send an email, slack message, or page an engineer. It hands that problem off to Alertmanager.

The Workflow: How an Alert Moves

[ Your Servers ] ──(Metrics)──► [ Prometheus ]
│
▼ (Metrics hit a threshold)
[ Alertmanager ]
│
┌──────────────────────┼──────────────────────┐
▼ ▼ ▼
[ Email ] [ Slack ] [ PagerDuty ]
  1. Prometheus constantly evaluates your data against rules you write (e.g., Is CPU usage > 90% for more than 5 minutes?). If yes, Prometheus fires the alert and shoots it over to Alertmanager.
  2. Alertmanager takes that raw alert and figures out how to handle it intelligently.
  3. The Receiver (Slack, Email, PagerDuty, Discord, etc.) actually delivers the message to your team.

Why do we need a separate tool just for alerts?

You might wonder why Prometheus doesn’t just send emails directly. Alertmanager exists because in a large infrastructure (like your 20+ servers), raw alerts can quickly turn into a nightmare without three critical features:

1. Grouping (Preventing “Alert Fatigue”)

Imagine a network switch dies, cutting off 15 of your Docker hosts at the exact same moment. If Prometheus sent alerts directly, you would get 15 individual emails or Slack pings within 10 seconds, blowing up your phone.

Alertmanager groups them. It sees that 15 servers failed for the same reason, bundles them into one single notification: “Alert: 15 hosts are currently unreachable.”

2. Inhibition (Muting Dependent Alerts)

If a physical server completely crashes, both the Host Down alert and the 10 Containers on this Host are Down alerts will trigger. You don’t need to be told the containers are down if you already know the whole machine is dead. Alertmanager allows you to mute (inhibit) the container alerts if the main host alert is already active.

3. Silencing

If you are doing planned maintenance on server-05 on a Tuesday morning, you can log into Alertmanager’s web interface and click Silence for that specific server for 2 hours. Prometheus will keep tracking the metrics, but Alertmanager will block any notifications from waking up your team while you work on it.

A Quick Look at How It Connects to Your Stack

Since you are running your central monitoring stack via Portainer, you would add an Alertmanager service to your existing Docker Compose file right alongside Prometheus and Grafana:

YAML

  alertmanager:
    image: prom/alertmanager:latest
    container_name: central-alertmanager
    ports:
      - "9093:9093"
    volumes:
      - /opt/alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml
    restart: unless-stopped


In your alertmanager.yml file, you define your routing. For example, telling it to route critical alerts to Slack and warning alerts to Email:

YAML

route:
receiver: 'slack-notifications'
receivers:
- name: 'slack-notifications'
slack_configs:
- api_url: 'https://hooks.slack.com/services/T000/B000/XXXXXX'
channel: '#ops-alerts'
Alertmanager vs. Grafana Alerting

Note that Grafana can also send alerts based on your dashboards. However, in professional setups using Prometheus, engineers prefer Alertmanager because it can process math across your entire fleet instantly at the database level, rather than waiting for a Grafana graph to load and trigger.

Install Prometheus Node Exporter on Bare-metal Linux

To install the Prometheus Node Exporter directly onto a bare-metal Linux server without using Docker, you want to set it up as a native system service. This ensures it starts automatically when the server reboots and runs quietly in the background.

Here is the step-by-step production setup guide. You can do this manually on one server, or drop these steps into your Ansible workflow.

Step 1: Create a Dedicated System User

For security, Node Exporter should never run as root. Create a system user with no login shells or home directory.

Bash

sudo useradd --no-create-home --shell /bin/false node_exporter

Step 2: Download and Extract the Binary

Navigate to the /tmp folder, download the latest stable release of Node Exporter, and extract it.

Bash

cd /tmp
wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
tar -xvf node_exporter-1.8.2.linux-amd64.tar.gz
install last version

(Note: You can check the Prometheus Downloads page to see if a newer version is available and change the version number accordingly.)

This is the last version:

wget https://github.com/prometheus/node_exporter/releases/download/v1.11.1/node_exporter-1.11.1.linux-amd64.tar.gz

Step 3: Move the Binary into System Path

Move the executable binary to /usr/local/bin so it can be executed system-wide, and assign ownership to the user you created in Step 1.

Bash

sudo mv node_exporter-1.8.2.linux-amd64/node_exporter /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

Clean up the remaining installation files you no longer need:

Bash

rm -rf node_exporter-1.8.2.linux-amd64*

Step 4: Create a Systemd Service File

To manage Node Exporter like any other Linux service (e.g., systemctl start), create a systemd service definition file.

Open a new file using your preferred text editor:

Bash

sudo nano /etc/systemd/system/node_exporter.service

Paste the following configuration into the file:

Ini, TOML

[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target

Save and exit the file (in Nano, press Ctrl+O, Enter, then Ctrl+X).

Step 5: Start and Enable Node Exporter

Reload the systemd manager to recognize your new service configuration, start Node Exporter, and enable it so it boots up automatically with the server.

Bash

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter

Verify it’s running:

Check the status of the service to ensure there are no errors:

Bash

sudo systemctl status node_exporter

You can also verify that it is successfully broadcasting metrics by hitting its port (9100) via curl:

Bash

curl http://localhost:9100/metrics

If you see a wall of text containing metrics starting with node_..., the installation was a complete success.

Step 6: Update Your Central Prometheus Server

Just like you did with your Docker containers, you must now tell your central Prometheus server to scrape this physical machine. Add the bare metal server’s IP address under your node-host-os job configuration in your central prometheus.yml file:

YAML

  - job_name: 'node-host-os'
    static_configs:
      - targets: ['192.168.1.50:9100'] # Your existing Docker hosts
      - targets: ['192.168.1.85:9100'] # Your new bare metal server IP
        labels: 
          instance: 'bare-metal-01'


Reload or restart your central Prometheus container, and the bare-metal server will instantly feed into your existing Grafana Node Exporter Full dashboard!

Effective Prometheus and Grafana Setup for Linux and Windows Servers

A complete Prometheus + Grafana monitoring solution for 20 Linux servers + 20 Windows servers is absolutely achievable with a single, scalable monitoring stack. Below is a clear architecture, required components, and step‑by‑step setup based on authoritative sources.


Concise Takeaway

You will deploy Prometheus on a central Linux server, install node_exporter on all 20 Linux hosts, install windows_exporter on all 20 Windows hosts, and visualize everything in Grafana. Prometheus scrapes all 40 servers on a schedule and Grafana queries Prometheus for dashboards. Grafana Labs mylinux.work


Recommended Architecture

  • Prometheus server (Linux) — scrapes metrics from all exporters.
  • Node Exporter on each Linux server — exposes CPU, RAM, disk, network metrics.
  • Windows Exporter on each Windows server — exposes Windows OS metrics on port 9182. devopsofworld.com
  • Grafana server — dashboards + alert visualization.
  • Alertmanager (optional but recommended) — email/Slack/Teams alerts. mylinux.work

1. Install Exporters on All Servers

Linux (20 servers)

Install node_exporter on each Linux host:

  • Prometheus documentation recommends Node Exporter for Linux metrics. Grafana Labs
  • Metrics exposed on port 9100.

Windows (20 servers)

Install windows_exporter:

Example Windows service install (from source):

nssm install Windows_exporter C:\path\windows_exporter.exe --collectors.enabled="cpu,logical_disk,net,os,service,system"

2. Prometheus Configuration

Prometheus scrapes all 40 servers.
Example prometheus.yml:

scrape_configs:
- job_name: 'linux'
static_configs:
- targets:
- 'linux01:9100'
- 'linux02:9100'
# ... all 20 Linux servers
- job_name: 'windows'
static_configs:
- targets:
- 'win01:9182'
- 'win02:9182'
# ... all 20 Windows servers

Windows scrape configuration is consistent with documented examples. mylinux.work


3. Grafana Setup

Grafana connects to Prometheus as a data source.

Recommended dashboards:

  • Node Exporter Full (Linux)
  • Windows Exporter Dashboard (Windows) — available from Grafana community. devopsofworld.com

4. Alerting (Optional but Recommended)

Use Alertmanager for notifications:

  • Email
  • Slack / Teams
  • Webhooks

Alertmanager is part of the standard Linux monitoring stack. mylinux.work


5. Hardware Requirements (for 40 servers)

For a single Prometheus + Grafana node:

ComponentRecommended
CPU4 vCPUs
RAM8–16 GB
Storage100–200 GB SSD
OSLinux (Ubuntu recommended)

This is inferred from typical Prometheus deployments for ~50 nodes.


6. Deployment Model Options

Option A — Single Monitoring Server (simplest)

  • Prometheus + Grafana on one VM
  • Good for up to ~100 nodes

Option B — Distributed (Prometheus + Grafana separate)

  • Prometheus on one VM
  • Grafana on another
  • More scalable; recommended by LinuxConfig architecture. LinuxConfig.org

7. What You Will Monitor

  • CPU, RAM, Disk, Network
  • Windows services
  • Windows event metrics
  • Linux system metrics
  • Optional: Docker, cAdvisor, SQL Server, IIS, etc. kx.cloudingenium.com

Next step

  • Prometheus configuration
  • Deployment diagram
  • Step-by-step guide

Here is a clear, production‑grade monitoring architecture diagram for your Prometheus + Grafana setup covering 20 Linux + 20 Windows servers.
I’m giving you both a visual ASCII diagram and a conceptual architecture so you can use it in documentation or design reviews.


Monitoring Architecture Diagram (Prometheus + Grafana)

                           ┌──────────────────────────────┐
                           │        Grafana Server         │
                           │  - Dashboards & Visualization │
                           │  - Connects to Prometheus     │
                           └───────────────▲──────────────┘
                                           │
                                           │  Queries
                                           │
                           ┌───────────────┴──────────────┐
                           │       Prometheus Server      │
                           │  - Scrapes all exporters     │
                           │  - Stores time‑series data   │
                           │  - Pushes alerts to AM       │
                           └───────────────▲──────────────┘
                                           │
                                           │  Scrape (HTTP)
                                           │
        ┌──────────────────────────┬───────┴───────────────┬──────────────────────────┐
        │                          │                       │                          │
        │                          │                       │                          │
┌───────────────┐        ┌────────────────┐       ┌────────────────┐        ┌───────────────┐
│ 20 Linux       │       │ 20 Windows     │       │ Alertmanager   │        │ Optional Tools │
│ Servers        │       │ Servers        │       │ (Email/Slack)  │        │  - cAdvisor    │
│ node_exporter  │       │ windows_exporter│      │ Notifications  │        │  - Blackbox    │
│ Port 9100      │       │ Port 9182       │      │ Routing        │        │  - SNMP Export │
└───────────────┘        └────────────────┘       └────────────────┘        └───────────────┘




How the Architecture Works

  • Prometheus scrapes metrics from:
    • node_exporter on each Linux server
    • windows_exporter on each Windows server
  • Grafana queries Prometheus to build dashboards.
  • Alertmanager receives alerts from Prometheus and sends notifications.
  • Exporters expose metrics over HTTP; Prometheus pulls them on a schedule.

Component Breakdown

  • Prometheus Server
    Central collector + time‑series database.
  • Grafana Server
    Dashboards, visualization, alerting UI.
  • Node Exporter
    Linux OS metrics (CPU, RAM, disk, network).
  • Windows Exporter
    Windows OS metrics (services, CPU, memory, disks).
  • Alertmanager
    Email, Slack, Teams, webhook alerts.

Optional Add‑Ons

  • Blackbox Exporter — HTTP/TCP/ICMP checks
  • cAdvisor — container metrics
  • SNMP Exporter — network devices

Install Prometheus Node Exporter on Bare-Metal Linux

To install the Prometheus Node Exporter directly onto a bare-metal Linux server without using Docker, you want to set it up as a native system service. This ensures it starts automatically when the server reboots and runs quietly in the background.

Here is the step-by-step production setup guide. You can do this manually on one server, or drop these steps into your Ansible workflow.

Step 1: Create a Dedicated System User

For security, Node Exporter should never run as root. Create a system user with no login shells or home directory.

Bash

sudo useradd --no-create-home --shell /bin/false node_exporter

Step 2: Download and Extract the Binary

Navigate to the /tmp folder, download the latest stable release of Node Exporter, and extract it.

Bash

cd /tmp
wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
tar -xvf node_exporter-1.8.2.linux-amd64.tar.gz

(Note: You can check the Prometheus Downloads page to see if a newer version is available and change the version number accordingly.)

Step 3: Move the Binary into System Path

Move the executable binary to /usr/local/bin so it can be executed system-wide, and assign ownership to the user you created in Step 1.

Bash

sudo mv node_exporter-1.8.2.linux-amd64/node_exporter /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

Clean up the remaining installation files you no longer need:

Bash

rm -rf node_exporter-1.8.2.linux-amd64*

Step 4: Create a Systemd Service File

To manage Node Exporter like any other Linux service (e.g., systemctl start), create a systemd service definition file.

Open a new file using your preferred text editor:

Bash

sudo nano /etc/systemd/system/node_exporter.service

Paste the following configuration into the file:

Ini, TOML

[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target

Save and exit the file (in Nano, press Ctrl+O, Enter, then Ctrl+X).

Step 5: Start and Enable Node Exporter

Reload the systemd manager to recognize your new service configuration, start Node Exporter, and enable it so it boots up automatically with the server.

Bash

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter
Verify it’s running:

Check the status of the service to ensure there are no errors:

Bash

sudo systemctl status node_exporter

You can also verify that it is successfully broadcasting metrics by hitting its port (9100) via curl:

Bash

curl http://localhost:9100/metrics

If you see a wall of text containing metrics starting with node_..., the installation was a complete success.

Step 6: Update Your Central Prometheus Server

Just like you did with your Docker containers, you must now tell your central Prometheus server to scrape this physical machine. Add the bare metal server’s IP address under your node-host-os job configuration in your central prometheus.yml file:

YAML

  - job_name: 'node-host-os'
    static_configs:
      - targets: ['192.168.1.50:9100'] # Your existing Docker hosts
      - targets: ['192.168.1.85:9100'] # Your new bare metal server IP
        labels: 
          instance: 'bare-metal-01'

Reload or restart your central Prometheus container, and the bare-metal server will instantly feed into your existing Grafana Node Exporter Full dashboard!

Building a Grafana Dashboard for Multi-Host Metrics

Centralized monitoring architecture.

One central Grafana and Prometheus server that pulls metrics from 20 lightweight cAdvisor agents running across your network.

Here is the cleanest way to architect and deploy this.

Centralized Architecture Overview

[ Central Monitor Server ] [ 20x Remote Docker Hosts ]
┌────────────────────────┐ ┌─────────────────────────┐
│ Grafana │ │ Docker Host 01 │
│ ▲ │ │ └─ cAdvisor (Port 8080)│
│ │ (Queries) │ └─────────────────────────┘
│ Prometheus │◄─────────────┐ ▲
└────────────────────────┘ (Scrapes │ │
over HTTP) ├─────────────┤
│ ▼
│┌─────────────────────────┐
││ Docker Host 20 │
└┤ └─ cAdvisor (Port 8080)│
└─────────────────────────┘
  • The Agents (Remote Hosts): Every one of your 20 servers runs just a single, lightweight cAdvisor container. They expose their metrics on port 8080.
  • The Core (Central Host): One designated management server runs Prometheus and Grafana. Prometheus is configured to reach out over your network to all 20 servers to grab their metrics.

Step 1: Deploy cAdvisor on all 20 Remote Hosts

You need to spin up cAdvisor on every Docker server. If you use an automation tool like Ansible, this is a one-click playbook. Otherwise, run this docker run command on each machine:

Bash

docker run \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:ro \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--volume=/dev/disk/:/dev/disk:ro \
--publish=8080:8080 \
--detach=true \
--name=cadvisor \
--privileged \
--device=/dev/kmsg \
--restart=always \
gcr.io/cadvisor/cadvisor:latest

Security Note: By publishing 8080:8080, cAdvisor metrics are public to anyone who can reach that IP. Ensure your internal firewall/security groups only allow traffic to port 8080 from your Central Prometheus Server IP.

Step 2: Configure the Central Prometheus Server

On your central monitoring server, create a prometheus.yml file. Instead of pointing to localhost, you will list your 20 servers using Prometheus targets and labels. Labels are crucial here because they allow you to filter your Grafana dashboard by specific servers.

YAML

global:
scrape_interval: 15s
scrape_configs:
- job_name: 'docker-swarm-nodes'
static_configs:
- targets: ['192.168.1.50:8080']
labels:
instance: 'prod-web-01'
env: 'production'
- targets: ['192.168.1.51:8080']
labels:
instance: 'prod-db-01'
env: 'production'
- targets: ['192.168.1.70:8080']
labels:
instance: 'stage-app-01'
env: 'staging'
# ... add the remaining 17 servers here

Start Prometheus and Grafana on this central machine using a simplified version of the Docker Compose template from earlier (removing the local cAdvisor block from it).

Step 3: Multi-Host Grafana Dashboard

Once Prometheus is scraping all 20 hosts, you need a Grafana dashboard that lets you switch between them smoothly.

  1. Go to Grafana -> Dashboards -> Import.
  2. Use Dashboard ID: 14282 or 10619 (both are heavily optimized for multi-host setups).
  3. These dashboards automatically generate a dropdown menu at the top left labeled “Instance” or “Host” based on the labels you defined in your prometheus.yml.

You can now view your entire fleet’s aggregated resource consumption, or drill down into a specific container running on prod-db-01.

Pro-Tips for Managing 20+ Hosts

  • Node Exporter: cAdvisor only monitors container metrics. If you want to monitor the host Linux OS itself (host disk space, total RAM usage, bare-metal CPU temperature), deploy Prometheus Node Exporter alongside cAdvisor on all 20 nodes, exposing it on port 9100.
  • Scale Warning: Scraping 20 hosts every 15 seconds will generate a decent chunk of data. Ensure your Central Prometheus server has a reasonable storage retention policy set (e.g., --storage.tsdb.retention.time=15d to keep data for 15 days) so it doesn’t quietly fill up the server’s hard drive.

Monitoring Docker with Grafana and cAdvisor

Combining Grafana and cAdvisor (Container Advisor) is the standard open-source recipe for monitoring Docker and Kubernetes container metrics (like CPU, memory, network, and disk usage).

Because cAdvisor only keeps a tiny buffer of real-time data in memory, you need a time-series database (almost always Prometheus or Grafana Alloy) to scrape that data and hand it off to Grafana for visualization.

Here is a breakdown of how the architecture works, how to set it up, and how to get a dashboard running.

The Monitoring Pipeline

  1. cAdvisor: Sits on the host machine, hooks into the Linux kernel cgroups, and collects resource usage from all running containers. It exposes these raw numbers at a /metrics endpoint.
  2. Prometheus: Periodically “scrapes” (pulls) the data from cAdvisor’s /metrics endpoint and stores it as historical time-series data.
  3. Grafana: Queries Prometheus using PromQL and plots the data onto clean, interactive dashboards.

Quick Setup: Docker Compose Example

The easiest way to spin up cAdvisor, Prometheus, and Grafana all at once to monitor your local Docker containers is by using a docker-compose.yml file.

YAML

version: '3.8'
services:
cadvisor:
image: gcr.io/cadvisor/cadvisor:latest
container_name: cadvisor
ports:
- "8080:8080"
volumes:
- /:/rootfs:ro
- /var/run:/var/run:ro
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
- /dev/disk/:/dev/disk:ro
devices:
- /dev/kmsg
restart: unless-stopped
prometheus:
image: prom/prometheus:latest
container_name: prometheus
ports:
- "9090:9090"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
restart: unless-stopped
grafana:
image: grafana/grafana:latest
container_name: grafana
ports:
- "3000:3000"
restart: unless-stopped
The Prometheus Config (prometheus.yml)

To tell Prometheus to scrape your cAdvisor container, create a prometheus.yml file in the same directory:

YAML

global:
scrape_interval: 15s
scrape_configs:
- job_name: 'cadvisor'
static_configs:
- targets: ['cadvisor:8080']

Run docker compose up -d, and your basic infrastructure is live!

Visualizing with Grafana Dashboards

Instead of building your container monitoring dashboards from scratch, you can import highly optimized community templates.

Recommended Pre-built Dashboards
  1. Log into your Grafana instance (usually http://localhost:3000 — default credentials are admin / admin).
  2. Add Prometheus as your data source (Connections > Data sources > Add data source).
  3. Go to Dashboards > New > Import.
  4. Paste one of these popular community Dashboard IDs:
    • 19908 (cAdvisor Docker Insights – clean, official-feeling modern dashboard)
    • 14282 (Cadvisor Exporter – great for focused container-by-container metrics)
    • 19792 (Advanced cAdvisor dashboard with support for Docker Compose projects)

Key cAdvisor Metrics to Know

When creating your own panels or alerts, look out for these fundamental metric names:

  • CPU Usage: container_cpu_usage_seconds_total (usually paired with rate() to calculate CPU percentage like sum(rate(container_cpu_usage_seconds_total[5m])) by (name))
  • Memory Usage: container_memory_usage_bytes (tells you the exact RAM consumption)
  • Network Traffic: container_network_receive_bytes_total and container_network_transmit_bytes_total
  • Disk I/O: container_fs_reads_bytes_total and container_fs_writes_bytes_total