docker

Understanding Container Scanners: Trivy, Docker Scout, Clair Features

techhadoop docker, security

When comparing Clair, Trivy, and Docker Scout, you are looking at three highly capable container vulnerability scanners that target completely different stages of the DevOps lifecycle.

Rather than one being “better,” they are architected for different deployment models: Trivy is the Swiss Army knife for CI/CD and terminal tasks, Docker Scout is built directly into the developer’s local inner loop, and Clair is designed to run silently inside container registries.

The High-Level Breakdown

Feature / AttributeTrivy (Aqua Security)Docker Scout (Docker)Clair (Red Hat / Quay)
Primary Use CaseLocal CLI scans, CI/CD pipelines, and IaC/Kubernetes auditing.Local developer workflow, Docker Hub integration, and real-time remediation.Automated scanning built natively into container registries.
ArchitectureSingle binary. Self-contained and lightweight with zero external dependencies.SaaS-hybrid. CLI/Desktop plugin that pushes metadata to a Docker-managed backend.Microservices. Stateless API engine that requires an external PostgreSQL database.
What it Can ScanContainer images, Git repositories, local filesystems, Kubernetes manifests, and IaC.Container images, base-image layers, and supply chain dependencies.Container images (specifically broken down layer-by-layer).
Remediation HelpLists CVEs, severity, and fixed versions.Shows exact base image recommendations and path updates.Lists vulnerabilities, but offers no direct “fix-it” guidance.
LicenseOpen Source (Apache-2.0).Proprietary (Free tier available, paid tiers for advanced features).Open Source (Apache-2.0).

1. Trivy: The Open-Source Standard for CI/CD

Trivy is incredibly popular because of its simplicity. You download the binary, point it at an image, and it immediately prints out a vulnerability table.

  • The Setup: It requires no configuration, daemon, or database setup. It pulls and caches its own vulnerability database locally during runtime.
  • Beyond Container Images: Trivy doesn’t just scan packages. It can scan your Terraform files, Helm charts, and Dockerfiles for security misconfigurations.
  • VEX Support: It supports modern standards like Vulnerability Exploitability eXchange (VEX), allowing you to filter out false positives or unexploitable CVEs in your images.

2. Docker Scout: The Developer-First Assistant

Docker Scout (which replaced Docker’s older scanning engines) is deeply embedded inside the Docker CLI and Docker Desktop.

  • The Integration: If you use Docker Desktop, you already have it. Running docker scout quickview or docker scout cves gives you instant feedback.
  • Remediation focus: Traditional scanners give you a massive list of 300 CVEs and leave you to figure out what to do. Docker Scout analyzes your image’s base layers and tells you exactly what to do: “If you update your base image from Python 3.10-slim to 3.10.12-slim, you will eliminate 45 critical vulnerabilities.”
  • Real-time Monitoring: It tracks security policies and notifies you if a newly discovered zero-day affects an image you pushed weeks ago.

3. Clair: The Registry Sentinel

Clair was built by CoreOS (now Red Hat) to act as the scanning backend for container registries like Quay and Harbor.

  • Registry-First design: Unlike Trivy or Scout, Clair is not really meant to be run as a quick local CLI tool. It operates as an API-driven daemon.
  • Efficient Layered Scanning: When you push an image, Clair scans it layer-by-layer. If you push a new version of your app where only the top layer changed, Clair only scans that new layer, saving massive amounts of compute and database I/O.
  • Operational Overhead: Running Clair means deploying its API services and managing a PostgreSQL database. This makes it a great choice for Platform Engineers hosting private enterprise registries, but overkill for individual developers.

Which One Should You Choose?

  • Use Trivy if: You want a 100% open-source tool that integrates beautifully with GitHub Actions, GitLab CI, or Tekton, or if you need to scan Kubernetes clusters and Terraform code alongside your container images.
  • Use Docker Scout if: Your team is already heavily dependent on Docker Desktop and Docker Hub, and you want actionable, clear advice on how to rewrite your Dockerfiles to remediate vulnerabilities quickly.
  • Use Clair if: You are building or maintaining a self-hosted private container registry (like Harbor) and want a background engine to automatically scan images every time a developer pushes code.

Install Clair on Linux Using Docker: A Step-by-Step Guide

techhadoop docker, linux, security

The easiest and most reliable way to install Clair (the open-source static container vulnerability scanner) on a Linux host is by using Docker.

Because Clair is a stateless API service, it relies on a PostgreSQL database to store its vulnerability definitions and indexing data.

Follow this step-by-step guide to set up a PostgreSQL database and run Clair on your Linux system.

Prerequisites

Ensure your Linux system has Docker and curl installed:

Bash

sudo apt update && sudo apt install -y docker.io curl   # Debian/Ubuntu
# OR
sudo dnf install -y docker curl                       # RHEL/Rocky Linux/Fedora

Step 1: Start the PostgreSQL Database

Clair requires PostgreSQL (version 13 or newer). Start a database container and create the database for Clair:

Bash

docker run -d \
  --name clair-db \
  -p 5432:5432 \
  -e POSTGRES_DB=clair \
  -e POSTGRES_USER=postgres \
  -e POSTGRES_PASSWORD=clair_password_123 \
  postgres:15

Step 2: Create the Clair Configuration

Clair needs a config.yaml file to tell it how to connect to PostgreSQL and run its updaters.

  1. Create a directory for the config:mkdir -p ./clair_config
  2. Generate a basic config.yaml using this content (ensure your database connection string matches the password you used in Step 1):cat <<EOF > ./clair_config/config.yaml http_listen_addr: ":6060" introspection_addr: ":6061" log_level: "info" database: type: "pgsql" options: # Replace with your host IP if Docker cannot resolve localhost back to the host source: "host=host.docker.internal port=5432 user=postgres password=clair_password_123 dbname=clair sslmode=disable" migrations: true updater: interval: "12h" EOF

Step 3: Run the Clair Container

Deploy the official Clair v4 image from Red Hat’s Quay registry:

Bash

docker run -d \
  --name clair \
  -p 6060:6060 \
  -p 6061:6061 \
  --add-host=host.docker.internal:host-gateway \
  -v $(pwd)/clair_config:/config \
  -e CLAIR_CONF=/config/config.yaml \
  -e CLAIR_MODE=combo \
  quay.io/projectquay/clair:latest
  • CLAIR_MODE=combo: Instructs Clair to run the indexer, matcher, and updater all within a single process (standard for single-host deployments).
  • --add-host...: Allows the Clair container to reach the Postgres container running on the host’s localhost port.

Step 4: Verify the Installation

Verify that Clair is running and healthy:

Bash

curl http://localhost:6061/health

You should receive an HTTP/1.1 200 OK or a JSON status indicating that the application is healthy.

Note on Initial Setup: On its first boot, Clair’s updater will immediately begin downloading massive vulnerability feeds (CVEs, Red Hat, Ubuntu, Alpine lists). This synchronization can take anywhere from 15 minutes to an hour depending on your network speed.

Step 5: Interacting with Clair (clairctl)

To submit local or remote container images to Clair for scanning, install clairctl (the official command-line interface):

Bash

# Download the latest Linux amd64 binary
wget https://github.com/quay/clair/releases/latest/download/clairctl-linux-amd64
# Make it executable and move it to your PATH
chmod +x clairctl-linux-amd64
sudo mv clairctl-linux-amd64 /usr/local/bin/clairctl

To scan an image (such as ubuntu:focal), run:

Bash

clairctl --config ./clair_config/config.yaml report ubuntu:focal

Understanding CNI: The Backbone of Container Networking

techhadoop CNI, docker, kubernetes, OCP

A CNI (Container Network Interface) is a standardized specification and set of libraries used by container runtimes (like Kubernetes, OpenShift, or Mesos) to configure network interfaces for containers and pods dynamically.

In the early days of containerization, every orchestration platform had its own hardcoded way of handling container networking, which created massive vendor lock-in. The CNI was created as a neutral cloud-native standard to solve this. It decouples the core container runtime engine from the underlying network plug-in architecture.

1. The Core Mechanics: How CNI Works

In Kubernetes, a pod cannot exist without a network interface and a unique IP address. However, the kubelet (the node agent) doesn’t actually know how to allocate IPs or configure virtual network switches. It delegates this task completely to the CNI plug-in.

When a user triggers a request to launch a new pod, the following sequence happens under the hood:

  1. Pod Creation: The container runtime initializes the network namespace for the new pod.
  2. The CNI Call: The runtime executes a specific binary file located in the node’s /opt/cni/bin/ directory, invoking a standard verb command like ADD, DEL, or CHECK. It passes configuration JSON metadata down via standard input.
  3. Interface Provisioning: The CNI plug-in intercepts the request, generates a virtual ethernet pair (veth), and binds one end inside the pod’s isolated network namespace (usually named eth0) and anchors the other end to the host node’s virtual bridge interface.
  4. IPAM Allocation: The CNI plug-in invokes its internal IPAM (IP Address Management) module to fetch a vacant IP address from the cluster’s CIDR block allocation map and binds it to the pod’s interface.
  5. Registration: The CNI plug-in finishes its setup, returns the assigned IP, MAC address, and routing table metadata back to the runtime container engine as a structured JSON payload, and exits.

2. Common Enterprise CNI Implementations

Because CNI is simply a standardized interface specification, organizations can choose different plugins based on their networking and performance requirements:

Flannel (Legacy/Simple)

A minimalist, mature CNI provider that configures a simple Layer-3 overlay network using VXLAN encapsulation. It maps a flat, routing web across the entire cluster. It does not support advanced traffic controls like Kubernetes Network Policies (firewalls), meaning all pods can communicate with all other pods unhindered.

Calico (Security & Scale)

An enterprise-grade plug-in favored for production networks. Instead of using heavy packet encapsulation overhead, Calico routes packets using the native BGP (Border Gateway Protocol). This allows your Kubernetes worker nodes to announce pod IP routes directly to your physical enterprise top-of-rack routers. It also features a highly performant firewall engine to enforce granular Network Policies.

Cilium (Modern / eBPF Native)

The cutting-edge standard for high-performance cloud-native networking. Cilium completely bypasses the traditional Linux iptables or IPVS firewall routing structures. Instead, it utilizes eBPF (Extended Berkeley Packet Filter) to inject bytecode logic directly inside the live Linux kernel space. This allows Cilium to process packets, enforce security policies, and trace telemetry metrics at near-instant speeds with minimal CPU consumption.

3. The OpenShift Standard: OVNKubernetes CNI

Red Hat OpenShift 4 standardizes on OVNKubernetes as its default out-of-the-box CNI provider. This plugin is an enterprise network engine built natively on top of OVN (Open Virtual Network) and OVS (Open vSwitch).

OVNKubernetes brings several critical architectural benefits to OpenShift clusters:

  • Kernel-Level Security Rules: It translates standard Kubernetes NetworkPolicy manifests directly into optimized access control lists (ACLs) processed inside the OVS kernel space, blocking unauthorized lateral attacks instantly.
  • Native Egress IPs: It allows administrators to assign static, public IP addresses to specific application namespaces. This ensures traffic leaving the cluster for external corporate databases carries a predictable, whitelisable signature.
  • Hybrid-Cloud Mesh Integration: It includes built-in hooks for tools like Submariner, allowing an OpenShift cluster running on-premises to build direct, encrypted VPN tunnels over OVS to a separate OpenShift cluster running in public clouds like AWS or Azure, establishing seamless multi-cluster pod-to-pod communication.

4. Advanced Pattern: Multi-NIC Isolation via Multus CNI

By default, the Kubernetes architecture mandates that a pod can only have one network interface hooked into the central CNI overlay network. However, telco systems, specialized databases, and legacy modernization frameworks often require containers to talk to multiple entirely distinct physical network zones.

OpenShift and advanced Kubernetes grids solve this using Multus CNI.

Multus is a meta-plugin—a CNI provider that acts as a wrapper around other CNI plugins.

When a pod launches with a Multus definition, Multus calls the primary cluster CNI (like OVNKubernetes) to establish the mandatory eth0 network for standard cluster communication.

Simultaneously, it reads custom network attachment definitions to spin up secondary interfaces (net1, net2) using plugins like SR-IOV or Macvlan, mapping the container directly into low-latency storage fabrics or isolated corporate VLANs.

Introducing Gordon: The AI Agent for Docker Users

techhadoop ai, docker

Gordon is Docker’s official built-in AI agent.

Instead of being a generic AI assistant (like ChatGPT or Claude) where you have to manually copy and paste error logs, Gordon is purpose-built to understand your actual containerized environment. It is integrated directly into Docker Desktop and the Docker CLI.

Here is a quick overview of what Gordon does and how to use it:

1. What Makes Gordon Different?

Most AI assistants guess based on what you paste. Gordon has direct, secure access to your system (with your approval) to check the real state of your machine:

  • Real Context: It can read your running containers, inspect logs, look at your docker-compose.yml files, and analyze your local project directories.
  • Action-Oriented: If a container crashes with a vague exit code, Gordon doesn’t just explain what the code means; it can look at the logs, pinpoint the issue, rewrite the Dockerfile, and offer a button for you to approve the fix.
  • Approval First: Gordon will never execute a command, modify a file, or stop a container without showing you the exact command first and waiting for your permission.

2. Main Capabilities

  • Debugging: “My container exited with code 137, why?” Gordon will check your local memory limits, see if it was an Out-Of-Memory (OOM) error, and propose a fix.
  • Containerization: “Make this Node.js app run in Docker.” Gordon will scan your directory, detect dependencies, and write optimized multi-stage Dockerfiles.
  • Optimization & Security: It can review your active Docker files and recommend ways to reduce image sizes or implement security health checks.

3. How to Use It

  • In Docker Desktop: You will see the Gordon icon in the left sidebar. You can open it, select your project folder, and start chatting.
  • In the Terminal: You can interact with it directly by running:Bashdocker ai This opens an interactive terminal assistant that lets you ask questions like “Why is my container unhealthy?” or “Clean up my unused volumes”, giving you commands you can hit y to approve and execute immediately.

Optimize Your Server: Understand Hourly CPU Spikes

techhadoop docker, Grafana, monitoring

This is an updated, 24-hour historical view of your Linux server using the same Node Exporter Full dashboard template.

Looking at this timeline compared to your first screenshot, the server behaves completely differently over a 24-hour window. This dashboard tells a clear story: The server is running a recurring, hyper-aggressive batch job.

The Main Takeaway: A Periodic “Sawtooth” Spiking Pattern

Instead of being permanently pinned at 95% CPU like before, your server is currently relaxed at 1.9% CPU Busy and a 1.3% Sys Load.

However, looking at the historical graphs, the server experiences aggressive, identical resource spikes every single hour, on the hour (e.g., exactly at 18:00, 19:00, 20:00, 21:00, etc.). The load shoots up to 90%+ CPU, stays pinned for roughly 15 to 20 minutes, and then drops back down to zero.

Detailed Metric Breakdown

1. The Real-Time Gauges (Current State)
  • CPU Busy (1.9%) & Sys Load (1.3%): Right now, the server is completely idling. The CPU queue is empty.
  • RAM Used (14.0%): Memory consumption is flat, stable, and completely safe at roughly 3.2 GiB used out of 23 GiB.
  • Root FS Used (33.0%): Disk capacity is unchanging and perfectly fine.

2. The 24-Hour Historical Patterns

  • CPU Basic: The spikes are entirely solid blue blocks, meaning they are driven by Busy User space applications. This is exactly what those authentik-worker containers running Celery processes look like when they fire off a major scheduled task.
  • Network Traffic Basic: Notice the perfect synchronization between network activity and CPU. Every single hour, network traffic dramatically pulses (transmitting/receiving tens of megabits per second) at the exact same millisecond the CPU usage maxes out.
Diagnosis: This is a Scheduled Synchronization Task

This behavior confirms that your Authentik stack is running a heavy, automated background job. Because it happens precisely every hour, this isn’t random user traffic.

This pattern is typical of an hourly directory synchronization. Authentik is waking up every 60 minutes, querying an external provider (like an active directory, LDAP server, SCIM endpoint, or massive database), pulling down a large volume of network data, processing the differences, writing to Postgres, and then going back to sleep.

Optimization Tips

While the server is handling it fine right now because it has down-time to recover between spikes, you can optimize this so it doesn’t slam your CPU 24 times a day:

  1. Increase the Sync Interval: If you don’t need real-time user updates, change the background task sync interval in your Authentik provider settings from 1 hour to 6 hours, 12 hours, or have it run once a day at midnight.
  2. Implement Throttling / Batch Size Limits: If your LDAP/Active Directory has thousands of objects, check if you can reduce the pagination page size in Authentik so the worker processes smaller chunks over a longer period, flattening those sharp blue spikes.

The 4 Main Docker Network Drivers and Their Use Cases

techhadoop docker

At its core, Docker Networking is the mechanism that allows your containers to talk to each other, to the host machine, and to the outside world (the internet or your local corporate network).

Because containers are isolated environments, they don’t automatically share a network stack. Docker uses Linux kernel primitives (like network namespaces, virtual ethernet pairs, and iptables rules) to create virtual networks.

How It Works Under the Hood

When you start a container, Docker isolates it by giving it its own private Network Namespace. To connect this isolated space to the rest of the world, Docker creates a virtual wire called a veth pair (virtual ethernet).

One end of this wire is plugged into the container (usually showing up as eth0), and the other end is plugged into a virtual switch on the host machine (the Docker bridge).

The 4 Main Native Network Drivers

When you deploy containers, you can choose how isolated or exposed they are by changing the network driver.

1. Bridge Network (The Default)
  • What it is: A private virtual network created by Docker on the host machine.
  • How it works: Containers attached to the same bridge network get their own internal private IP addresses (e.g., 172.17.0.x) and can talk to each other seamlessly. To expose them to the outside world, you must map ports (e.g., -p 8080:80).
  • Best for: Standard application stacks (like an Nginx container talking to a Node.js container on the same host).
2. Host Network (--net=host)
  • What it is: Complete removal of network isolation between the container and the host.
  • How it works: The container shares the host’s network stack directly. If your container opens port 80, it opens port 80 on the physical machine immediately. There are no virtual IPs or port mappings.
  • Best for: System monitoring tools (like Node Exporter, which we discussed earlier) or ultra-high-performance applications where you cannot afford the tiny speed penalty of Docker’s virtual routing layer.
3. None Network (--net=none)
  • What it is: Absolute network isolation.
  • How it works: The container is spawned with its own network loopback interface (127.0.0.1), but it has no external network interfaces, no IP address, and no access to the internet or other containers.
  • Best for: Running secure batch processing jobs, calculation scripts, or cron tasks that do not require network access and shouldn’t be exposed to security vulnerabilities.
4. Overlay Network
  • What it is: A multi-host network driver.
  • How it works: It creates a distributed network across multiple physical servers. It allows containers running on Server A to talk securely to containers running on Server B as if they were on the exact same machine, handling the encryption and encapsulation automatically.
  • Best for: Docker Swarm clusters or production microservices distributed over several bare-metal/cloud nodes.

Docker Compose & Embedded DNS Discovery

One of the best features of Docker’s custom networking is Service Discovery.

When you create a docker-compose.yml file, Docker Compose automatically creates a dedicated custom Bridge Network just for that stack. Inside this custom network, Docker runs a tiny embedded DNS server at the IP address 127.0.0.11.

Instead of hardcoding fragile IP addresses into your application configs, your containers can talk to each other using their Service Names:

YAML

version: '3.8'
services:
  web:
    image: nginx
    networks:
      - app-net
  db:
    image: postgres
    networks:
      - app-net
networks:
  app-net:
    driver: bridge

Because they are on the same custom network, the web container can reach the database simply by connecting to http://db:5432. Docker’s internal DNS dynamically maps the name db to whatever private IP the Postgres container holds at that moment.

Essential Docker Networking Troubleshooting Commands

techhadoop docker

Here is a collection of essential Docker network troubleshooting commands and techniques, ranging from quick inspect tricks to advanced network namespace debugging.

Level 1: Quick Layout Auditing

Before diving into a container, use the host CLI to map out your subnets and see who is talking to what.

1. Map Every Container to Its IP Address

Instead of looking through pages of JSON from docker network inspect, use a Go template format string to instantly spit out a list of all custom networks, the containers attached to them, and their internal IP addresses:

Bash

docker network ls -q | xargs docker network inspect --format \
'{{range .}}{{ .Name }}:{{range .Containers}}  {{.Name}} -> {{.IPv4Address}}{{end}}{{end}}'
2. Trace a Container’s Network Configuration

If you have a specific container that can’t connect out, run this to see exactly which virtual Docker bridge networks it belongs to and what gateway it is using to reach the internet:

Bash

docker inspect <container_name> --format \
'{{range $net, $cfg := .NetworkSettings.Networks}}{{$net}} -> IP: {{$cfg.IPAddress}} | GW: {{$cfg.Gateway}}{{"\n"}}{{end}}'

Level 2: Runtime Network Probing (The Sandbox Way)

Containers stripped of standard network tools (like ping, curl, or nslookup) make troubleshooting internal DNS tricky. Instead of modifying your production application container to install tools, spin up a temporary diagnostics container inside your exact application network stack.

1. Test Internal Embedded DNS Resolution

Docker Compose automatically sets up an internal DNS server at 127.0.0.11 for service discovery. If web-app can’t reach your postgres database container by its service name, spin up an alpine diagnostic container on that network to test resolution:

Bash

docker run --rm -it --network=<your_compose_network_name> alpine nslookup postgres
2. Verify Port Connectivity

If DNS resolves correctly but connections still timeout, test if the application port is actually listening and accepting traffic using nc (Netcat):

Bash

docker run --rm -it --network=<your_compose_network_name> alpine nc -zv postgres 5432
  • Success response: postgres (172.18.0.3:5432) open
  • Failure response: Connection refused (The container is up, but the app inside isn’t running or listening on that port).

Level 3: Deep Host-Level Troubleshooting (SRE Mode)

Sometimes Docker commands lie, or the issue is rooted in the host Linux kernel’s routing tables and iptables rules.

1. Access a Container’s Network Namespace Directly From the Host

Every Docker container runs in an isolated Linux network namespace (netns). You don’t actually need to enter a container to run network diagnostics on it; you can force your host’s native tools (like ip addr or ss) to run inside the container’s private sandbox.

First, get the Process ID (PID) of your stuck container:

Bash

PID=$(docker inspect --format '{{.State.Pid}}' <container_name>)

Now, pass that PID to the host’s nsenter command to view the container’s internal interfaces and socket status using host-level utilities:

Bash

# View actual internal interfaces from the host
sudo nsenter -t $PID -n ip addr
# View real-time active connections and listening ports inside the container
sudo nsenter -t $PID -n ss -tulpn

2. Capture Traffic with tcpdump

If you suspect an application protocol error or dropping packets between containers, you can attach tcpdump directly to the container’s network stack from the host interface using nsenter:

Bash

sudo nsenter -t $PID -n tcpdump -i any port 80 -vv

(This allows you to sniff live HTTP traffic running through a container without having to install a packet capture tool inside the container itself).

Common Multi-Server Pitfall: Subnet Overlaps

When managing 20+ servers on an enterprise or on-premise network, a common point of failure occurs when a Docker Compose auto-generated bridge network clashes with your physical company Wi-Fi/LAN subnets.

If Docker auto-assigns a pool like 192.168.1.0/24 to a local compose stack, and your central database server lives on that same subnet in your physical data center, your containers will never be able to reach that physical database—they will think the traffic belongs inside the host.

The Fix: Enforce Strict IPAM in Compose

For production servers, explicitly declare your subnets in your docker-compose.yml to prevent accidental overlaps with corporate infrastructure:

YAML

networks:
  app-tier:
    driver: bridge
    ipam:
      config:
        - subnet: 172.28.10.0/24
          gateway: 172.28.10.1

Optimizing Docker Logs with Loki and Grafana

techhadoop ansible, docker, loki, monitoring

Implementing Loki across 20+ bare-metal or VM servers running Docker Compose requires an architectural choice. You have two main approaches to collect logs and forward them to a central Loki instance:

  1. The Modern Agent Approach (Grafana Alloy via Docker Bind-Mounts): Run Grafana Alloy as a standalone container on each of your 20 servers. It automatically sniffs out the log files from /var/lib/docker/containers/ and ships them to Loki. (Recommended for stability and filtering power).
  2. The Plugin Approach (Loki Docker Driver): Install a Docker plugin on all 20 hosts to direct the Docker engine itself to stream logs directly to Loki over HTTP.

Here is the step-by-step blueprint for Approach 1 (Grafana Alloy), which is the most reliable production setup because it won’t crash or lock up your Docker daemon if your central Loki server goes down temporarily.

Step 1: Set Up the Central Loki + Grafana Server

Pick one of your 20 servers (or a dedicated monitoring server) to act as your Central Observability Hub.

Create a docker-compose.yml file on that central machine:

YAML

version: "3.8"
services:
  loki:
    image: grafana/loki:3.0.0
    ports:
      - "3100:3100"
    volumes:
      - loki-data:/loki
    command: -config.file=/etc/loki/local-config.yaml
    restart: unless-stopped
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    volumes:
      - grafana-data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=your_secure_password
    restart: unless-stopped
volumes:
  loki-data:
  grafana-data:

Run docker compose up -d on this central server. Ensure port 3100 is reachable over your internal network from the other 19 servers.

Step 2: Deploy Grafana Alloy on the 19 Remote Servers

On each of your other servers, you will deploy a tiny Grafana Alloy container. Its job is to read the standard JSON logs Docker naturally writes to the host disk.

1. Create the Alloy Config File

Create a directory /etc/alloy/ on the host, and save the following as config.alloy. This configuration file auto-discovers all running Docker containers and forwards their streams to your central Loki instance.

Code snippet

// Discover all local running Docker containers
discovery.docker "local_containers" {
  host = "unix:///var/run/docker.sock"
}
// Read the log files of those discovered containers
loki.source.docker "docker_logs" {
  targets    = discovery.docker.local_containers.targets
  forward_to = [loki.write.central_loki.receiver]
}
// Send the logs to your central Loki instance
loki.write "central_loki" {
  endpoint {
    url = "http://<CENTRAL_LOKI_IP>:3100/loki/api/v1/push"
  }
}

(Replace <CENTRAL_LOKI_IP> with the actual internal network IP of your central monitoring server).

2. Launch Alloy via Docker Compose

On each server, deploy Alloy using this lightweight compose file. It maps the host’s Docker socket and log paths into the Alloy container:

YAML

version: "3.8"
services:
  alloy:
    image: grafana/alloy:latest
    container_name: alloy-logger
    volumes:
      - /etc/alloy/config.alloy:/etc/alloy/config.alloy:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
    command: run --storage.path=/var/lib/alloy/data /etc/alloy/config.alloy
    network_mode: host
    restart: unless-stopped

Run docker compose up -d. Alloy will immediately start streaming logs to your central hub.

Step 3: View the Logs in Grafana

  1. Open your browser and head to http://<CENTRAL_SERVER_IP>:3000 (Log in using the admin password you set in Step 1).
  2. Navigate to Connections -> Data Sources and click Add data source.
  3. Select Loki.
  4. In the URL field, type http://loki:3100 (or http://localhost:3100) and click Save & test.
  5. Go to the Explore tab on the left sidebar, select your Loki data source, and use the label browser to query your logs:

Code snippet

# Find logs from a specific container name across your fleet
{container_name="authentik-worker"}

Pro-Tips for Managing 20+ Servers

  • Automate with Ansible: Don’t configure 20 servers manually! Write a quick Ansible playbook to drop the config.alloy and docker-compose.yml onto all 19 target machines and run docker compose up -d.
  • Add a host Label: If you want to differentiate which logs came from which physical server, add an environment variable to Alloy or adjust the config.alloy file to inject a static host="server-01" metric label.
  • Keep json-file Rotation Enabled: Because Alloy reads the raw logs from /var/lib/docker/containers, make sure your existing application compose files don’t have log rotation disabled. Set a global limit in /etc/docker/daemon.json on your hosts to prevent logs from filling up local disks:JSON{ "log-driver": "json-file", "log-opts": { "max-size": "50m", "max-file": "3" } }

Fixing High CPU Usage: Analyzing `htop` Results

techhadoop docker, Grafana, monitoring

This is a screenshot of htop, an interactive process viewer and system monitor for Linux. It perfectly confirms our theory from the Grafana dashboard: your server’s CPU is maxed out, and we now know exactly what application is causing it.

Here is the breakdown of what this screen is telling you:

Top Section: System Resource Usage

  • CPUs (0, 1, 2, 3): These represent your 4 CPU cores. All of them are heavily saturated, running between 87.4% and 96.0%. The red bars at the very end of the green rows represent kernel/system processing overhead, but the overwhelming green color means the CPU is busy handling user-space tasks.
  • Load Average (4.15 2.03 1.34): This tracks the average system load over the last 1, 5, and 15 minutes. Because you have 4 cores, a load of 4.0 means your CPU is at 100% capacity. The 1-minute average is 4.15, meaning the CPU queue is fully saturated and slightly overflowing right now. However, looking at the 15-minute average (1.34), we can see this heavy load spike is relatively recent.
  • Mem (3.71G/23.5G): Just like Grafana showed, RAM usage is perfectly fine—only about 3.71 GB out of 23.5 GB is being used.
  • Swp (0K/8.00G): Swap memory is completely untouched (0K), meaning there’s zero memory pressure.

Bottom Section: The Culprit Processes

The table lists the active processes sorted by CPU consumption.

  • The Culprit: Celery (celeryd)Almost all the top CPU-consuming processes are Celery ForkPoolWorkers (e.g., PID 4191, 4193, 4194, 3260, etc.). Celery is an asynchronous task/job queue system commonly used in Python applications.
  • What App is this? Further down the list, you can see postgres: authentik and gunicorn: worker [authentik.root.asgi:application]. This means the server is running Authentik—an open-source identity provider/authentication platform.
  • What are they doing? Each of these Celery workers is actively chewing through roughly 45% to 65% of an individual CPU core (indicated under the CPU% column). Under the S (State) column, they are marked as R (Running), meaning they are actively crunching data, not waiting or sleeping (S).

Summary & What to do Next

Your Authentik deployment is currently running heavy background tasks via Celery, which is maxing out all 4 of your CPU cores. This lines up with the network traffic spike we saw earlier—Authentik is likely processing a massive batch of user syncs, LDAP/Active Directory synchronizations, or a sudden flood of login/outbound API requests.

This is the output of the docker stats command, which provides a live data stream for your running Docker containers.

It acts as the final piece of the puzzle. We started with Grafana showing high CPU and network traffic, moved to htop showing Celery workers burning CPU, and now we can see exactly which containers are responsible and how much data they are churning through.

The Primary Culprits: Authentik Workers

Look closely at these three containers:

  • authentik-worker-1 (ID: b82523a11915): 118.72% CPU
  • authentik-worker2-1 (ID: 00d42ac8a162): 115.24% CPU
  • authentik-worker3-1 (ID: 03aecaf9ef51): 101.79% CPU

💡 Note on CPU %: In Docker, 100% means one full CPU core is completely saturated. Because your server has 4 cores, the maximum total CPU available is 400%. These three workers combined are consuming roughly 335% out of 400%, leaving almost nothing for the rest of the system.

The Smoking Gun: Massive Network I/O

If you look at the NET I/O column for these three workers, they have processed an astronomical amount of network data:

  • They have received (~2.8 TB each) and transmitted (~90 GB to 117 GB each) of data over their lifespans.
  • By comparison, your main web server application container (authentik-server-1) has only handled around 1.54 GB. This proves that the workers are doing some massive, continuous background data processing.

Supporting Evidence: The Database

  • authentik-postgresql-1: Running at 41.61% CPU.This is moderately high for a database and means it is working hard to keep up with the constant read/write queries being thrown at it by those three overloaded Celery workers.
  • dokploy-... containers: Dokploy (your deployment/panel manager) and its internal database/redis/traefik containers are completely idling (0% to 0.25% CPU), meaning this issue is strictly isolated to the Authentik stack.

What is causing this in Authentik?

Given that the workers have pulled terabytes of data, this isn’t a normal login event. This behavior is classic for one of two scenarios in Authentik:

  1. LDAP / Active Directory Sync Loop: If you have an LDAP or Active Directory provider configured, it might be caught in an aggressive sync loop, constantly downloading your entire directory over and over again.
  2. Outbound Sync / User Provisioning: A configured connection to an external application (like SCIM syncing to Slack, Google Workspace, or Microsoft 365) might be failing and continuously retrying a massive user sync.

How to Fix It Right Now

  1. Restart the Workers to clear the immediate load:Run this command on your server to give the CPU some breathing room:Bashdocker restart authentik-worker-1 authentik-worker2-1 authentik-worker3-1
  2. Check the logs immediately after restarting:See what task they pick back up. Run:Bashdocker logs --tail 100 -f authentik-worker-1
  3. Log into the Authentik UI:Go to Directory -> Providers or System -> Background Tasks and check for any LDAP or Sync tasks that are currently running, stuck, or failing frequently. If you find one, increase its sync interval (e.g., change it from every few minutes to once an hour or once a day).
  4. Look for any active sync tasks (like an LDAP/Active Directory provider sync) or heavy background jobs that kicked off recently and see if they can be optimized, throttled, or rescheduled for off-peak hours.

Safe vs. Risky: Node Exporter Docker Commands Compared

techhadoop docker, monitoring

Diff on those two commands :

docker run -d \
  –name=node-exporter \
  –restart=always \
  –net=”host” \
  –pid=”host” \
  -v “/:/host:ro” \
  –log-driver json-file \
  –log-opt max-size=10m \
  –log-opt max-file=45 \
  quay.io/prometheus/node-exporter:latest \
  –path.rootfs=/host

docker run -d   \
–name=node-exporter \   
–restart=always  \
–net=”host”  \
–pid=”host”  \ 
-v “/:/host:ro”   quay.io/prometheus/node-exporter:latest   –path.rootfs=/host

Both commands are designed to spin up the Prometheus Node Exporter to monitor your host machine’s physical hardware. While they look nearly identical, the first command is production-ready, while the second command has a hidden trap that can eventually crash your server.

Here is the exact breakdown of the two major differences between them.

Difference 1: Log Rotation (The Crucial Difference)

The primary difference lies in these three lines present only in the first command:

Bash

--log-driver json-file \
--log-opt max-size=10m \
--log-opt max-file=45 \
The First Command (Safe & Controlled)

This explicitly configures Docker’s logging mechanisms. It tells Docker:

  • Treat container logs as standard JSON files.
  • max-size=10m: Once Node Exporter’s log file reaches 10 Megabytes, roll it over and start a new one.
  • max-file=45: Keep a maximum of 45 old log files.

This sets a strict upper bound on how much space Node Exporter’s logs can take up on your host: $45 \times 10\text{ MB} = 450\text{ MB}$ maximum.

The Second Command (The Hidden Danger)

Because it omits these lines, it uses your system’s default Docker logging behavior. By default, Docker does not limit log file sizes.

If Node Exporter encounters a continuous warning or error (for example, if a specific disk partition becomes unreadable), it will spam logs rapidly. Without a cap, that single log file will grow to 10 GB, 50 GB, or more, until it completely consumes your host’s hard drive and locks up the server.

Difference 2: Readability and Syntax

The second difference comes down to formatting and execution safety.

  • First Command: Uses the backslash (\) at the end of every line. This tells the Linux terminal, “This command isn’t finished yet, continue reading on the next line.” It makes the script highly readable and easy to modify.
  • Second Command: Missing the backslashes. If you attempt to copy and paste the second block exactly as it is typed into a standard Linux terminal, it will break. The terminal will try to execute docker run -d by itself on line 1, fail because it has no image name, and then treat the remaining lines as separate, unrecognized commands.

Summary Matrix

FeatureCommand 1 (With Log Options)Command 2 (Without Log Options)
Maximum Disk Used by LogsCapped at 450 MBInfinite (Will grow until disk is full)
Log Files Auto-Cleaned?Yes, old logs are purged automatically.No, requires manual intervention or a global Docker daemon config.
Terminal Copy-Paste ReadyYes (Clean line breaks via \).No (Will fail unless flattened into a single line).
Which one should you use?

Always use the first command. When you are managing + 20 different production servers, you cannot afford to have a single chatty container quietly fill up a local hard drive. The first command ensures predictable storage safety.

Explain docker stats output

docker run command is 100% correct for a production Node Exporter setup (--net="host", --pid="host", etc. are all perfectly placed).

The reason it showed up with a 3.824GiB limit in docker stats is a harmless quirk of how DDEV or Docker calculates default fallbacks when using host networking.

Here is exactly why it happened and why you don’t need to worry about it:

The “Host Network” Illusion

When you pass --net="host" to a container, you are telling it: “Do not create an isolated network sandbox. Use the host’s actual network stack directly.”

Because of this, when you run docker stats, the Docker daemon gets a bit confused about the boundaries of that container. In certain environments (especially management setups like DDEV or Dokploy where default resource profiles or parent cgroups are enforced across the docker daemon), Docker will simply mirror the memory limit of the host or the surrounding project cluster onto any container running in host network mode.

How to Verify It’s actually fine

You can prove that your Node Exporter isn’t actually being restricted or grouped into a restricted application pool by running this command on your host:

root@cvm-srv52:~# docker inspect node-exporter | grep -i memory
“Memory”: 0,
“MemoryReservation”: 0,
“MemorySwap”: 0,
“MemorySwappiness”: null,

You will see "Memory": 0. A value of 0 in Docker means unlimited—it has full access to the bare-metal server’s resource pool.

Verdict

Your command is great. The 3.824GiB limit you saw in docker stats is just a visual reporting artifact because of --net="host". You can safely leave it exactly as it is!