monitoring

Optimize Your Server: Understand Hourly CPU Spikes

techhadoop docker, Grafana, monitoring

This is an updated, 24-hour historical view of your Linux server using the same Node Exporter Full dashboard template.

Looking at this timeline compared to your first screenshot, the server behaves completely differently over a 24-hour window. This dashboard tells a clear story: The server is running a recurring, hyper-aggressive batch job.

The Main Takeaway: A Periodic “Sawtooth” Spiking Pattern

Instead of being permanently pinned at 95% CPU like before, your server is currently relaxed at 1.9% CPU Busy and a 1.3% Sys Load.

However, looking at the historical graphs, the server experiences aggressive, identical resource spikes every single hour, on the hour (e.g., exactly at 18:00, 19:00, 20:00, 21:00, etc.). The load shoots up to 90%+ CPU, stays pinned for roughly 15 to 20 minutes, and then drops back down to zero.

Detailed Metric Breakdown

1. The Real-Time Gauges (Current State)
  • CPU Busy (1.9%) & Sys Load (1.3%): Right now, the server is completely idling. The CPU queue is empty.
  • RAM Used (14.0%): Memory consumption is flat, stable, and completely safe at roughly 3.2 GiB used out of 23 GiB.
  • Root FS Used (33.0%): Disk capacity is unchanging and perfectly fine.

2. The 24-Hour Historical Patterns

  • CPU Basic: The spikes are entirely solid blue blocks, meaning they are driven by Busy User space applications. This is exactly what those authentik-worker containers running Celery processes look like when they fire off a major scheduled task.
  • Network Traffic Basic: Notice the perfect synchronization between network activity and CPU. Every single hour, network traffic dramatically pulses (transmitting/receiving tens of megabits per second) at the exact same millisecond the CPU usage maxes out.
Diagnosis: This is a Scheduled Synchronization Task

This behavior confirms that your Authentik stack is running a heavy, automated background job. Because it happens precisely every hour, this isn’t random user traffic.

This pattern is typical of an hourly directory synchronization. Authentik is waking up every 60 minutes, querying an external provider (like an active directory, LDAP server, SCIM endpoint, or massive database), pulling down a large volume of network data, processing the differences, writing to Postgres, and then going back to sleep.

Optimization Tips

While the server is handling it fine right now because it has down-time to recover between spikes, you can optimize this so it doesn’t slam your CPU 24 times a day:

  1. Increase the Sync Interval: If you don’t need real-time user updates, change the background task sync interval in your Authentik provider settings from 1 hour to 6 hours, 12 hours, or have it run once a day at midnight.
  2. Implement Throttling / Batch Size Limits: If your LDAP/Active Directory has thousands of objects, check if you can reduce the pagination page size in Authentik so the worker processes smaller chunks over a longer period, flattening those sharp blue spikes.

Optimizing Docker Logs with Loki and Grafana

techhadoop ansible, docker, loki, monitoring

Implementing Loki across 20+ bare-metal or VM servers running Docker Compose requires an architectural choice. You have two main approaches to collect logs and forward them to a central Loki instance:

  1. The Modern Agent Approach (Grafana Alloy via Docker Bind-Mounts): Run Grafana Alloy as a standalone container on each of your 20 servers. It automatically sniffs out the log files from /var/lib/docker/containers/ and ships them to Loki. (Recommended for stability and filtering power).
  2. The Plugin Approach (Loki Docker Driver): Install a Docker plugin on all 20 hosts to direct the Docker engine itself to stream logs directly to Loki over HTTP.

Here is the step-by-step blueprint for Approach 1 (Grafana Alloy), which is the most reliable production setup because it won’t crash or lock up your Docker daemon if your central Loki server goes down temporarily.

Step 1: Set Up the Central Loki + Grafana Server

Pick one of your 20 servers (or a dedicated monitoring server) to act as your Central Observability Hub.

Create a docker-compose.yml file on that central machine:

YAML

version: "3.8"
services:
  loki:
    image: grafana/loki:3.0.0
    ports:
      - "3100:3100"
    volumes:
      - loki-data:/loki
    command: -config.file=/etc/loki/local-config.yaml
    restart: unless-stopped
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    volumes:
      - grafana-data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=your_secure_password
    restart: unless-stopped
volumes:
  loki-data:
  grafana-data:

Run docker compose up -d on this central server. Ensure port 3100 is reachable over your internal network from the other 19 servers.

Step 2: Deploy Grafana Alloy on the 19 Remote Servers

On each of your other servers, you will deploy a tiny Grafana Alloy container. Its job is to read the standard JSON logs Docker naturally writes to the host disk.

1. Create the Alloy Config File

Create a directory /etc/alloy/ on the host, and save the following as config.alloy. This configuration file auto-discovers all running Docker containers and forwards their streams to your central Loki instance.

Code snippet

// Discover all local running Docker containers
discovery.docker "local_containers" {
  host = "unix:///var/run/docker.sock"
}
// Read the log files of those discovered containers
loki.source.docker "docker_logs" {
  targets    = discovery.docker.local_containers.targets
  forward_to = [loki.write.central_loki.receiver]
}
// Send the logs to your central Loki instance
loki.write "central_loki" {
  endpoint {
    url = "http://<CENTRAL_LOKI_IP>:3100/loki/api/v1/push"
  }
}

(Replace <CENTRAL_LOKI_IP> with the actual internal network IP of your central monitoring server).

2. Launch Alloy via Docker Compose

On each server, deploy Alloy using this lightweight compose file. It maps the host’s Docker socket and log paths into the Alloy container:

YAML

version: "3.8"
services:
  alloy:
    image: grafana/alloy:latest
    container_name: alloy-logger
    volumes:
      - /etc/alloy/config.alloy:/etc/alloy/config.alloy:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
    command: run --storage.path=/var/lib/alloy/data /etc/alloy/config.alloy
    network_mode: host
    restart: unless-stopped

Run docker compose up -d. Alloy will immediately start streaming logs to your central hub.

Step 3: View the Logs in Grafana

  1. Open your browser and head to http://<CENTRAL_SERVER_IP>:3000 (Log in using the admin password you set in Step 1).
  2. Navigate to Connections -> Data Sources and click Add data source.
  3. Select Loki.
  4. In the URL field, type http://loki:3100 (or http://localhost:3100) and click Save & test.
  5. Go to the Explore tab on the left sidebar, select your Loki data source, and use the label browser to query your logs:

Code snippet

# Find logs from a specific container name across your fleet
{container_name="authentik-worker"}

Pro-Tips for Managing 20+ Servers

  • Automate with Ansible: Don’t configure 20 servers manually! Write a quick Ansible playbook to drop the config.alloy and docker-compose.yml onto all 19 target machines and run docker compose up -d.
  • Add a host Label: If you want to differentiate which logs came from which physical server, add an environment variable to Alloy or adjust the config.alloy file to inject a static host="server-01" metric label.
  • Keep json-file Rotation Enabled: Because Alloy reads the raw logs from /var/lib/docker/containers, make sure your existing application compose files don’t have log rotation disabled. Set a global limit in /etc/docker/daemon.json on your hosts to prevent logs from filling up local disks:JSON{ "log-driver": "json-file", "log-opts": { "max-size": "50m", "max-file": "3" } }

Why Switch from ELK to Grafana Loki for Kubernetes Logging?

techhadoop Grafana, kubernetes, monitoring

The title says it all. If you are running Kubernetes at scale, moving away from the traditional ELK (Elasticsearch, Logstash, Kibana) stack—or even its modern OpenSearch equivalent—is one of the biggest infrastructure wins you can achieve.

Here is a simplified architectural breakdown of why the combination of Grafana Alloy + Grafana Loki + Cloud Object Storage (S3) has become the modern standard for high-efficiency cloud-native logging.

The Structural Flaw of ELK: The “Index Everything” Penalty

To understand why the Loki stack wins, you have to look at how Elasticsearch works.

Elasticsearch is fundamentally a full-text search engine. When a log line comes in, Logstash parses it, and Elasticsearch splits the text into tokens and builds a massive, complex inverted index (similar to the index at the back of a massive textbook).

The Problem at Scale:
  • Storage Bloat: The index itself can often take up as much disk space as—or more than—the actual raw log data.
  • RAM Hunger: To search that index quickly, Elasticsearch must keep massive portions of it cached in memory. As your cluster grows, your JVM heap requirements skyrocket.
  • The High-SSD Tax: You are forced to run Elasticsearch on expensive, high-speed block storage (like AWS EBS gp3 or NVMe drives) just to keep up with index writes and queries.

The Lean Alternative: Alloy + Loki + S3

Grafana Loki turns the ELK philosophy completely on its head. It is frequently described as “Prometheus, but for logs.”

Instead of parsing and indexing the full text of every log line, Loki only indexes the metadata (labels) attached to the stream—such as kubernetes_pod_name, namespace, or container_name. The actual text of the log line is compressed into raw chunks and sent directly to cheap object storage.

[ Pod Logs ] ──> [ Grafana Alloy ] ──> [ Grafana Loki ] ──> [ Amazon S3 / Object Storage ]
                       │                                             │
                       └───── (Metadata Only Indexed) ───────────────┘
1. Grafana Alloy: The Advanced, Single-Agent Collector

Alloy is Grafana’s modern, OpenTelemetry-compatible collector that replaces older agents like Fluentd, Fluent Bit, or Promtail.

  • It natively auto-discovers Kubernetes pods, extracts their metadata, and forwards the streams.
  • Because it is written in Go and shares components with OpenTelemetry, it consumes an incredibly small CPU and memory footprint on your Kubernetes nodes compared to heavy Logstash instances.
2. Grafana Loki: The Index-Free Engine

Because Loki only indexes labels, its index is microscopic compared to Elasticsearch.

  • Since the index is tiny, it fits completely into RAM, making query routing incredibly fast.
  • Loki doesn’t care if a log line contains a 500 error or a success message; it treats the text as an unindexed blob, eliminating the computational overhead of real-time text parsing during ingestion.
3. S3/Object Storage: The Ultimate Cost Hack

Instead of paying a premium for fast SSDs, Loki batches log chunks and writes them directly to cheap, durable Object Storage (like AWS S3, Google Cloud Storage, or MinIO).

  • You get infinite storage scaling out of the box without ever having to re-shard a database or worry about running out of disk space on a node.

Side-by-Side: ELK vs. Loki Stack

FeatureELK / OpenSearch StackAlloy + Loki + S3 Stack
Indexing PhilosophyFull-text indexing of every log word.Metadata (labels) indexing only.
Storage MediumExpensive SSDs / Block Storage.Ultra-cheap Cloud Object Storage (S3).
Resource ConsumptionHeavy RAM (JVM) and high CPU overhead.Minimal RAM and CPU foot-print.
TCO (Total Cost)High (Scales linearly with log volume).Very Low (Up to 80% cheaper at scale).
The TradeoffLightning-fast ad-hoc text search across billions of lines.Blazing-fast target searches; slower brute-force full-text queries over massive windows.

The Bottom Line: Why it Wins

The Loki architecture accepts a pragmatic tradeoff: It sacrifices raw full-text search speed across historical massive timeframes in exchange for massive operational simplicity and cost reductions.

When debugging Kubernetes applications, engineers rarely need to search the entire infrastructure blindly. They almost always know the namespace, the pod, or the service they are investigating. Because Loki filters by these labels instantly, it pulls the relevant compressed log blocks out of S3 and hands them to the developer in seconds—giving you 95% of the utility of ELK at a fraction of the infrastructure bill.

Understanding Jaeger and Tempo for Distributed Tracing

techhadoop monitoring, OTel

If OpenTelemetry is the camera that takes the pictures (traces), Jaeger and Tempo are the albums where you store them. They are both open-source backends designed for distributed tracing, helping you visualize exactly how a single user request flows across multiple microservices.

However, they are built with completely opposite engineering philosophies.

Jaeger: The Rich, Standalone Heavyweight

Created by Uber and hosted by the CNCF, Jaeger is the older, battle-tested veteran of distributed tracing.

  • How it works: Jaeger loves databases. It is designed to index everything. When you search for a trace, it queries database backends like Elasticsearch, OpenSearch, or Cassandra.
  • The Interface: It has its own dedicated, standalone web UI.
  • Key Strength (Powerful Searching): Because it indexes everything, Jaeger’s search capability is top-tier. You can effortlessly hunt for traces using custom tags, status codes (e.g., http.status_code=500), specific services, or durations right out of the box.
  • The Catch (High Cost): Managing Elasticsearch or Cassandra at a massive scale is an operational nightmare. Indexing every single trace requires huge amounts of RAM and disk space, making Jaeger very expensive to run if you have terabytes of trace volume.

Grafana Tempo: The Cheap, High-Scale Disruptor

Created by Grafana Labs, Tempo is the modern challenger built specifically to fix Jaeger’s massive storage bills.

  • How it works: Tempo is completely index-free. Instead of setting up massive, complex databases, Tempo batches traces together and throws them directly into Object Storage (like Amazon S3, Google Cloud Storage, or Azure Blob).
  • The Interface: It doesn’t have its own UI; it uses Grafana natively.
  • Key Strength (Insanely Cheap & Scalable): Object storage is incredibly cheap. Tempo can run with 10x to 100x lower storage costs than Jaeger because it doesn’t waste resources building heavy search indexes.
  • The Catch (How do you find a trace?): Because there are no indexes, you traditionally couldn’t just “search” Tempo for a random trace attribute easily. Instead, it relies on a “Metrics-to-Logs-to-Traces” workflow. You look at a Prometheus chart, click a spike, find the log line in Grafana Loki, click the trace_id embedded in the log, and Tempo instantly pulls it up via that ID. (Note: Tempo now includes a query language called TraceQL to allow searching, but it still heavily relies on object storage scanning).

Side-by-Side Comparison

FeatureJaegerGrafana Tempo
Primary Storage BackendElasticsearch, Cassandra, OpenSearchAmazon S3, Google Cloud Storage, Azure Blob
Cost at ScaleHigh (Requires heavy compute/RAM for DBs)Extremely Low (Relies on cheap object storage)
User InterfaceStandalone Jaeger UINatively integrated inside Grafana
Search MechanismFull-text, database-indexed searchTraceQL or structural discovery via logs/metrics
EcosystemStandalone CNCF toolTightly bound to the Grafana LGTM stack (Loki/Grafana/Tempo/Mimir)

Which one should you pick?

  • Choose Jaeger if: You want a standalone, rock-solid tracing tool, you have a massive budget for Elasticsearch, or your engineers absolutely need to query traces using complex tag filtering without depending on logs or Grafana.
  • Choose Tempo if: You are already using Grafana, you want to view metrics, logs, and traces side-by-side in a single window, and you want to scale up tracing across millions of requests without destroying your cloud infrastructure budget.

Fixing High CPU Usage: Analyzing `htop` Results

techhadoop docker, Grafana, monitoring

This is a screenshot of htop, an interactive process viewer and system monitor for Linux. It perfectly confirms our theory from the Grafana dashboard: your server’s CPU is maxed out, and we now know exactly what application is causing it.

Here is the breakdown of what this screen is telling you:

Top Section: System Resource Usage

  • CPUs (0, 1, 2, 3): These represent your 4 CPU cores. All of them are heavily saturated, running between 87.4% and 96.0%. The red bars at the very end of the green rows represent kernel/system processing overhead, but the overwhelming green color means the CPU is busy handling user-space tasks.
  • Load Average (4.15 2.03 1.34): This tracks the average system load over the last 1, 5, and 15 minutes. Because you have 4 cores, a load of 4.0 means your CPU is at 100% capacity. The 1-minute average is 4.15, meaning the CPU queue is fully saturated and slightly overflowing right now. However, looking at the 15-minute average (1.34), we can see this heavy load spike is relatively recent.
  • Mem (3.71G/23.5G): Just like Grafana showed, RAM usage is perfectly fine—only about 3.71 GB out of 23.5 GB is being used.
  • Swp (0K/8.00G): Swap memory is completely untouched (0K), meaning there’s zero memory pressure.

Bottom Section: The Culprit Processes

The table lists the active processes sorted by CPU consumption.

  • The Culprit: Celery (celeryd)Almost all the top CPU-consuming processes are Celery ForkPoolWorkers (e.g., PID 4191, 4193, 4194, 3260, etc.). Celery is an asynchronous task/job queue system commonly used in Python applications.
  • What App is this? Further down the list, you can see postgres: authentik and gunicorn: worker [authentik.root.asgi:application]. This means the server is running Authentik—an open-source identity provider/authentication platform.
  • What are they doing? Each of these Celery workers is actively chewing through roughly 45% to 65% of an individual CPU core (indicated under the CPU% column). Under the S (State) column, they are marked as R (Running), meaning they are actively crunching data, not waiting or sleeping (S).

Summary & What to do Next

Your Authentik deployment is currently running heavy background tasks via Celery, which is maxing out all 4 of your CPU cores. This lines up with the network traffic spike we saw earlier—Authentik is likely processing a massive batch of user syncs, LDAP/Active Directory synchronizations, or a sudden flood of login/outbound API requests.

This is the output of the docker stats command, which provides a live data stream for your running Docker containers.

It acts as the final piece of the puzzle. We started with Grafana showing high CPU and network traffic, moved to htop showing Celery workers burning CPU, and now we can see exactly which containers are responsible and how much data they are churning through.

The Primary Culprits: Authentik Workers

Look closely at these three containers:

  • authentik-worker-1 (ID: b82523a11915): 118.72% CPU
  • authentik-worker2-1 (ID: 00d42ac8a162): 115.24% CPU
  • authentik-worker3-1 (ID: 03aecaf9ef51): 101.79% CPU

💡 Note on CPU %: In Docker, 100% means one full CPU core is completely saturated. Because your server has 4 cores, the maximum total CPU available is 400%. These three workers combined are consuming roughly 335% out of 400%, leaving almost nothing for the rest of the system.

The Smoking Gun: Massive Network I/O

If you look at the NET I/O column for these three workers, they have processed an astronomical amount of network data:

  • They have received (~2.8 TB each) and transmitted (~90 GB to 117 GB each) of data over their lifespans.
  • By comparison, your main web server application container (authentik-server-1) has only handled around 1.54 GB. This proves that the workers are doing some massive, continuous background data processing.

Supporting Evidence: The Database

  • authentik-postgresql-1: Running at 41.61% CPU.This is moderately high for a database and means it is working hard to keep up with the constant read/write queries being thrown at it by those three overloaded Celery workers.
  • dokploy-... containers: Dokploy (your deployment/panel manager) and its internal database/redis/traefik containers are completely idling (0% to 0.25% CPU), meaning this issue is strictly isolated to the Authentik stack.

What is causing this in Authentik?

Given that the workers have pulled terabytes of data, this isn’t a normal login event. This behavior is classic for one of two scenarios in Authentik:

  1. LDAP / Active Directory Sync Loop: If you have an LDAP or Active Directory provider configured, it might be caught in an aggressive sync loop, constantly downloading your entire directory over and over again.
  2. Outbound Sync / User Provisioning: A configured connection to an external application (like SCIM syncing to Slack, Google Workspace, or Microsoft 365) might be failing and continuously retrying a massive user sync.

How to Fix It Right Now

  1. Restart the Workers to clear the immediate load:Run this command on your server to give the CPU some breathing room:Bashdocker restart authentik-worker-1 authentik-worker2-1 authentik-worker3-1
  2. Check the logs immediately after restarting:See what task they pick back up. Run:Bashdocker logs --tail 100 -f authentik-worker-1
  3. Log into the Authentik UI:Go to Directory -> Providers or System -> Background Tasks and check for any LDAP or Sync tasks that are currently running, stuck, or failing frequently. If you find one, increase its sync interval (e.g., change it from every few minutes to once an hour or once a day).
  4. Look for any active sync tasks (like an LDAP/Active Directory provider sync) or heavy background jobs that kicked off recently and see if they can be optimized, throttled, or rescheduled for off-peak hours.

Enhance Grafana Alerts with Custom Annotations

techhadoop Grafana, monitoring, Prometheus

Right now alert email is showing only the default Grafana labels:

alertname = Sys Load
instance = 192.168.231.43:9100
grafana_folder = Alerts

You can make the email much more useful by adding custom annotations to the alert rule.

Option 1 – Add Summary and Description

In the alert rule, scroll down to Annotations and add:

Summary
High system load detected on {{ $labels.instance }}
Description
Server: {{ $labels.instance }}
Current Load: {{ printf "%.2f" $values.A.Value }}%
Threshold: 25%
Please investigate CPU utilization, running processes, and system responsiveness.

Then the email will look like:

ALERT: Sys Load
High system load detected on 192.168.231.43:9100
Server: 192.168.231.43:9100
Current Load: 87.45%
Threshold: 25%
Please investigate CPU utilization, running processes, and system responsiveness.

Option 2 – Add Severity Labels

Add labels:

severity = warning
team = infrastructure
environment = production

Then email shows:

severity: warning
team: infrastructure
environment: production
instance: 192.168.231.43:9100

Very useful when you have many alerts.

Option 3 – Include Useful Dashboard Links

Annotation:

Runbook URL
https://wiki.company.com/runbooks/linux-high-load
Dashboard URL
https://grafana.company.com/d/linux-server/linux-server-dashboard

Then engineers can click directly from the email.

Option 4 – Add Hostname Instead of IP

Currently you see:

192.168.231.43:9100

Much better if Prometheus exposes:

labels:
  hostname: cvm-srv44

Then annotation:

Host: {{ $labels.hostname }}
Instance: {{ $labels.instance }}

Email:

Host: cvm-srv44
Instance: 192.168.231.43:9100
Current Load: 87%

Much easier for operations teams.

Option 5 – Enterprise Style Alert

Summary
[{{ $labels.severity | toUpper }}] High System Load on {{ $labels.instance }}
Description
Environment: {{ $labels.environment }}
Server: {{ $labels.instance }}
Current Load: {{ printf "%.2f" $values.A.Value }}%
Alert Threshold: 25%
Recommended Checks:
1. top
2. htop
3. vmstat 1
4. iostat -x 1
5. journalctl -xe
Investigate CPU saturation, runaway processes, or I/O bottlenecks.

Bonus: Add Hostname Automatically in Prometheus

In your prometheus.yml:

scrape_configs:
  - job_name: linux-servers
    static_configs:
      - targets:
          - 192.168.231.43:9100
        labels:
          hostname: cvm-srv44
      - targets:
          - 192.168.231.122:9100
        labels:
          hostname: cvm-srv45

Then use:

Host: {{ $labels.hostname }}

in Grafana alerts.

This is usually the biggest improvement because emails become:

[WARNING] High System Load
Host: cvm-srv44
IP: 192.168.231.43:9100
Current Load: 91.7%
Threshold: 80%

instead of just showing an IP address.

Interview Questions Often Asked for Senior Grafana Architects

techhadoop Grafana, monitoring, Prometheus

1. How do you reduce Prometheus cardinality?

Cardinality means too many unique metric/label combinations.

Example of bad label:

http_requests_total{user_id="12345"}

This creates millions of time series.

To reduce cardinality:

- Avoid dynamic labels: user_id, session_id, request_id, pod_uid
- Keep only useful labels: job, instance, namespace, pod, status_code
- Drop unnecessary labels using relabel_configs
- Use recording rules for expensive queries
- Reduce scrape targets if not needed
- Avoid exposing too many custom metrics
- Set retention limits

Interview answer:

I reduce Prometheus cardinality by controlling labels, dropping high-cardinality labels, reviewing /targets and /tsdb-status, and avoiding dynamic values like user IDs or request IDs in metric labels.

2. Difference between recording rules and alert rules

Recording rule

Pre-calculates a query and saves the result as a new metric.

Example:

- record: node:cpu_usage:avg5m
  expr: 100 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100

Used for:

- Faster dashboards
- Reusable metrics
- Reducing query load
Alert rule

Triggers an alert when a condition is met.

Example:

- alert: HighCPU
  expr: node:cpu_usage:avg5m > 80
  for: 5m

Interview answer:

Recording rules improve performance by precomputing metrics. Alert rules evaluate conditions and trigger notifications.

3. How do you monitor Kubernetes at scale?

Use:

- Prometheus HA
- kube-state-metrics
- node-exporter
- cAdvisor/container metrics
- Alertmanager
- Grafana dashboards
- Loki for logs
- Tempo for traces
- Mimir/Thanos for long-term metrics

For large clusters:

- Use recording rules
- Limit scrape intervals
- Reduce metric cardinality
- Use remote_write to Mimir/Thanos
- Separate platform and application monitoring
- Use namespace/team-based dashboards

Interview answer:

At scale, I avoid a single large Prometheus doing everything. I deploy Prometheus per cluster or per domain, use remote_write to a central backend like Mimir or Thanos, apply cardinality controls, and visualize everything centrally in Grafana.

4. How would you deploy Grafana HA?

Architecture:

Load Balancer
     |
+----+----+
|         |
Grafana  Grafana
|         |
+----+----+
     |
PostgreSQL/MySQL

Key points:

- Multiple Grafana instances
- Shared external database: PostgreSQL or MySQL
- Load balancer in front
- Shared configuration
- SSO integration
- Persistent dashboards stored in DB
- Provision dashboards/data sources using GitOps/IaC

Important:

Do not use SQLite for HA.

Interview answer:

Grafana is mostly stateless. For HA, I run multiple Grafana pods or VMs behind a load balancer and use PostgreSQL as the shared backend database.

5. How would you design LGTM for 1000 servers?

LGTM means:

L = Loki      Logs
G = Grafana   Dashboards
T = Tempo     Traces
M = Mimir     Metrics

Design:

1000 Servers
   |
Node Exporter / Alloy / Promtail / OpenTelemetry Collector
   |
Metrics → Mimir
Logs    → Loki
Traces  → Tempo
   |
Grafana

Important design points:

- Use Grafana Alloy or OpenTelemetry Collector agents
- Use Mimir for scalable metrics
- Use Loki for centralized logs
- Use Tempo for distributed tracing
- Store data in object storage like S3/Azure Blob
- Use retention policies
- Use tenant/team separation
- Use alerting with Alertmanager or Grafana Alerting

Interview answer:

For 1000 servers, I would avoid one single Prometheus scraping everything. I would use agents on servers, remote_write metrics to Mimir, send logs to Loki, traces to Tempo, and use Grafana as the single visualization and alerting layer.

6. How do you troubleshoot missing metrics?

Check in this order:

1. Is the target up?
2. Is Prometheus scraping the endpoint?
3. Is the exporter running?
4. Is the metric exposed on /metrics?
5. Is the scrape config correct?
6. Are relabeling rules dropping it?
7. Is the time range correct in Grafana?
8. Is the PromQL query correct?
9. Is authentication/TLS blocking scrape?
10. Is the metric name changed?

Useful PromQL:

up
scrape_samples_scraped
scrape_duration_seconds

Useful commands:

curl http://server:9100/metrics
kubectl get servicemonitor -A
kubectl get podmonitor -A
kubectl get targets

Interview answer:

I start from the source. First I verify the exporter exposes the metric, then Prometheus target status, scrape configuration, relabeling, and finally Grafana query/time range.

7. How do you secure Grafana in an enterprise environment?

Use:

- HTTPS/TLS
- SSO with SAML/OIDC/LDAP
- MFA through identity provider
- RBAC
- Team-based permissions
- Folder/dashboard permissions
- Disable anonymous access
- Secure admin account
- Use secrets management
- Audit logging
- Network restrictions
- Backup database and dashboards

Also:

- Do not expose Grafana directly to the internet
- Use reverse proxy or ingress with TLS
- Use least privilege for data sources
- Separate dev/test/prod dashboards

Interview answer:

I secure Grafana using enterprise identity integration, RBAC, TLS, least-privilege data source credentials, dashboard permissions, audit logs, and network restrictions.

8. How do you integrate Grafana with OpenShift?

Options:

- Use OpenShift monitoring Prometheus/Thanos as data source
- Connect Grafana to Thanos Querier
- Use ServiceAccount token for authentication
- Import Kubernetes/OpenShift dashboards
- Monitor nodes, pods, namespaces, etcd, API server, ingress, OVN

Common OpenShift components:

- Prometheus
- Alertmanager
- Thanos Querier
- kube-state-metrics
- node-exporter
- Cluster Monitoring Operator

Example data source:

Grafana → Thanos Querier → OpenShift metrics

Interview answer:

In OpenShift, I usually connect Grafana to Thanos Querier or Prometheus using a service account token. Then I build dashboards for cluster health, nodes, pods, namespaces, etcd, API server, ingress, and OVN networking.

9. How do you implement multi-tenancy?

Ways to implement multi-tenancy:

- Separate organizations in Grafana
- Separate folders per team
- Team-based RBAC
- Dashboard permissions
- Data source permissions
- Separate tenants in Mimir/Loki/Tempo
- Separate Kubernetes namespaces
- Separate alert contact points

Example:

Team A → Folder A → Data source tenant A
Team B → Folder B → Data source tenant B

For strong isolation:

- Use separate Grafana organizations
- Use separate Mimir/Loki tenants
- Use SSO groups mapped to Grafana teams

Interview answer:

I implement multi-tenancy using SSO group mapping, teams, folders, RBAC, data source permissions, and backend tenant isolation in Mimir, Loki, or Tempo.

10. How do you monitor AWS, Azure, and OpenShift from one Grafana instance?

Architecture:

AWS CloudWatch
Azure Monitor
OpenShift Thanos/Prometheus
Linux Node Exporter
Kubernetes Metrics
        |
        v
     Grafana

Data sources:

AWS        → CloudWatch
Azure      → Azure Monitor
OpenShift  → Prometheus/Thanos
Logs       → Loki or Elasticsearch
Traces     → Tempo

Best practices:

- Use separate folders per platform
- Use variables: cloud, region, cluster, namespace
- Use RBAC per team
- Use central alerting
- Use standardized dashboards
- Use tags and labels consistently

Interview answer:

I would use Grafana as the central visualization layer with multiple data sources: CloudWatch for AWS, Azure Monitor for Azure, and Prometheus/Thanos for OpenShift. Then I would organize dashboards by platform, region, cluster, and application, with RBAC and alerting policies per team.

Strong closing answer for interview

My approach is to design Grafana as the central observability portal, but not as the only backend. Prometheus handles metrics collection, Loki handles logs, Tempo handles traces, and Mimir or Thanos handles scalable long-term metrics. For enterprise environments, I focus on HA, RBAC, SSO, cardinality control, recording rules, proper alerting, and multi-tenant separation.

Resolving CPU Bottlenecks on Linux Servers with Grafana

techhadoop Grafana, monitoring, Prometheus

Based on the provided Grafana dashboard image (utilizing the popular Node Exporter Full template), here is a breakdown of what is happening on this Linux server (cvm-srv51).

The Main Takeaway: CPU Bottleneck

Your server is currently under a heavy CPU bottleneck, while memory and disk resources are mostly relaxed. Around 12:43 to 12:45, there was a brief dip where the load dropped significantly, but it immediately spiked back up to its previous high levels.

Detailed Metric Breakdown

1. Quick CPU / Mem / Disk (Gauges)
  • CPU Busy (95.2%): Your CPU is running almost at maximum capacity. The red ring indicates this is in a critical threshold.
  • Sys Load (136.3%): This represents a system load average that exceeds the server’s capacity. Since the server has 4 CPU Cores (shown in the info box), a 100% load capacity would mean a load average of 4.0. Running at 136.3% means the CPU queue is constantly backed up, and processes are waiting for CPU time.
  • RAM Used (17.2%) & SWAP Used (0.0%): Out of 23 GiB available, memory usage is very low and healthy. SWAP isn’t being touched, meaning this is strictly a compute problem, not a memory-starvation problem.
  • Root FS Used (32.7%): Storage space is perfectly fine, with plenty of room left on the root filesystem.
2. Historical Graphs (Last 15 Minutes)
  • CPU Basic: The large blue block dominating the graph represents Busy User space. This means the high CPU usage is being driven by user-level applications or services (e.g., a heavy application, database queries, a running script, or containerized apps), rather than kernel overhead (Busy System) or waiting for hardware (Busy Iowait).
  • Memory Basic: A completely flat line showing steady, low memory utilization.
  • Network Traffic Basic: There is a noticeable drop in network traffic that perfectly correlates with the dip in CPU usage around 12:44. When network traffic resumed, CPU usage shot back up. This suggests that incoming network requests or data transfers are likely triggering the high CPU workload.
Next Steps for Troubleshooting

Because the CPU is being hammered by user-space applications linked to network activity, you should log into the server (cvm-srv51) via SSH and investigate:

  1. Run top or htop: Check which specific processes are consuming the most CPU.
  2. Inspect Containers: Since the network graph lists several virtual interfaces (like docker0 and veth...), it’s highly likely this server is running Docker containers. Use docker stats to see which container is pulling all the compute.
  3. Check Application Logs: Match the timeline (the dip at 12:44) against your application or web server logs to see what traffic paused or restarted at that moment.

Analyzing RAM Usage in Linux with Grafana

techhadoop Grafana, linux, monitoring

The Grafana Node Exporter Full dashboard memory breakdown above showing how Linux RAM is being used over time.

Let’s interpret it as a Linux administrator or SRE would.

Total Memory Usage Breakdown
CategoryMean
Applications717 MiB
Cache5.46 GiB
Buffers614 MiB
Slab380 MiB
Unused623 MiB
Everything elseSmall

The first thing to notice:

Most of the memory is being used for Cache and Buffers, not applications.

This is normal Linux behavior.

Apps – Memory used by user-space applications

Min: 632 MiB
Mean: 717 MiB
Max: 1.37 GiB

This is the memory actually consumed by running processes:

Examples:

  • Java
  • Nginx
  • Apache
  • PostgreSQL
  • Docker containers
  • Tomcat
  • Kong Gateway

Your server’s applications are only using about:

~717 MB average

which is very low.

Cache

Mean: 5.46 GiB

Linux uses free RAM to cache file contents.

Examples:

  • Recently read files
  • Database files
  • Log files
  • Container images

Linux philosophy:

Free RAM is wasted RAM.

So Linux fills unused memory with cache.

The good news:

5.46 GiB Cache

can be reclaimed instantly if applications need RAM.

This is NOT memory pressure.

Buffers

Mean: 614 MiB

Buffers are cache for block devices.

Examples:

  • Disk metadata
  • Filesystem operations
  • Read/write operations

This is normal.

Slab

Mean: 380 MiB

Kernel memory used for internal structures:

Examples:

  • inode cache
  • dentry cache
  • network structures
  • process descriptors

For example:

slabtop

might show:

dentry
inode_cache
kmalloc

380 MB is reasonable.

PageTables

Mean: 9.09 MiB

Used to translate:

Virtual Address
     ↓
Physical RAM Address

Every process needs page tables.

9 MB is tiny.

No concern.

Swap

Mean: 2.48 MiB

Almost no swap usage.

This is excellent.

Typically:

0 MB - 100 MB

is negligible.

If you saw:

2 GB
5 GB
10 GB

then we’d investigate memory pressure.

SwapCache

Mean: 131 KiB

Tracks swapped pages that may still exist in RAM.

Tiny amount.

Ignore.

Unused

Mean: 623 MiB
Max: 891 MiB

Actually free RAM.

Linux tries to keep this low because it prefers using RAM for caching.

Having only 623 MB free is not necessarily bad because:

Cache = 5.46 GB

can be reclaimed instantly.

Hardware Corrupted

0 B

No RAM errors detected by the kernel.

Good.

What would I conclude?

If I were reviewing this server:

Memory Health

No memory pressure

No excessive swap usage

Applications use very little RAM

Plenty of reclaimable cache

Kernel memory usage is normal

No hardware memory issues

Interview Answer

If an interviewer asks:

“Memory usage is 90%. Should I worry?”

You could answer:

Not necessarily. Linux aggressively uses free RAM for filesystem cache and buffers. I would look at “Available Memory” and swap activity rather than “Used Memory.” In this example, applications consume less than 1 GB while more than 5 GB is file cache, which can be reclaimed when needed. The negligible swap usage indicates there is no memory pressure.

Useful Commands to Verify
free -h

Look especially at:

available

rather than:

used
vmstat 1

Check:

si
so

(Swap In / Swap Out)

top

or

htop

to identify memory-consuming processes.

From these numbers, the server appears healthy and has significant RAM available through reclaimable cache, even though the “used memory” percentage may look high in Grafana.

Understanding PromQL: A Complete Guide

techhadoop Grafana, monitoring, Prometheus

What is PromQL?

PromQL (Prometheus Query Language) is the query language used by Prometheus to retrieve, filter, aggregate, and analyze time-series metrics.

It is the primary language used in:

  • Prometheus UI
  • Grafana dashboards
  • Alerting rules
  • Recording rules

PromQL Data Model

Metrics are stored as:

metric_name{label1="value1",label2="value2"} value timestamp

Example:

node_cpu_seconds_total{instance="server1",mode="idle"} 12345

Where:

ComponentMeaning
node_cpu_seconds_totalMetric name
instance=”server1″Label
mode=”idle”Label
12345Metric value

Basic PromQL Examples

1. Show a Metric
up

Returns all monitored targets.

Example:

up{instance="server1"} 1
up{instance="server2"} 1
  • 1 = healthy
  • 0 = down
2. Filter by Label
up{instance="server1:9100"}

Returns metrics only for that server.

3. Multiple Labels
node_cpu_seconds_total{
  instance="server1:9100",
  mode="idle"
}

Range Queries

Retrieve values over a time period.

Example:

node_cpu_seconds_total[5m]

Returns the last 5 minutes of data.

Rate Functions

One of the most common interview topics.

rate()

Calculates the per-second increase of a counter.

Example:

rate(http_requests_total[5m])

Meaning:

How many requests per second occurred during the last 5 minutes?

irate()

Calculates the rate using only the two most recent samples.

irate(http_requests_total[5m])

More responsive but noisier.

CPU Usage Example

Node Exporter provides:

node_cpu_seconds_total

Idle CPU:

avg(rate(node_cpu_seconds_total{mode="idle"}[5m]))

CPU Usage Percentage:

100 - (
 avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100
)

Very common Grafana dashboard query.

Memory Usage Example

Used Memory:

node_memory_MemTotal_bytes
-
node_memory_MemAvailable_bytes

Memory Percentage:

(
(node_memory_MemTotal_bytes
-
node_memory_MemAvailable_bytes)
/
node_memory_MemTotal_bytes
) * 100

Aggregation Functions

sum()
sum(http_requests_total)

Adds all values together.

avg()
avg(node_load1)

Average load.

max()
max(node_memory_MemAvailable_bytes)

Highest value.

min()
min(node_memory_MemAvailable_bytes)

Lowest value.

Group By

Example:

sum(rate(http_requests_total[5m])) by (instance)

Output:

server1 = 100 req/s
server2 = 150 req/s

Top Consumers

Top 5 CPU-consuming containers:

topk(
5,
sum(rate(container_cpu_usage_seconds_total[5m]))
by (pod)
)

Very common in Kubernetes/OpenShift interviews.

Kubernetes Examples

Pod Count
count(kube_pod_info)

Running Pods

count(kube_pod_status_phase{phase="Running"})
Node Count
count(kube_node_info)

OpenShift Examples

API Server Latency
histogram_quantile(
0.99,
sum(rate(apiserver_request_duration_seconds_bucket[5m]))
by (le)
)
etcd Latency
histogram_quantile(
0.99,
rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])
)
OVN Pod Status
up{job="ovn-kubernetes-node"}

Alert Rule Example

CPU > 80%

groups:
- name: cpu-alerts
  rules:
  - alert: HighCPU
    expr: 100 - (
      avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100
    ) > 80
    for: 5m

Meaning:

  • CPU above 80%
  • For 5 minutes
  • Fire alert

Recording Rule Example

Instead of calculating CPU every dashboard refresh:

- record: node:cpu_usage:avg
  expr: 100 - (
    avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100
  )

Then dashboards query:

node:cpu_usage:avg

This improves performance.

Common Interview Questions

What is the difference between rate() and irate()?
rate()irate()
Uses many samplesUses last two samples
SmootherMore responsive
Good for alertsGood for graphs
What is a counter?

A metric that only increases.

Examples:

http_requests_total
container_cpu_usage_seconds_total
What is a gauge?

A metric that can increase or decrease.

Examples:

node_memory_MemAvailable_bytes
node_load1
What is a histogram?

Used to measure distributions such as latency.

Example:

http_request_duration_seconds_bucket
What is cardinality?

The number of unique metric/label combinations.

Example:

http_requests_total{user="1"}
http_requests_total{user="2"}
http_requests_total{user="3"}
...

Millions of unique labels create high cardinality, which can cause Prometheus performance and memory issues.

Interview answer

PromQL is Prometheus’s query language used to retrieve, filter, aggregate, and calculate metrics. It supports functions such as rate(), sum(), avg(), histogram_quantile(), and label filtering, making it the foundation for Grafana dashboards, alerting rules, and monitoring in Kubernetes and OpenShift environments.