Right now alert email is showing only the default Grafana labels:

alertname = Sys Load
instance = 192.168.231.43:9100
grafana_folder = Alerts

You can make the email much more useful by adding custom annotations to the alert rule.

Option 1 – Add Summary and Description

In the alert rule, scroll down to Annotations and add:

Summary

High system load detected on {{ $labels.instance }}

Description

Server: {{ $labels.instance }}
Current Load: {{ printf "%.2f" $values.A.Value }}%
Threshold: 25%
Please investigate CPU utilization, running processes, and system responsiveness.

Then the email will look like:

ALERT: Sys Load
High system load detected on 192.168.231.43:9100
Server: 192.168.231.43:9100
Current Load: 87.45%
Threshold: 25%
Please investigate CPU utilization, running processes, and system responsiveness.

Option 2 – Add Severity Labels

Add labels:

severity = warning
team = infrastructure
environment = production

Then email shows:

severity: warning
team: infrastructure
environment: production
instance: 192.168.231.43:9100

Very useful when you have many alerts.

Option 3 – Include Useful Dashboard Links

Annotation:

Runbook URL

https://wiki.company.com/runbooks/linux-high-load

Dashboard URL

https://grafana.company.com/d/linux-server/linux-server-dashboard

Then engineers can click directly from the email.

Option 4 – Add Hostname Instead of IP

Currently you see:

192.168.231.43:9100

Much better if Prometheus exposes:

labels:
  hostname: cvm-srv44

Then annotation:

Host: {{ $labels.hostname }}
Instance: {{ $labels.instance }}

Email:

Host: cvm-srv44
Instance: 192.168.231.43:9100
Current Load: 87%

Much easier for operations teams.

Option 5 – Enterprise Style Alert

Summary

[{{ $labels.severity | toUpper }}] High System Load on {{ $labels.instance }}

Description

Environment: {{ $labels.environment }}
Server: {{ $labels.instance }}
Current Load: {{ printf "%.2f" $values.A.Value }}%
Alert Threshold: 25%
Recommended Checks:
1. top
2. htop
3. vmstat 1
4. iostat -x 1
5. journalctl -xe
Investigate CPU saturation, runaway processes, or I/O bottlenecks.

Bonus: Add Hostname Automatically in Prometheus

In your prometheus.yml:

scrape_configs:
  - job_name: linux-servers
    static_configs:
      - targets:
          - 192.168.231.43:9100
        labels:
          hostname: cvm-srv44
      - targets:
          - 192.168.231.122:9100
        labels:
          hostname: cvm-srv45

Then use:

Host: {{ $labels.hostname }}

in Grafana alerts.

This is usually the biggest improvement because emails become:

[WARNING] High System Load
Host: cvm-srv44
IP: 192.168.231.43:9100
Current Load: 91.7%
Threshold: 80%

instead of just showing an IP address.

1. How do you reduce Prometheus cardinality?

Cardinality means too many unique metric/label combinations.

Example of bad label:

http_requests_total{user_id="12345"}

This creates millions of time series.

To reduce cardinality:

- Avoid dynamic labels: user_id, session_id, request_id, pod_uid
- Keep only useful labels: job, instance, namespace, pod, status_code
- Drop unnecessary labels using relabel_configs
- Use recording rules for expensive queries
- Reduce scrape targets if not needed
- Avoid exposing too many custom metrics
- Set retention limits

Interview answer:

I reduce Prometheus cardinality by controlling labels, dropping high-cardinality labels, reviewing /targets and /tsdb-status, and avoiding dynamic values like user IDs or request IDs in metric labels.

2. Difference between recording rules and alert rules

Recording rule

Pre-calculates a query and saves the result as a new metric.

Example:

- record: node:cpu_usage:avg5m
  expr: 100 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100

Used for:

- Faster dashboards
- Reusable metrics
- Reducing query load

Alert rule

Triggers an alert when a condition is met.

Example:

- alert: HighCPU
  expr: node:cpu_usage:avg5m > 80
  for: 5m

Interview answer:

Recording rules improve performance by precomputing metrics. Alert rules evaluate conditions and trigger notifications.

3. How do you monitor Kubernetes at scale?

Use:

- Prometheus HA
- kube-state-metrics
- node-exporter
- cAdvisor/container metrics
- Alertmanager
- Grafana dashboards
- Loki for logs
- Tempo for traces
- Mimir/Thanos for long-term metrics

For large clusters:

- Use recording rules
- Limit scrape intervals
- Reduce metric cardinality
- Use remote_write to Mimir/Thanos
- Separate platform and application monitoring
- Use namespace/team-based dashboards

Interview answer:

At scale, I avoid a single large Prometheus doing everything. I deploy Prometheus per cluster or per domain, use remote_write to a central backend like Mimir or Thanos, apply cardinality controls, and visualize everything centrally in Grafana.

4. How would you deploy Grafana HA?

Architecture:

Load Balancer
     |
+----+----+
|         |
Grafana  Grafana
|         |
+----+----+
     |
PostgreSQL/MySQL

Key points:

- Multiple Grafana instances
- Shared external database: PostgreSQL or MySQL
- Load balancer in front
- Shared configuration
- SSO integration
- Persistent dashboards stored in DB
- Provision dashboards/data sources using GitOps/IaC

Important:

Do not use SQLite for HA.

Interview answer:

Grafana is mostly stateless. For HA, I run multiple Grafana pods or VMs behind a load balancer and use PostgreSQL as the shared backend database.

5. How would you design LGTM for 1000 servers?

LGTM means:

L = Loki      Logs
G = Grafana   Dashboards
T = Tempo     Traces
M = Mimir     Metrics

Design:

1000 Servers
   |
Node Exporter / Alloy / Promtail / OpenTelemetry Collector
   |
Metrics → Mimir
Logs    → Loki
Traces  → Tempo
   |
Grafana

Important design points:

- Use Grafana Alloy or OpenTelemetry Collector agents
- Use Mimir for scalable metrics
- Use Loki for centralized logs
- Use Tempo for distributed tracing
- Store data in object storage like S3/Azure Blob
- Use retention policies
- Use tenant/team separation
- Use alerting with Alertmanager or Grafana Alerting

Interview answer:

For 1000 servers, I would avoid one single Prometheus scraping everything. I would use agents on servers, remote_write metrics to Mimir, send logs to Loki, traces to Tempo, and use Grafana as the single visualization and alerting layer.

6. How do you troubleshoot missing metrics?

Check in this order:

1. Is the target up?
2. Is Prometheus scraping the endpoint?
3. Is the exporter running?
4. Is the metric exposed on /metrics?
5. Is the scrape config correct?
6. Are relabeling rules dropping it?
7. Is the time range correct in Grafana?
8. Is the PromQL query correct?
9. Is authentication/TLS blocking scrape?
10. Is the metric name changed?

Useful PromQL:

up

scrape_samples_scraped

scrape_duration_seconds

Useful commands:

curl http://server:9100/metrics

kubectl get servicemonitor -A
kubectl get podmonitor -A
kubectl get targets

Interview answer:

I start from the source. First I verify the exporter exposes the metric, then Prometheus target status, scrape configuration, relabeling, and finally Grafana query/time range.

7. How do you secure Grafana in an enterprise environment?

Use:

- HTTPS/TLS
- SSO with SAML/OIDC/LDAP
- MFA through identity provider
- RBAC
- Team-based permissions
- Folder/dashboard permissions
- Disable anonymous access
- Secure admin account
- Use secrets management
- Audit logging
- Network restrictions
- Backup database and dashboards

Also:

- Do not expose Grafana directly to the internet
- Use reverse proxy or ingress with TLS
- Use least privilege for data sources
- Separate dev/test/prod dashboards

Interview answer:

I secure Grafana using enterprise identity integration, RBAC, TLS, least-privilege data source credentials, dashboard permissions, audit logs, and network restrictions.

8. How do you integrate Grafana with OpenShift?

Options:

- Use OpenShift monitoring Prometheus/Thanos as data source
- Connect Grafana to Thanos Querier
- Use ServiceAccount token for authentication
- Import Kubernetes/OpenShift dashboards
- Monitor nodes, pods, namespaces, etcd, API server, ingress, OVN

Common OpenShift components:

- Prometheus
- Alertmanager
- Thanos Querier
- kube-state-metrics
- node-exporter
- Cluster Monitoring Operator

Example data source:

Grafana → Thanos Querier → OpenShift metrics

Interview answer:

In OpenShift, I usually connect Grafana to Thanos Querier or Prometheus using a service account token. Then I build dashboards for cluster health, nodes, pods, namespaces, etcd, API server, ingress, and OVN networking.

9. How do you implement multi-tenancy?

Ways to implement multi-tenancy:

- Separate organizations in Grafana
- Separate folders per team
- Team-based RBAC
- Dashboard permissions
- Data source permissions
- Separate tenants in Mimir/Loki/Tempo
- Separate Kubernetes namespaces
- Separate alert contact points

Example:

Team A → Folder A → Data source tenant A
Team B → Folder B → Data source tenant B

For strong isolation:

- Use separate Grafana organizations
- Use separate Mimir/Loki tenants
- Use SSO groups mapped to Grafana teams

Interview answer:

I implement multi-tenancy using SSO group mapping, teams, folders, RBAC, data source permissions, and backend tenant isolation in Mimir, Loki, or Tempo.

10. How do you monitor AWS, Azure, and OpenShift from one Grafana instance?

Architecture:

AWS CloudWatch
Azure Monitor
OpenShift Thanos/Prometheus
Linux Node Exporter
Kubernetes Metrics
        |
        v
     Grafana

Data sources:

AWS        → CloudWatch
Azure      → Azure Monitor
OpenShift  → Prometheus/Thanos
Logs       → Loki or Elasticsearch
Traces     → Tempo

Best practices:

- Use separate folders per platform
- Use variables: cloud, region, cluster, namespace
- Use RBAC per team
- Use central alerting
- Use standardized dashboards
- Use tags and labels consistently

Interview answer:

I would use Grafana as the central visualization layer with multiple data sources: CloudWatch for AWS, Azure Monitor for Azure, and Prometheus/Thanos for OpenShift. Then I would organize dashboards by platform, region, cluster, and application, with RBAC and alerting policies per team.

Strong closing answer for interview

My approach is to design Grafana as the central observability portal, but not as the only backend. Prometheus handles metrics collection, Loki handles logs, Tempo handles traces, and Mimir or Thanos handles scalable long-term metrics. For enterprise environments, I focus on HA, RBAC, SSO, cardinality control, recording rules, proper alerting, and multi-tenant separation.

Based on the provided Grafana dashboard image (utilizing the popular Node Exporter Full template), here is a breakdown of what is happening on this Linux server (cvm-srv51).

The Main Takeaway: CPU Bottleneck

Your server is currently under a heavy CPU bottleneck, while memory and disk resources are mostly relaxed. Around 12:43 to 12:45, there was a brief dip where the load dropped significantly, but it immediately spiked back up to its previous high levels.

Detailed Metric Breakdown

1. Quick CPU / Mem / Disk (Gauges)

• CPU Busy (95.2%): Your CPU is running almost at maximum capacity. The red ring indicates this is in a critical threshold.
• Sys Load (136.3%): This represents a system load average that exceeds the server’s capacity. Since the server has 4 CPU Cores (shown in the info box), a 100% load capacity would mean a load average of 4.0. Running at 136.3% means the CPU queue is constantly backed up, and processes are waiting for CPU time.
• RAM Used (17.2%) & SWAP Used (0.0%): Out of 23 GiB available, memory usage is very low and healthy. SWAP isn’t being touched, meaning this is strictly a compute problem, not a memory-starvation problem.
• Root FS Used (32.7%): Storage space is perfectly fine, with plenty of room left on the root filesystem.

2. Historical Graphs (Last 15 Minutes)

• CPU Basic: The large blue block dominating the graph represents Busy User space. This means the high CPU usage is being driven by user-level applications or services (e.g., a heavy application, database queries, a running script, or containerized apps), rather than kernel overhead (Busy System) or waiting for hardware (Busy Iowait).
• Memory Basic: A completely flat line showing steady, low memory utilization.
• Network Traffic Basic: There is a noticeable drop in network traffic that perfectly correlates with the dip in CPU usage around 12:44. When network traffic resumed, CPU usage shot back up. This suggests that incoming network requests or data transfers are likely triggering the high CPU workload.

Next Steps for Troubleshooting

Because the CPU is being hammered by user-space applications linked to network activity, you should log into the server (cvm-srv51) via SSH and investigate:

1. Run top or htop: Check which specific processes are consuming the most CPU.
2. Inspect Containers: Since the network graph lists several virtual interfaces (like docker0 and veth...), it’s highly likely this server is running Docker containers. Use docker stats to see which container is pulling all the compute.
3. Check Application Logs: Match the timeline (the dip at 12:44) against your application or web server logs to see what traffic paused or restarted at that moment.

What is PromQL?

PromQL (Prometheus Query Language) is the query language used by Prometheus to retrieve, filter, aggregate, and analyze time-series metrics.

It is the primary language used in:

• Prometheus UI
• Grafana dashboards
• Alerting rules
• Recording rules

PromQL Data Model

Metrics are stored as:

metric_name{label1="value1",label2="value2"} value timestamp

Example:

node_cpu_seconds_total{instance="server1",mode="idle"} 12345

Where:

Basic PromQL Examples

1. Show a Metric

up

Returns all monitored targets.

Example:

up{instance="server1"} 1
up{instance="server2"} 1

• 1 = healthy
• 0 = down

2. Filter by Label

up{instance="server1:9100"}

Returns metrics only for that server.

3. Multiple Labels

node_cpu_seconds_total{
  instance="server1:9100",
  mode="idle"
}

Range Queries

Retrieve values over a time period.

Example:

node_cpu_seconds_total[5m]

Returns the last 5 minutes of data.

Rate Functions

One of the most common interview topics.

rate()

Calculates the per-second increase of a counter.

Example:

rate(http_requests_total[5m])

Meaning:

How many requests per second occurred during the last 5 minutes?

irate()

Calculates the rate using only the two most recent samples.

irate(http_requests_total[5m])

More responsive but noisier.

CPU Usage Example

Node Exporter provides:

node_cpu_seconds_total

Idle CPU:

avg(rate(node_cpu_seconds_total{mode="idle"}[5m]))

CPU Usage Percentage:

100 - (
 avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100
)

Very common Grafana dashboard query.

Memory Usage Example

Used Memory:

node_memory_MemTotal_bytes
-
node_memory_MemAvailable_bytes

Memory Percentage:

(
(node_memory_MemTotal_bytes
-
node_memory_MemAvailable_bytes)
/
node_memory_MemTotal_bytes
) * 100

Aggregation Functions

sum()

sum(http_requests_total)

Adds all values together.

avg()

avg(node_load1)

Average load.

max()

max(node_memory_MemAvailable_bytes)

Highest value.

min()

min(node_memory_MemAvailable_bytes)

Lowest value.

Group By

Example:

sum(rate(http_requests_total[5m])) by (instance)

Output:

server1 = 100 req/s
server2 = 150 req/s

Top Consumers

Top 5 CPU-consuming containers:

topk(
5,
sum(rate(container_cpu_usage_seconds_total[5m]))
by (pod)
)

Very common in Kubernetes/OpenShift interviews.

Kubernetes Examples

Pod Count

count(kube_pod_info)

Running Pods

count(kube_pod_status_phase{phase="Running"})

Node Count

count(kube_node_info)

OpenShift Examples

API Server Latency

histogram_quantile(
0.99,
sum(rate(apiserver_request_duration_seconds_bucket[5m]))
by (le)
)

etcd Latency

histogram_quantile(
0.99,
rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])
)

OVN Pod Status

up{job="ovn-kubernetes-node"}

Alert Rule Example

CPU > 80%

groups:
- name: cpu-alerts
  rules:
  - alert: HighCPU
    expr: 100 - (
      avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100
    ) > 80
    for: 5m

Meaning:

• CPU above 80%
• For 5 minutes
• Fire alert

Recording Rule Example

Instead of calculating CPU every dashboard refresh:

- record: node:cpu_usage:avg
  expr: 100 - (
    avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100
  )

Then dashboards query:

node:cpu_usage:avg

This improves performance.

Common Interview Questions

What is the difference between rate() and irate()?

What is a counter?

A metric that only increases.

Examples:

http_requests_total
container_cpu_usage_seconds_total

What is a gauge?

A metric that can increase or decrease.

Examples:

node_memory_MemAvailable_bytes
node_load1

What is a histogram?

Used to measure distributions such as latency.

Example:

http_request_duration_seconds_bucket

What is cardinality?

The number of unique metric/label combinations.

Example:

http_requests_total{user="1"}
http_requests_total{user="2"}
http_requests_total{user="3"}
...

Millions of unique labels create high cardinality, which can cause Prometheus performance and memory issues.

Interview answer

PromQL is Prometheus’s query language used to retrieve, filter, aggregate, and calculate metrics. It supports functions such as rate(), sum(), avg(), histogram_quantile(), and label filtering, making it the foundation for Grafana dashboards, alerting rules, and monitoring in Kubernetes and OpenShift environments.

OpenTelemetry (often abbreviated as OTel) is an open-source, vendor-neutral observability framework. It provides a standardized set of APIs, SDKs, and tools to design, generate, collect, and export telemetry data (metrics, logs, and traces) from your software applications.

Managed by the Cloud Native Computing Foundation (CNCF)—the same body behind Kubernetes and Prometheus—OpenTelemetry has quickly become the absolute industry standard for modern application monitoring.

Why OpenTelemetry Exists (The Problem It Solves)

Historically, if you wanted to monitor an application, you had to install vendor-specific code or agents.

• If you used Datadog, you had to install the Datadog SDK.
• If you switched to Dynatrace, New Relic, or AppDynamics, you had to rewrite parts of your application code to use their specific libraries.

This created massive vendor lock-in.

OpenTelemetry completely decouples data collection from data storage. You instrument your application once using the universal OpenTelemetry standard. If you want to change your monitoring backend later from Datadog to an open-source stack like Prometheus and Grafana, you just change a single line in a configuration file—no code changes required.

The Three Pillars of Telemetry (M.E.L.T.)

OpenTelemetry is designed to handle all three primary types of observability data:

1. Traces: Tracks the end-to-end journey of a single request as it travels across different microservices, databases, and APIs. It helps you pinpoint exactly which function or database query is causing a slowdown.
2. Metrics: Numeric values measured over time (e.g., CPU utilization, memory usage, request counts, error rates).
3. Logs: Structured text records of discrete events (e.g., a system crash log or an authentication failure message).

The Architecture: How It Works

An OpenTelemetry implementation generally consists of three main parts:

┌─────────────────────────┐
│   Your Application      │  (App is instrumented with OTel SDK)
│  [Go, Python, Java...]  │
└────────────┬────────────┘
             │ (OTLP Protocol)
             ▼
┌─────────────────────────┐
│  OpenTelemetry Collector│  (A lightweight proxy running on the host)
└────────────┬────────────┘
             ├────────────────────────┬────────────────────────┐
             ▼                        ▼                        ▼
     [ Prometheus ]             [ Grafana Loki ]         [ Jaeger / Tempo ]
       (Metrics)                     (Logs)                  (Traces)

1. The API and SDK

You include the OpenTelemetry library directly inside your application code. OpenTelemetry features Auto-Instrumentation for popular languages. If you run a Python FastAPI or Java Spring Boot application, OTel can automatically capture database queries and HTTP requests without you writing a single line of custom telemetry code.

2. The OTLP Protocol

All data generated by the SDK uses a unified language called OTLP (OpenTelemetry Protocol), ensuring standard formatting across every application in your environment.

3. The OpenTelemetry Collector (The Muscle)

The Collector is a separate, lightweight binary or container that runs alongside your application (similar to how Node Exporter runs on a Linux host). It receives the data from your applications, processes it (batches it, strips sensitive PII data, compresses it), and exports it to your chosen backend databases.

How It Fits With Your Prometheus & Grafana Stack

If you are already working with Prometheus, Grafana, and Alertmanager, OpenTelemetry integrates seamlessly into your world:

• Prometheus excels at pulling metrics, but it historically doesn’t handle tracing well.
• OpenTelemetry is unmatched at generating application traces and logs.

In a modern production stack, engineers often use OpenTelemetry inside their application code to generate traces and metrics. They send that data to the OpenTelemetry Collector, which is then configured to forward the metrics straight into Prometheus, logs into Grafana Loki, and traces into Grafana Tempo.

Ultimately, everything gets visualized on a unified Grafana dashboard, giving you total visibility from the bare-metal hardware all the way down to a single line of application code.