Enhance Grafana Alerts with Custom Annotations

techhadoop Grafana, monitoring, Prometheus

Right now alert email is showing only the default Grafana labels:

alertname = Sys Load
instance = 192.168.231.43:9100
grafana_folder = Alerts

You can make the email much more useful by adding custom annotations to the alert rule.

Option 1 – Add Summary and Description

In the alert rule, scroll down to Annotations and add:

Summary
High system load detected on {{ $labels.instance }}
Description
Server: {{ $labels.instance }}
Current Load: {{ printf "%.2f" $values.A.Value }}%
Threshold: 25%
Please investigate CPU utilization, running processes, and system responsiveness.

Then the email will look like:

ALERT: Sys Load
High system load detected on 192.168.231.43:9100
Server: 192.168.231.43:9100
Current Load: 87.45%
Threshold: 25%
Please investigate CPU utilization, running processes, and system responsiveness.

Option 2 – Add Severity Labels

Add labels:

severity = warning
team = infrastructure
environment = production

Then email shows:

severity: warning
team: infrastructure
environment: production
instance: 192.168.231.43:9100

Very useful when you have many alerts.

Option 3 – Include Useful Dashboard Links

Annotation:

Runbook URL
https://wiki.company.com/runbooks/linux-high-load
Dashboard URL
https://grafana.company.com/d/linux-server/linux-server-dashboard

Then engineers can click directly from the email.

Option 4 – Add Hostname Instead of IP

Currently you see:

192.168.231.43:9100

Much better if Prometheus exposes:

labels:
  hostname: cvm-srv44

Then annotation:

Host: {{ $labels.hostname }}
Instance: {{ $labels.instance }}

Email:

Host: cvm-srv44
Instance: 192.168.231.43:9100
Current Load: 87%

Much easier for operations teams.

Option 5 – Enterprise Style Alert

Summary
[{{ $labels.severity | toUpper }}] High System Load on {{ $labels.instance }}
Description
Environment: {{ $labels.environment }}
Server: {{ $labels.instance }}
Current Load: {{ printf "%.2f" $values.A.Value }}%
Alert Threshold: 25%
Recommended Checks:
1. top
2. htop
3. vmstat 1
4. iostat -x 1
5. journalctl -xe
Investigate CPU saturation, runaway processes, or I/O bottlenecks.

Bonus: Add Hostname Automatically in Prometheus

In your prometheus.yml:

scrape_configs:
  - job_name: linux-servers
    static_configs:
      - targets:
          - 192.168.231.43:9100
        labels:
          hostname: cvm-srv44
      - targets:
          - 192.168.231.122:9100
        labels:
          hostname: cvm-srv45

Then use:

Host: {{ $labels.hostname }}

in Grafana alerts.

This is usually the biggest improvement because emails become:

[WARNING] High System Load
Host: cvm-srv44
IP: 192.168.231.43:9100
Current Load: 91.7%
Threshold: 80%

instead of just showing an IP address.

Leave a Reply