Interview Questions Often Asked for Senior Grafana Architects

techhadoop Grafana, monitoring, Prometheus

1. How do you reduce Prometheus cardinality?

Cardinality means too many unique metric/label combinations.

Example of bad label:

http_requests_total{user_id="12345"}

This creates millions of time series.

To reduce cardinality:

- Avoid dynamic labels: user_id, session_id, request_id, pod_uid
- Keep only useful labels: job, instance, namespace, pod, status_code
- Drop unnecessary labels using relabel_configs
- Use recording rules for expensive queries
- Reduce scrape targets if not needed
- Avoid exposing too many custom metrics
- Set retention limits

Interview answer:

I reduce Prometheus cardinality by controlling labels, dropping high-cardinality labels, reviewing /targets and /tsdb-status, and avoiding dynamic values like user IDs or request IDs in metric labels.

2. Difference between recording rules and alert rules

Recording rule

Pre-calculates a query and saves the result as a new metric.

Example:

- record: node:cpu_usage:avg5m
  expr: 100 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100

Used for:

- Faster dashboards
- Reusable metrics
- Reducing query load
Alert rule

Triggers an alert when a condition is met.

Example:

- alert: HighCPU
  expr: node:cpu_usage:avg5m > 80
  for: 5m

Interview answer:

Recording rules improve performance by precomputing metrics. Alert rules evaluate conditions and trigger notifications.

3. How do you monitor Kubernetes at scale?

Use:

- Prometheus HA
- kube-state-metrics
- node-exporter
- cAdvisor/container metrics
- Alertmanager
- Grafana dashboards
- Loki for logs
- Tempo for traces
- Mimir/Thanos for long-term metrics

For large clusters:

- Use recording rules
- Limit scrape intervals
- Reduce metric cardinality
- Use remote_write to Mimir/Thanos
- Separate platform and application monitoring
- Use namespace/team-based dashboards

Interview answer:

At scale, I avoid a single large Prometheus doing everything. I deploy Prometheus per cluster or per domain, use remote_write to a central backend like Mimir or Thanos, apply cardinality controls, and visualize everything centrally in Grafana.

4. How would you deploy Grafana HA?

Architecture:

Load Balancer
     |
+----+----+
|         |
Grafana  Grafana
|         |
+----+----+
     |
PostgreSQL/MySQL

Key points:

- Multiple Grafana instances
- Shared external database: PostgreSQL or MySQL
- Load balancer in front
- Shared configuration
- SSO integration
- Persistent dashboards stored in DB
- Provision dashboards/data sources using GitOps/IaC

Important:

Do not use SQLite for HA.

Interview answer:

Grafana is mostly stateless. For HA, I run multiple Grafana pods or VMs behind a load balancer and use PostgreSQL as the shared backend database.

5. How would you design LGTM for 1000 servers?

LGTM means:

L = Loki      Logs
G = Grafana   Dashboards
T = Tempo     Traces
M = Mimir     Metrics

Design:

1000 Servers
   |
Node Exporter / Alloy / Promtail / OpenTelemetry Collector
   |
Metrics → Mimir
Logs    → Loki
Traces  → Tempo
   |
Grafana

Important design points:

- Use Grafana Alloy or OpenTelemetry Collector agents
- Use Mimir for scalable metrics
- Use Loki for centralized logs
- Use Tempo for distributed tracing
- Store data in object storage like S3/Azure Blob
- Use retention policies
- Use tenant/team separation
- Use alerting with Alertmanager or Grafana Alerting

Interview answer:

For 1000 servers, I would avoid one single Prometheus scraping everything. I would use agents on servers, remote_write metrics to Mimir, send logs to Loki, traces to Tempo, and use Grafana as the single visualization and alerting layer.

6. How do you troubleshoot missing metrics?

Check in this order:

1. Is the target up?
2. Is Prometheus scraping the endpoint?
3. Is the exporter running?
4. Is the metric exposed on /metrics?
5. Is the scrape config correct?
6. Are relabeling rules dropping it?
7. Is the time range correct in Grafana?
8. Is the PromQL query correct?
9. Is authentication/TLS blocking scrape?
10. Is the metric name changed?

Useful PromQL:

up
scrape_samples_scraped
scrape_duration_seconds

Useful commands:

curl http://server:9100/metrics
kubectl get servicemonitor -A
kubectl get podmonitor -A
kubectl get targets

Interview answer:

I start from the source. First I verify the exporter exposes the metric, then Prometheus target status, scrape configuration, relabeling, and finally Grafana query/time range.

7. How do you secure Grafana in an enterprise environment?

Use:

- HTTPS/TLS
- SSO with SAML/OIDC/LDAP
- MFA through identity provider
- RBAC
- Team-based permissions
- Folder/dashboard permissions
- Disable anonymous access
- Secure admin account
- Use secrets management
- Audit logging
- Network restrictions
- Backup database and dashboards

Also:

- Do not expose Grafana directly to the internet
- Use reverse proxy or ingress with TLS
- Use least privilege for data sources
- Separate dev/test/prod dashboards

Interview answer:

I secure Grafana using enterprise identity integration, RBAC, TLS, least-privilege data source credentials, dashboard permissions, audit logs, and network restrictions.

8. How do you integrate Grafana with OpenShift?

Options:

- Use OpenShift monitoring Prometheus/Thanos as data source
- Connect Grafana to Thanos Querier
- Use ServiceAccount token for authentication
- Import Kubernetes/OpenShift dashboards
- Monitor nodes, pods, namespaces, etcd, API server, ingress, OVN

Common OpenShift components:

- Prometheus
- Alertmanager
- Thanos Querier
- kube-state-metrics
- node-exporter
- Cluster Monitoring Operator

Example data source:

Grafana → Thanos Querier → OpenShift metrics

Interview answer:

In OpenShift, I usually connect Grafana to Thanos Querier or Prometheus using a service account token. Then I build dashboards for cluster health, nodes, pods, namespaces, etcd, API server, ingress, and OVN networking.

9. How do you implement multi-tenancy?

Ways to implement multi-tenancy:

- Separate organizations in Grafana
- Separate folders per team
- Team-based RBAC
- Dashboard permissions
- Data source permissions
- Separate tenants in Mimir/Loki/Tempo
- Separate Kubernetes namespaces
- Separate alert contact points

Example:

Team A → Folder A → Data source tenant A
Team B → Folder B → Data source tenant B

For strong isolation:

- Use separate Grafana organizations
- Use separate Mimir/Loki tenants
- Use SSO groups mapped to Grafana teams

Interview answer:

I implement multi-tenancy using SSO group mapping, teams, folders, RBAC, data source permissions, and backend tenant isolation in Mimir, Loki, or Tempo.

10. How do you monitor AWS, Azure, and OpenShift from one Grafana instance?

Architecture:

AWS CloudWatch
Azure Monitor
OpenShift Thanos/Prometheus
Linux Node Exporter
Kubernetes Metrics
        |
        v
     Grafana

Data sources:

AWS        → CloudWatch
Azure      → Azure Monitor
OpenShift  → Prometheus/Thanos
Logs       → Loki or Elasticsearch
Traces     → Tempo

Best practices:

- Use separate folders per platform
- Use variables: cloud, region, cluster, namespace
- Use RBAC per team
- Use central alerting
- Use standardized dashboards
- Use tags and labels consistently

Interview answer:

I would use Grafana as the central visualization layer with multiple data sources: CloudWatch for AWS, Azure Monitor for Azure, and Prometheus/Thanos for OpenShift. Then I would organize dashboards by platform, region, cluster, and application, with RBAC and alerting policies per team.

Strong closing answer for interview

My approach is to design Grafana as the central observability portal, but not as the only backend. Prometheus handles metrics collection, Loki handles logs, Tempo handles traces, and Mimir or Thanos handles scalable long-term metrics. For enterprise environments, I focus on HA, RBAC, SSO, cardinality control, recording rules, proper alerting, and multi-tenant separation.

Leave a Reply