Your Prettiest Dashboard Is Lying to You (And Grafana Will Never Tell You)

Pull up a chair, grab a cup of chai, and let’s talk about the prettiest graph on your big screen. It might be completely empty.

Somewhere right inside your team’s Grafana setup, there is a panel that has not drawn a single real data point for six months. No bright red banners. No screaming error messages. Nothing. The graph line is dead flat, the legend at the bottom is blank, and whenever a sev-1 (severity-1, drop-everything) incident breaks out at 2:00 AM, everyone glances at that wall of monitors and breathes a sigh of relief: “Look, the payment error rate is zero! System is calm.”

The system is not calm, my friend. The query is dead. The metric name it asks for simply does not exist anymore. Maybe a backend developer refactored code and renamed it, maybe someone tweaked a scrape job (the routine background fetch where Prometheus pulls numbers from your app’s /metrics endpoint), or maybe someone deleted an old recording rule (a saved query that calculates averages ahead of time to make dashboards load faster). And Grafana? Grafana will never, ever tap you on the shoulder to tell you.

In PromQL (Prometheus Query Language, the syntax we use to pull and slice metrics), if a query matches nothing on disk, it returns an empty vector — meaning a clean, valid list with zero items in it. And Grafana renders an empty vector as a quiet, peaceful flatline. Today, we are setting up a lie-detector test for that entire wall of screens. First, a manual audit to catch the dead weight hiding right now, and second, a quick gate in your CI pipeline (Continuous Integration, the automated test checks that run before code merges) so broken dashboards never make it past code review again.

And if you think this is a theoretical corner case, check the calendar. Kubernetes v1.37 landed in September 2026, switching native histograms on by default. If your platform team adjusts scrape configs without keeping classic histogram scraping during the transition, Prometheus quietly drops classic _bucket, _count, and _sum time series (the individual timestamped streams of metric data). Overnight, every histogram_quantile panel you rely on turns pitch black. The ground under your dashboards is shifting — so we need this pre-flight check running today.

Why Grafana never tells you the query is dead

Here is the quirk with Prometheus and Grafana that trips up every engineer early on: querying a metric name that does not exist is not an error. To Prometheus, it is a completely valid question that just happens to have zero answers. Open a terminal and run this against your Prometheus box:

# a metric that exists: returns series
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=up' | jq '.data.result | length'
# a metric that never existed: returns nothing, status still "success"
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=definitely_not_a_real_metric_total' \
  | jq '{status, results: (.data.result | length)}'
# {"status": "success", "results": 0}

Look at that second response. Status: success. Results: 0. Grafana takes that zero and renders a neat little box with a polite whisper: “No data”. To the query engine, nothing failed. It did its job perfectly.

Think of it like walking up to a librarian and asking for a book that was never written. Instead of saying, “Arre bhai, that book does not exist!” , the librarian smiles warmly and says, “Search successful! Here are zero books.” Technically accurate, completely useless, and delivered with 100% confidence.

Prometheus makes this even sneakier because of how time series disappear. When a service stops publishing a metric — or when an entire pod gets terminated — Prometheus records a stale marker (a special tombstone flag that tells the database this specific stream stopped). Once that happens, queries ignore it. Even without a stale marker, Prometheus uses a setting called lookback-delta (the five-minute window it looks back in time to grab the newest data point). If a metric hasn’t checked in for five minutes, it evaporates from instant queries.

Your metric didn’t throw an exception. It didn’t throw an alert. It simply ghosted you. And your Grafana dashboard is still sitting at the dinner table, setting a plate and waiting for it to show up.

Don’t blame the query engine. In monitoring, “no data” is often normal — if your payment gateway has zero 500 errors right now, an error query legitimately returns an empty vector. The real breakdown is human: nobody owns the contract between the dashboard JSON and the live metric inventory. Someone builds a shiny dashboard on a Tuesday, commits it, and assumes the world stands still forever.

Grafana never errors on a query that matches nothing. The panel just draws nothing — and an empty panel looks exactly like a healthy system.

The three ways a metric name dies

In practice, dead panels rot in three predictable ways. On your monitor they look identical — a flat, dead line — but under the hood, each happens for a totally different reason:

• The rename — a developer renames the metric in the app’s instrumentation. In Prometheus: old name stops being emitted; series go stale and disappear. Panel shows: flat, empty panel. Spot it: old name absent from the live inventory; new name present.

• The dropped scrape — target removed from discovery, job renamed, or exporter decommissioned. In Prometheus: all series from the target marked stale at once. Panel shows: every panel for that job goes empty together. Spot it: /api/v1/targets shows the target gone; up for the job disappears.

• The deleted recording rule — a rule is removed or renamed in the rules file. In Prometheus: recorded series stop being produced; stale markers written. Panel shows: panels and alerts built on the recorded name go dark. Spot it: name missing from /api/v1/rules; underlying raw metrics still exist.

Let’s break those down:

• The Rename: Super common. A developer updates a library, cleans up code, and changes http_requests_total to http_server_requests_total. The old time series goes stale and vanishes. The dashboard is still looking for the old name. You can fix this in two minutes once you find the new metric name in Prometheus.

• The Dropped Scrape: Very loud. Someone updates a Kubernetes ServiceMonitor or renames a Prometheus job label, and Prometheus stops scraping the pod altogether. Suddenly ten panels tied to that service flatline simultaneously. Because everything broke at once, people notice quickly.

• The Deleted Recording Rule: The sneakiest killer of all. A recording rule pre-calculates an expensive query — say, job:http_errors:rate5m — and saves it under that shortcut name. Someone cleans up an alert rules YAML file and accidentally deletes that recording rule. The raw container metrics are still alive and ticking! But the pre-computed metric name your executive dashboard was querying has vanished into thin air.

Three different causes, one identical corpse: a panel that queries a name that no longer exists.

Step one: see the lie — the manual audit

Before we jump into automation or CI pipelines, sit down and run this once by hand. It takes ten minutes, and I promise you will never look at your team’s monitoring boards the same way again.

1. Get the dashboard JSON. In Grafana, every dashboard is just a big JSON document. Inside each panel object, there is an array called targets, and inside each target is the raw PromQL query sitting under the expr key. Go to your dashboard settings, click JSON Model , and copy it out — or even better, pull the JSON file straight from your Git repository if you practice Dashboards-as-Code.

2. Extract every PromQL expression. Using jq (the lightweight command-line tool for parsing JSON), we can rip out every single query across every panel and row in one shot:

# every PromQL expression in the dashboard, one per line
jq -r '.. | .expr? // empty' dashboard.json | sort -u
# example output (illustrative)
# sum(rate(http_requests_total{job="api"}[$__rate_interval]))
# job:http_errors:rate5m / job:http_requests:rate5m

3. Check each expression against your live metric inventory. Prometheus exposes an API endpoint listing every single metric name currently stored in memory and on disk. Dump that list, and then test your dashboard queries against the live server:

# the full live metric inventory, one name per line
curl -s http://prometheus:9090/api/v1/label/__name__/values \
  | jq -r '.data[]' | sort -u > live-metrics.txt
# and the bluntest possible check: does this query return anything at all?
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=job:http_errors:rate5m' \
  | jq '.data.result | length'
# 0  <-- this panel has been lying to you

Now, keep two practical things in mind before you go deleting panels in anger:

First, Grafana dashboards use template variables — those drop-down selectors at the top like $namespace or dynamic interval helpers like $__rate_interval. Raw Prometheus does not know what $__rate_interval means; that is Grafana’s internal magic. If you send raw variable names straight to Prometheus, the query fails or returns empty. You have to substitute safe defaults (like replacing $__rate_interval with 5m) before querying.

Second, zero results does not always mean a dead metric. If a panel monitors a nightly backup batch job or a canary deployment metric that only fires during rollouts, it might naturally return zero at 11:00 AM on a Tuesday. The script finds suspects; you, the engineer, decide who goes to jail. Automate the search, but keep a human in the loop for the verdict.

Every dashboard is a snapshot of the metric inventory on the day it was written. The audit is how you find out how old the snapshot is.

Automate it: the audit script

Running this by hand once is a great eye-opener. But systems drift constantly. Developers ship code, recording rules get pruned, and targets get reconfigured. We need a script that scans an entire folder of dashboard files automatically.

The audit pipeline: dashboard JSON on one side, the live metric inventory on the other, a diff in the middle.

Here is a clean Bash script that does the heavy lifting. It loops through your dashboard JSON files, pulls every panel query, replaces Grafana’s template variables with generic fallback patterns using regex, fires instant queries against Prometheus, and flags every expression that comes back completely empty:

#!/usr/bin/env bash
# dashboard-dead-panel-audit.sh — flag panels querying metrics that no longer exist
# usage: PROM_URL=http://prometheus:9090 ./dashboard-dead-panel-audit.sh ./dashboards
set -euo pipefail
PROM_URL="${PROM_URL:-http://prometheus:9090}"
DASH_DIR="${1:-./dashboards}"
failures=0
# the variables Grafana would normally fill in; substitute safe defaults
sanitize() {
  sed -e 's/\$__rate_interval/5m/g' \
      -e 's/\$__interval/1m/g' \
      -e 's/\$[a-zA-Z_][a-zA-Z0-9_]*/.*/g'
}
for dash in "$DASH_DIR"/*.json; do
  [ -e "$dash" ] || continue
  title=$(jq -r '.title // "untitled"' "$dash")
  # panel title + expression pairs, including panels nested in rows
  jq -r '.. | objects | select(has("targets")) | "\(.title)\t\(.targets[].expr // empty)"' "$dash" |
  while IFS=$'\t' read -r panel expr; do
    [ -n "$expr" ] || continue
    clean=$(printf '%s' "$expr" | sanitize)
    count=$(curl -sG "$PROM_URL/api/v1/query" \
      --data-urlencode "query=$clean" | jq '.data.result | length')
    if [ "$count" -eq 0 ]; then
      echo "DEAD  dashboard=[$title] panel=[$panel]"
      echo "      expr=$expr"
      failures=$((failures + 1))
    fi
  done
done
echo "---"
echo "$failures dead panel(s) found"
exit 0

Stick this on a weekly cron job or scheduler. Pipe the findings directly into Slack or your team’s sprint board. If nobody looks at the report, you haven’t solved the problem; you’ve just automated the silence.

Put the gate in CI

Auditing finds the panels that rotted in the past. But what stops someone from opening a pull request tomorrow that introduces three new ghost queries? We put a bouncer at the door: a CI workflow check.

Before: two of three panels query names that no longer exist. After: the audit caught the renames, and the CI gate blocks the next ones.

Whenever an engineer submits a pull request updating dashboards, GitHub Actions runs this check. It validates that the JSON syntax is valid, verifies that no panel was committed with an empty query box, and executes the audit script against an accessible Prometheus instance (such as a staging cluster or a read-only production replica):

# .github/workflows/dashboard-lint.yml
name: dashboard-lint
on:
  pull_request:
    paths: ['dashboards/**/*.json']
jobs:
  check-dashboards:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: JSON must parse
        run: |
          for f in dashboards/**/*.json; do
            jq empty "$f" || exit 1
          done
      - name: Every panel target must carry a query
        run: |
          jq -e '[.. | objects | select(has("targets")) | .targets[]]
                 | all(.expr != null and .expr != "")' \
            dashboards/**/*.json
      - name: No panel may query a metric that does not exist
        env:
          PROM_URL: ${{ secrets.PROM_READ_URL }}
        run: ./scripts/dashboard-dead-panel-audit.sh ./dashboards \
               | tee audit.txt
          # fail the build if the audit flagged anything unexpected;
          # maintainers can explicitly allow-list pre-deployment dashboards
          ! grep -q '^DEAD' audit.txt || \
            (echo "dead panels found — see audit.txt"; exit 1)

The gate as a decision tree: three checks, one verdict — dead panels do not merge.

To keep your team from hating this check, set up three basic ground rules:

1. Support allow-lists for unreleased features: If an engineer is setting up observability ahead of next week’s v2 service deployment, those metrics will naturally be missing today. Provide a clean comment or allow-list mechanism so genuine pre-deploy panels can pass with explicit sign-off.

2. Query the right Prometheus environment: CI runners running against staging will flag metrics that only exist in production traffic. Hook this workflow up to a safe, read-only endpoint pointing at your production Prometheus inventory.

3. Bundle recording rules and dashboards together: Whenever someone modifies or deletes a recording rule, make it team policy that the corresponding dashboard PR must be merged in the same batch. If someone deletes a recording rule first, this CI gate will immediately yell at them — exactly as intended.

A dead panel is not a cosmetic bug. It is a missing instrument in the cockpit — and you only discover it when you reach for it mid-incident.

The takeaway

Your monitoring dashboards are software artifacts. They are code. And their queries are dependencies tied directly to a live database of metrics that changes constantly underneath them. Treat your dashboards with the same hygiene you give your backend services: test them in CI, audit them on a schedule, and make sure every board has a living, breathing team owner attached to it.

If your team is planning a Kubernetes v1.37 upgrade or tuning your Prometheus scrapers this quarter, run this audit right now. Native histograms change the metric inventory layout completely; running this check beforehand prevents you from walking into an incident with pitch-black monitors.

Run the script across your repository today. You will almost certainly find more dead panels than you expected, but fewer than you dread. Just remember: when you catch them, the panels won’t apologize. They’ll just sit there, perfectly calm, looking innocent while your house burns down.

• [ ] Export every dashboard to JSON (or confirm the provisioned files in git are the source of truth)

• [ ] Run the manual audit once: extract expressions, check each against the live inventory

• [ ] Fix or delete every confirmed dead panel — a deleted panel beats a lying one

• [ ] Schedule the audit script weekly; route its output to dashboard owners

• [ ] Add the CI gate on dashboard JSON changes, with an explicit allow-list for pre-deployment panels

• [ ] Pair recording-rule changes with their dashboard changes in the same pull request

• [ ] Name an owner for every dashboard; orphaned dashboards rot first

So here's my question for you: when you run the audit, how many of your panels turn out to be querying metrics that no longer exist?

Disclaimer

• This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company's intentions, policies, or internal decisions. Where I do not know something, the post says so.

• I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.

• Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.

• Nothing in this post is professional, financial, legal, or security advice. Examples are illustrative — validate in your own environment before running anything in production.

• Corrections are welcome: if a company named here publishes an official statement, or a detail above is wrong, I will update this post and note the correction. The goal is an accurate public record of my experience, not a verdict on anyone.

© 2026 Honey Kiran. All rights reserved.

Honey Kiran · DevOps & Platform Engineering

Writing about the unsexy plumbing that keeps production standing at 3am — monitoring, SRE, Terraform, and platform engineering.

Run this audit against your dashboard repos this week, and see how many ghost queries you dig up.

Examples are illustrative — validate in your own environment before running in production.