Your Prometheus Is Hoarding. Time for a Funeral Policy.

Every dead metric is a bill you’re still paying.

I know, you’re tired. Tired of dashboards that load like they owe you money, of PromQL autocomplete offering you three names for the same counter, of a TSDB bill that grows while your actual traffic doesn’t. Here is the short version of a very long problem: your Prometheus is hoarding.

Somewhere in your TSDB there are thousands of series that no dashboard queries, no alert touches, and no human has looked at in months — and every single one of them is still collecting rent in memory, disk, and cardinality.

The vendors just admitted it. Grafana Labs announced that customers using its Adaptive Telemetry suite cut telemetry spend by 30–50% on average — and buried in the same announcement is the industry’s biggest confession: most organizations already collect far more data than they ever query, alert on, or act upon. Grafana CTO Tom Wilkie put it bluntly: “The economics of observability break down when every new AI workload just adds more data to store and more noise to sort through. Adaptive Telemetry exists so cost scales with the value of your data, not the volume of it.” Translation: even the people selling you observability agree your TSDB is an attic. Their fix is a paid feature that identifies low-value data and drops it. This post is the free, vendor-neutral version of the same idea — a lifecycle policy with four verbs: adopt → audit → deprecate → delete.

This is not a monitoring problem. It is a lifecycle problem. Metrics are born every day — new features, new exporters, new histograms — and nothing in the default toolchain ever kills them. Prometheus will scrape whatever you point it at, keep it for exactly as long as you configured (15 days by default, then off a cliff), and never once ask the obvious question: does anybody still want this?

Here’s the menu for this post:

• What a dead metric is, and the four ways metrics die

• How to find them: the series inventory and the query-log diff

• A naming taxonomy so you stop minting new ones

• Retention tiers: hot, warm, cold — matched to the question each answers

• The full adopt → audit → deprecate → delete pipeline, with the two config gates that enforce it

Save this one. You will want it open the next time someone ships a histogram named stuff_time2.

What a dead metric actually is

A dead metric is a series — or a whole metric name — that is still being stored but no longer serves any consumer. Not broken. Dead. The distinction matters: broken metrics get fixed, dead ones get a funeral nobody schedules.

• The rename — How it dies: someone renames the metric in the instrumentation; the old name stops being emitted. Symptom: two names for one thing; the old one goes stale. How you spot it: old name absent from the live inventory; new name present.

• The orphan — How it dies: exporter or target decommissioned; its series stop cold. Symptom: every series from that job goes stale at once. How you spot it: up{job="..."} disappears; targets endpoint shows it gone.

• The experiment — How it dies: a debug metric ships “temporarily” and never leaves. Symptom: series with a handful of label values, never queried. How you spot it: present in the inventory, absent from 90 days of query logs.

• The duplicate — How it dies: the same thing measured two ways by two teams. Symptom: two names, one truth, double the cardinality. How you spot it: inventory search surfaces near-identical names.

Notice the pattern: every one of these is a naming or ownership failure wearing a storage costume. The rename is the most common. Picture it (illustrative): a developer renames http_request_duration_seconds to http_requests_duration_seconds on a Friday afternoon, updates the dashboards, and moves on. The old name goes stale within minutes — and then lives in the TSDB for the rest of its retention, queried by nothing, owned by no one, billed to everyone.

Every metric you ship is a bill you keep paying — in memory, in disk, in cardinality — until somebody deletes it.

Find them: run the series inventory

You cannot delete what you cannot see. Step one is the inventory — every metric name your Prometheus currently knows. I did the reading so you don’t have to; the endpoints below are straight from the official Prometheus docs.

# every metric name in the local TSDB, one per line
curl -s http://prometheus:9090/api/v1/label/__name__/values \
  | jq -r '.data[]' | sort -u > live-metrics.txt
wc -l live-metrics.txt

# the blunt version: which names had zero samples in the last 30 days?
# (illustrative shape of the check — run it off-hours or against a replica,
#  since it issues one query per metric name)
for m in $(cat live-metrics.txt); do
  n=$(curl -sG http://prometheus:9090/api/v1/query \
    --data-urlencode "query=count_over_time({__name__=\"$m\"}[30d])" \
    | jq '[.data.result[].value[1] | tonumber] | add // 0')
  [ "$n" = "0" ] && echo "STALE: $m"
done

Two smarter signals if you have them. The TSDB stats endpoint (/api/v1/status/tsdb) shows head series counts and the top metric names by series count — the cardinality hogs announce themselves. And your query logs, if you ship them anywhere (Prometheus can log queries with the --query.log-file flag), tell you which names anyone actually asked for in the last 90 days. The dead ones are the names in the inventory that appear in no query log. That diff is the entire audit.

The fix: run the inventory once by hand this week. The number will be higher than you expect — and every stale name is a candidate for the lifecycle below.

Naming conventions: stop minting new ways to die

Most dead metrics are born dead — named so vaguely at instrumentation time that nobody can find them, trust them, or tell them apart from their three siblings. myapp_stuff_time2 was never going to survive contact with a second team.

One formula for every name. If it cannot be parsed, it does not get instrumented — the cheapest gate in this post.

The rule that makes this stick: a metric name that cannot be parsed by the formula does not get instrumented. Put it in the contributing guide. Lint it in CI if you are ambitious. The formula costs nothing; the renames it prevents are the expensive ones, because every rename manufactures one dead metric and one live one.

The fix: adopt the taxonomy before your next instrumentation PR — not after your next round of renames. There is no incident in this story. That is the point.

Retention tiers: not everything deserves forever

The default Prometheus setup keeps everything for 15 days and then drops it off a cliff (that’s the --storage.tsdb.retention.time default, straight from the storage docs). That is not a retention policy — it is a timer. A policy matches the storage to the question the data answers.

Hot answers “is it broken right now?”, warm answers “how are we trending?”, cold answers “prove it happened.” Most setups keep everything hot. Flip it.

Note what the keep-list does in this picture: it is the bouncer at the door between hot and warm. Only series on the list leave the cluster. Everything else lives fast and dies young in the local TSDB — which is exactly what a local TSDB is for.

A metric nobody queries is not monitoring. It is a souvenir.

The lifecycle: adopt → audit → deprecate → delete

Here is the whole policy as a pipeline. Every series enters on the left; only the living ones survive to the right:

Six stages, two gates, one feedback loop. The audit is the only stage most teams skip — and it is the stage the whole thing pivots on.

Walk it once. 1. Instrument — under the naming convention, so names arrive parseable. 2. Scrape — with relabel rules that drop junk before it ever touches disk. Metric relabeling applies to samples as the last step before ingestion, per the official config docs — so this is the earliest, cheapest place to say no:

scrape_configs:
  - job_name: "api"
    static_configs:
      - targets: ["api:8080"]
    metric_relabel_configs:
      # drop high-cardinality debug series at scrape time —
      # never stored, never billed, never mourned
      - source_labels: [__name__]
        regex: "debug_.*"
        action: drop

3. Store — short local retention. The TSDB is a cache, not an archive; treat it like one. 4. Audit — the inventory-vs-query-log diff, on a schedule, with an owner who reads the report. 5. Deprecate — mark the name, notify the owner, set a deadline. A deprecation without a deadline is just a rumor. 6. Delete — drop the series, remove the rules, close the loop.

And the bouncer at the remote-write door — the keep-list that decides what leaves the cluster. Write relabeling runs on samples before they are sent to the remote endpoint, and exists precisely to limit what goes out:

remote_write:
  - url: "https://<long-term-storage>/api/v1/write"
    write_relabel_configs:
      # keep-list: only these names leave the cluster
      - source_labels: [__name__]
        regex: "(job:.+|http_requests_duration_seconds|node_.*)"
        action: keep

Two gates, both in config, both reviewable in a pull request. That is the entire enforcement mechanism. The paid vendor feature doing this job is the same audit with a price tag; this one runs in YAML.

Deletion is a feature. The delete button is the most underused reliability tool in the building.

The audit does not make your system smaller. It makes your system honest — and honest systems are cheaper to run. (Bars illustrative; the shape of the problem is not.)

The takeaway

Your TSDB is an attic. Attics are fine — until you cannot find the thing you actually need because it is buried under eleven boxes of cables for devices you no longer own. The lifecycle is scheduled attic-cleaning with a bouncer at the door: adopt names worth keeping, audit what you have, deprecate with a deadline, delete without ceremony.

Start this week. The whole program fits on an index card:

• [ ] Run the inventory query — count your metric names and sit with the number

• [ ] Diff the inventory against 90 days of query logs — list the names nobody reads

• [ ] Adopt the naming taxonomy — add it to the contributing guide

• [ ] Set retention tiers: 15 days hot, 13 months warm behind a keep-list, 2+ years cold

• [ ] Add scrape-time drop rules for debug and experiment series

• [ ] Add the remote-write keep-list — nothing leaves the cluster by accident

• [ ] Schedule the audit; name one owner for the report

• [ ] Delete something this week — start with one name and feel the power

So here's my question for you: how many of your metric names has nobody queried in the last 90 days, and which one will you delete first?

Disclaimer

• This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company's intentions, policies, or internal decisions. Where I do not know something, the post says so.

• I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.

• Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.

• Nothing in this post is professional, financial, legal, or security advice. Examples are illustrative — validate in your own environment before running anything in production.

• Corrections are welcome: if a company named here publishes an official statement, or a detail above is wrong, I will update this post and note the correction. The goal is an accurate public record of my experience, not a verdict on anyone.

© 2026 Honey Kiran. All rights reserved.

Honey Kiran · DevOps & Platform Engineering

I write about the unsexy work that keeps systems alive at 3am — monitoring, SRE, Terraform, and platform engineering, drawn from real production work.

New posts regularly. Follow along — and send this to the teammate who ships a new histogram every sprint.

Examples are illustrative — validate in your own environment before running in production.