Prometheus Cardinality: Avoiding Time Series Explosion

Prometheus stores one time series per unique combination of metric name and labels. A poorly chosen label turns a lightweight metric into a RAM-eating monster. Understanding cardinality means understanding how Prometheus sees your world.

What is a time series?

A time series is the unique identifier formed by a metric name and a set of label=value pairs. http_requests_total{method="GET", status="200"} and http_requests_total{method="POST", status="500"} are two distinct series. Prometheus stores, indexes, and compresses each independently. Every scrape adds a new data point to every active series.

The combinatorial explosion

Total cardinality is the product of each label's cardinality. A metric with method (5 values) × status (50 values) × endpoint (200 routes) × region (3 zones) × pod (100 pods) produces 5 × 50 × 200 × 3 × 100 = 150,000,000 series. A single user_id label with 10 million active users gives 10 million series for that one metric alone.

Memory and disk impact

Prometheus keeps a "head block" in RAM for recent data (typically 2h). Each active series takes about 1–2 KB (metadata, index, compressed data). 1 million active series = 1–2 GB of RAM just for the head block. Disk storage is around 1–2 bytes per sample with Gorilla compression: at 15s interval, that's ~11 KB per series per day.

The usual suspects

Some labels are almost universally dangerous: pod (recreated on every deploy, ever-growing cardinality), user_id, request_id, trace_id, IP address. Kubernetes often injects these automatically via service discovery. Histograms multiply cardinality by the number of le buckets (typically 10–15). The golden rule: if a label value could be a unique identifier, don't use it as a label.

Reduction strategies

Aggregate at scrape time rather than after. Use recording rules to pre-compute frequent aggregations. Limit labels via allow-lists in Prometheus jobs. Native histograms (Prometheus 2.40+) dramatically reduce histogram cardinality. Define a "cardinality budget" per team and monitor it via the prometheus_tsdb_head_series metric.