Alert Noise: Reducing False Pages

An alert that fires too often loses all credibility. Teams ignore it, then miss a real incident. The problem isn't the alert's sensitivity — it's its architecture.

The single-threshold trap

A threshold on raw error rate ("alert if error_rate > 1%") fires on everything: a 2-minute spike, a pod restart, a load test. These events consume less than 0.1% of your monthly budget — they don't deserve a page. A good alert doesn't measure whether something is wrong right now, it measures whether the current degradation pace threatens your SLO over the remaining window.

Burn rate as a noise filter

Burn rate measures how fast you're consuming your error budget relative to the "normal" pace. A rate of 1× means you're consuming exactly your budget in one window. A rate of 14.4× means you'll exhaust your monthly budget in 50 hours — a real emergency. Alerting only when burn rate exceeds a significant threshold naturally filters out micro-disruptions.

Single window vs dual window

A long window (e.g. 1h) smooths out short spikes and detects sustained burns. But a burn that ended 45 minutes ago can still trigger the alert — a delayed false positive. The solution: require TWO windows to be elevated simultaneously. The long window confirms the burn rate has been high on average; the short window (e.g. 5 min) confirms it's still high right now. If the incident resolved, the short window drops — no alert.

The Google SRE model

Google SRE published four default alert policies. Each pair defines a burn multiplier, a long window, and a short window. The first two trigger a page (immediate response); the last two create a ticket (planned action). This model covers both catastrophic incidents (budget exhausted in 1h) and slow degradations (budget exhausted in 30 days).