Design a Metrics Monitoring and Alerting System

A metrics monitoring and alerting system collects numeric time series (CPU, request rates, latencies, error counts) from thousands of hosts, stores them in a compressed time-series database, rolls older data up into coarser resolutions, and continuously evaluates alert rules that page the right on-call engineer. In an interview, strong answers focus on three things: a write path that absorbs hundreds of thousands of samples per second, storage that fits because of compression, and an alerting pipeline that sends one useful page instead of two hundred duplicates.

Specific numbers come from Prometheus, Datadog and Thanos documentation and Facebook's Gorilla paper; where a number is our own assumption, it is labeled as one.

1. Clarify the requirements and estimate the scale

Pin down scope first. A reasonable set to propose:

• Collect counters, gauges and histograms from hosts, containers and application code, each identified by a name plus labels, for example http_requests_total{service="checkout", status="500"}.
• Query them for dashboards and investigations, mostly over the last few hours.
• Alert when a rule is breached and notify email, chat or a paging tool.
• Retain full-resolution data briefly and reduced-resolution data for longer.

Say explicitly that logs and traces are out of scope. Then name the non-functional requirements that drive every later decision:

• Write-heavy. Facebook's Gorilla paper describes the read rate of its monitoring store as usually a couple of orders of magnitude lower than the write rate.
• Recent data matters most. The same paper reports that at least 85% of queries to Facebook's older monitoring store were for data from the past 26 hours.
• Alerting must stay up even when dashboards degrade. Losing a few data points is acceptable; missing an outage is not.

Back-of-the-envelope numbers

Our assumptions, for illustration only: 10,000 hosts, 500 active series per host, and a 15-second collection interval.

The takeaway to say out loud: compression keeps storage reasonable, and the number of active series, not raw disk, tends to be the first real limit, because every active series carries index and memory overhead. Prometheus's storage docs put compressed samples at about 1–2 bytes each and size disk as retention seconds × samples per second × bytes per sample.

2. Collection: push vs pull

This is the first real fork, and interviewers want the trade-off, not a verdict.

Two details that show depth:

• Push gateways are a narrow tool. Prometheus recommends its Pushgateway only in limited cases, mainly results of service-level batch jobs. Its docs warn that a shared gateway becomes a single point of failure, you lose the up health signal, and pushed series stay exposed until someone deletes them.
• Agents pre-aggregate. DogStatsD combines the points for each unique metric into one point per 10-second flush interval before submitting to Datadog, cutting network and ingestion load at the edge.

A strong answer is often a hybrid: an agent on each host collects local metrics, pre-aggregates, sends its own heartbeat, and pushes compressed batches to a central ingestion tier.

3. Ingestion and time-series storage

The write path

1. Ingestion gateway. Stateless, behind a load balancer; authenticates the tenant, validates names and labels, and enforces per-tenant rate and series limits.
2. Buffer. A partitioned log such as Kafka absorbs bursts (a deploy restarting every pod at once), decouples ingestion from storage, and allows replay after a crash.
3. Partition by series. Hash the metric name plus sorted label set into a series ID and shard on it, so every sample of a series lands on the same storage node. Compression depends on consecutive samples sitting together.
4. Storage writers consume their partitions and append to the time-series database, replicated to two or three nodes.

Why a time-series database

The data is append-only, rarely updated, and read as range scans over a few series. A row-per-sample relational table wastes space on row overhead and index maintenance. Time-series databases group samples by series and by time block instead:

• Head block plus write-ahead log. Prometheus keeps the newest samples in memory, protected by a WAL replayed after a crash, and persists them in two-hour blocks. Compaction later merges blocks into ones spanning up to 10% of the retention window or 31 days, whichever is smaller.
• Compression. Gorilla encodes timestamps as delta-of-delta (a perfectly regular interval costs one bit) and values as an XOR against the previous value. Facebook reported an average of 1.37 bytes per point, down from 16, a 12x reduction.
• Inverted index. Map each label pair to a postings list of series IDs. A query for service="checkout" and status="500" intersects two lists, then reads only the matching chunks.
• Tiered storage. Recent blocks on local SSD; older, immutable blocks in object storage, which is how systems such as Thanos extend Prometheus beyond one machine.

On replication, Gorilla wrote every series to two regions without guaranteeing consistency between them, accepting small discrepancies for availability. That fits here: metrics tolerate a lost point far better than a stalled write path. If the interviewer pushes on quorum and repair, the reasoning mirrors a distributed key-value store.

4. Downsampling and retention

A one-year chart over raw 15-second data would scan millions of points per series. Downsampling (rollup) stores coarser summaries of older data. An example plan:

For reference, plain Prometheus keeps 15 days when no retention flag is set, and the Thanos compactor downsamples raw data older than 40 hours to 5-minute resolution, then 5-minute data older than 10 days to 1-hour resolution. Real retention periods vary widely by team and budget.

• Store count, sum, min and max per bucket, not just the average. A five-minute average can smooth away a 30-second spike that the max would show, and count plus sum let you recompute averages at any coarser level.
• Percentiles cannot be averaged. The average of several p99 values is not a p99. Keep histogram bucket counters and compute percentiles from summed buckets at query time.
• Handle counter resets. Counters drop to zero when a process restarts; rollups must account for that or rates go negative.
• Downsampling is mainly about query speed. The Thanos docs state that keeping every resolution does not save space. You save storage only if raw data expires sooner than the rollups.

Rollups can run as a batch job over closed blocks or as a streaming job over the ingestion log; the windowing trade-offs match the ad click aggregator walkthrough.

5. The query layer

• Query frontend. Parses a PromQL-like expression, checks limits, and splits long ranges into aligned pieces (say, one per day) that run in parallel and can be cached.
• Resolution selection. The last hour reads raw data; the last 90 days reads 1-hour rollups. Return about as many points as the chart can draw.
• Fan-out and merge. Find matching series IDs in the index, query the shards that own them, and merge partial aggregates (sum of sums, max of maxes).
• Caching and precomputation. Cache completed time ranges but never the newest window. Precompute expensive expressions that many dashboards share, such as error rate per service; Prometheus calls these recording rules.
• Guardrails. Cap series touched, samples read and run time per query so one heavy dashboard cannot starve alert evaluation.

6. Alerting pipeline and deduplication

Treat alerting as its own pipeline with four stages.

1. Rule evaluation. An evaluator runs each rule on a fixed interval; Prometheus's global default is 1 minute. Example: 5xx error ratio for checkout above 5% for 10 minutes.
2. State machine. A breach puts the alert in pending; it becomes firing only if the condition holds for the whole for duration, which filters blips. Prometheus's keep_firing_for keeps an alert firing for a set time after the condition was last met, which helps prevent flapping and false resolutions caused by missing data.
3. Alert manager. A separate service deduplicates, groups, inhibits and silences firing alerts, then routes each group to a receiver. Prometheus's Alertmanager is the standard reference.
4. Delivery. Send to email, chat and paging integrations with retries and idempotency keys, essentially a small notification system.

How deduplication works

• Identity by labels. An alert's identity is its label set (rule name plus labels such as service and cluster), hashed into a fingerprint. Run two evaluator replicas; both send the same alert and the alert manager collapses them. Prometheus's docs advise pointing each server at the full list of clustered Alertmanager instances rather than a load balancer.
• Grouping. If hundreds of instances lose their database at once, group by cluster and alert name so on-call gets one notification. Alertmanager's defaults: wait 30 seconds before a new group's first notification, 5 minutes between updates to the group, and 4 hours before re-sending an unchanged one.
• Inhibition. Suppress symptom alerts while a cause alert fires: if a cluster is unreachable, do not also page for every service in it.
• Silences. Time-boxed, label-matched mutes for planned maintenance.

7. Cardinality limits

Cardinality is the number of unique time series. Prometheus's naming guide puts it directly: every unique combination of key-value label pairs is a new time series. Datadog counts custom metrics the same way; its example is a metric with an endpoint tag taking 2 values and a status tag taking 2 values, which counts as 4.

The multiplication is what hurts. A request counter with 1,000 endpoints, 5 status classes and 200 pods can reach 1 million series in the worst case. Add a user_id label with 100,000 users and the ceiling becomes 100 billion. Controls worth naming:

• Ban unbounded labels. Prometheus advises against labels such as user IDs or email addresses. Per-request detail belongs in logs or traces.
• Per-tenant active-series limits at the ingestion gateway, with a clear error, so one team's bad deploy cannot degrade everyone.
• Per-target limits. Prometheus's sample_limit and label_limit fail the entire scrape when exceeded, which is loud on purpose.
• Drop labels at the agent when only a service-level aggregate is needed, such as removing the pod name.
• Make cardinality visible with per-team top-series reports so owners see growth before hitting a limit.

8. Trade-offs to say out loud

A reasonable pacing for a 45-minute round: about 5 minutes on requirements and numbers, 10 on collection and ingestion, 10 on storage and rollups, 10 on alerting, and the rest on cardinality and the interviewer's follow-ups. Rehearse it out loud at least once; explaining delta-of-delta encoding under time pressure is harder than it looks. A free AI mock interview in CoPilot Interview lets you include system design in the question mix, answer out loud, and get a score on each answer with what worked and what to improve.

FAQ

Should a metrics monitoring system use push or pull?

Either is defensible. Pull, as in Prometheus, gives you a built-in health signal and lets the server control load, while push through a local agent, as with DogStatsD, suits short-lived jobs and hosts behind firewalls. Many designs combine them with a per-host agent that collects locally and pushes batches upstream.

Why not store metrics in a relational database?

Metrics are append-only, arrive at hundreds of thousands of samples per second, and are read as range scans over a few series. A time-series database groups samples by series and compresses them to roughly 1-2 bytes each, which a row-per-sample table cannot match.

What is cardinality in metrics monitoring?

Cardinality is the number of unique time series, where each unique combination of metric name and label values counts as one series. Unbounded labels such as user IDs or request IDs multiply it quickly, so many production systems enforce series limits at ingestion.

How does an alerting system avoid sending duplicate pages?

Each alert is identified by its label set, so identical alerts from redundant rule evaluators collapse into one. Grouping, inhibition and silences then cut related noise, and a repeat interval throttles re-sending an unchanged notification.

Does downsampling reduce storage costs?

Only if raw data is deleted sooner than the rollups. The Thanos documentation notes that keeping every resolution adds blocks rather than saving space; the main benefit is fast queries over long time ranges.