System Design Interview: How Would You Design a Metrics Monitoring and Alerting System?

Problem and Applicable Scenarios

Design a metrics monitoring and alerting system shared by multiple engineering teams. It monitors 50,000 service instances, each exposing 200 active time series, and collects every 15 seconds. It supports counters, gauges, and histograms; filtering and aggregation by labels; dashboards; and rule-based alerts. A new failure should enter notification routing within 60 seconds at p99. Common dashboard queries over the most recent six hours should complete within two seconds at p95. Raw samples are kept for seven days, one-minute rollups for 90 days, and one-hour rollups for 13 months.

Assume one region with three availability zones and trusted authentication as the source of tenant identity. Existing email, chat, and phone providers may deliver notifications; this design must reliably produce, deduplicate, group, and route them. Logs and traces are outside the primary scope. The interview should focus on high-cardinality labels, duplicates and late samples, missing data, alert state, query isolation, and detection of a failure in the monitoring platform itself.

Current English and Chinese material from 2026 presents metrics monitoring and alerting as a direct system design interview problem involving time-series ingestion, querying, alerts, and high availability. Prometheus documentation supplies primary semantics for series identity, pull collection, the WAL, pending and firing alerts, grouping, and inhibition. Google SRE supplies principles for low-noise alerts and black-box monitoring. The prompt is therefore both currently representative and technically verifiable.

What the Interviewer Is Evaluating

First, can the candidate separate sample throughput from active time-series cardinality? Instance count, series per instance, and collection interval determine writes. Memory, indexes, and query fan-out may fail first because of cardinality. Adding userid or requestid to labels can create orders of magnitude more series without increasing traffic.

Second, do durable ingestion, dashboard queries, and failure detection form a complete path? A collector receiving a sample does not mean that the sample is durable. Alert rules should not depend on a shared pool that expensive ad hoc queries can exhaust. A strong answer defines the acknowledgement boundary, reserves capacity for alerts, and specifies late, duplicate, missing, and partial-result semantics.

Third, does storage match the access pattern? Recent data is appended and queried frequently. Historical data fits immutable blocks, compression, indexes, and object storage. Downsampling cannot average averages. It should retain at least sum, count, min, and max; histograms require compatible bucket boundaries before they can be merged.

Finally, the alerting system must survive incidents. An incident can increase error metrics, queries, and notifications at once. The candidate should cover rule sharding, state recovery, notification deduplication, alert storms, noisy tenants, self- monitoring, and an end-to-end liveness probe outside the platform's failure domain.

Clarifying Questions Before Answering

• Are collection targets stable? Most are long-running services discovered by the platform; short jobs and restricted networks use a push gateway.
• Is the stated scale global or regional? Treat it as a regional peak. Additional regions collect independently and expose controlled global queries.
• What is the acknowledgement boundary? The write gateway succeeds only after the sample enters a durable log replicated across availability zones.
• Are duplicates and out-of-order samples allowed? Network retries may duplicate data, and samples may arrive within a bounded late window. Equal series and timestamps use a deterministic conflict rule.
• Does a missing sample mean zero? No. It may indicate a dead target, failed collection, or partition, which differs from a measured zero.
• Must every query finish in two seconds? No. The SLO covers bounded, common six-hour queries. High-cardinality and month-long raw scans have quotas or use asynchronous analysis.
• How is alert recovery decided? The rule defines evaluation interval, hold duration, missing-data policy, and recovery condition. The notification service does not infer them.
• May tenants create arbitrary labels? Within quotas, subject to label count, length, active-series, and new-series-rate limits.
• Is exactly-once notification required? No. Delivery is at least once, with idempotency based on an alert fingerprint and state transition.
• How do raw and rolled-up data relate? A rollup can be recomputed while source blocks remain retained, and records its resolution and coverage watermark.

30-Second Answer Framework

“The scale is 10 million active series, about 667,000 samples per second, or 57.6 billion samples per day. Sharded collectors pull targets from service discovery, while short jobs use a push gateway. Normalized samples enter a cross-zone durable log by tenant and series fingerprint, then a recent-data tier and immutable time blocks. Label indexes serve queries, and cardinality budgets constrain dangerous labels such as user_id. A separately provisioned rule engine evaluates fixed timestamps and keeps inactive, pending, and firing states. An alert manager deduplicates, groups, silences, inhibits, and routes alerts. I would verify the path with a tenfold new-series surge, duplicate and late samples, zone failure, and an external canary.”

Step-by-Step Deep Dive

Start with six invariants: sample labels cannot declare tenant identity; acknowledged samples have entered a durable log; one retry cannot produce a different result; missing data is never silently changed to zero; dashboard load cannot starve alerting; and every rejection, drop, delay, and degradation has an observable count.

Step one: recompute the scale and treat cardinality as a first-class resource.

The active-series calculation is:

50,000 instances × 200 series/instance = 10,000,000 active series
10,000,000 ÷ 15 seconds ≈ 666,667 samples/second
666,667 × 86,400 ≈ 57.6 billion samples/day

Prometheus storage documentation gives a rough local-storage starting point of one to two bytes per sample. At that rate, compressed sample chunks would occupy about 57.6–115.2 GB per day, or 172.8–345.6 GB per day with three copies. This estimate excludes label indexes, the WAL, the head, object metadata, safety margin beyond replicas, and observed distribution effects. It is a sanity check, not a substitute for a benchmark with representative metrics.

New series are more dangerous. If a request-level user_id enters the label set, series identity changes with every value. The platform must limit active series, new-series creation rate, label count, label length, and allowed keys per tenant and metric, with explicit warn, quarantine, and reject policies.

Step two: use pull collection by default and push where necessary.

Service discovery supplies the target set. A control plane assigns targets to collectors with consistent or rendezvous hashing. Every 15 seconds, collectors pull over HTTP, attach trusted tenant, cluster, job, and instance fields, and produce collection metrics such as up, scrape duration, sample count, and errors. Pulling makes “the target existed but collection failed” directly visible and lets the platform control pace.

Short-lived batch jobs, networks without inbound access, and environments that already emit OTLP use a regional gateway. The gateway authenticates identity, limits batch size, normalizes fields, and enters the same ingestion contract. A client's send success is not final storage success; gateway success means the platform reached its durable boundary. Collectors use bounded local spools during disconnection and reject with counters when space is exhausted rather than filling disks indefinitely.

Step three: define series identity and the ingestion contract.

A series is uniquely identified by tenantid + metricname + canonical_labels. Labels are sorted and canonically encoded before hashing. A fingerprint collision still requires comparing the full identity; the hash alone is insufficient. A sample contains series identity, source time, observed time, value or histogram, data type, and collection source.

WriteBatch {
  tenantId, sourceId, requestId,
  samples: [{ metric, labels, sourceTimestamp, value }]
}

The gateway performs authentication, label normalization, type validation, cardinality budgeting, time-range checks, and batch limits. It then partitions by tenantid + seriesfingerprint into a cross-zone replicated log. Log commit is the ack boundary. The same requestId can be retried safely, and storage deduplicates by (seriesid, sourcetimestamp). Different values at one series and timestamp follow a fixed conflict policy and increment a conflict metric; arrival order must not silently decide.

Step four: separate recent data, historical blocks, and indexes.

An appendable head holds the most recent hours and permits bounded changes for late data. Background workers freeze it into immutable, time- and shard-partitioned blocks, use timestamp-delta and value compression, then upload them to object storage. Block metadata contains minimum and maximum time, tenant, resolution, checksum, and coverage watermark. Compaction merges small blocks and removes only blocks completely covered by validated replacements.

An inverted index maps label key/value pairs to series IDs, which locate data blocks. Hot label and series metadata may be cached, but tenant identity is part of every cache key. Downsampling creates one-minute and one-hour blocks with sum/count/min/max. Counters require reset handling, and histograms merge only with compatible bucket schemas. Raw blocks last seven days, one-minute blocks 90 days, and one-hour blocks 13 months.

Step five: make query cost predictable.

The query coordinator parses the time range, label selectors, aggregation, and step. It authorizes first, uses the index to find series, and reads relevant blocks in parallel. A rollup is selected only when the requested step and function allow it. An average is recomputed from sum/count; an existing percentile cannot be rolled up into another correct percentile. Recent queries merge the head with persisted blocks and deduplicate at a shared watermark.

Each query has limits for series, scanned points, concurrency, memory, and deadline. Common dashboards use fixed steps, pre-aggregation, and short result caching to target two seconds at p95 over six recent hours. Month-long high-cardinality exploration may become an asynchronous job. When a shard fails, the API returns partial=true and the missing range. Alert rules reject partial results by default.

QueryRange {
  tenantId, expression, start, end, step, maxSeries
}
QueryResult { data, resolution, watermark, partial, warnings }

Step six: separate rule evaluation from notification routing.

A scheduler shards rule groups by tenant and evaluates every 15 or 30 seconds at an explicit evaluation timestamp. An alert instance has a stable fingerprint derived from the rule and result labels, and moves through inactive -> pending -> firing. A for duration filters transient spikes. keepfiringfor or explicit hysteresis prevents brief missing data from causing repeated resolutions. Every rule also declares whether missing data is normal, alerting, or unknown.

Two highly available evaluators may send the same fingerprint to alert managers, which deduplicate it. Alert managers route by team, service, and severity; group instances from one incident; inhibit downstream instance alerts when a cluster is unreachable; and apply expiring silences during maintenance. Notification state lives in replicated storage. Delivery is at least once and uses alertfingerprint + statetransition + receiver as an idempotency key. Rule computation, state, and notification queues use capacity isolated from ad hoc dashboard queries.

Step seven: handle failures, tenant isolation, and self-monitoring.

When a collector dies, target leases move to healthy instances. Brief overlapping collection is absorbed by deduplication. When the log or storage applies backpressure, gateways reduce batches and reject explicitly instead of building an unbounded memory queue. During an object-store outage, the replicated log and bounded head keep accepting data and block deletion stops. Beyond the safe watermark, tenant quotas protect alert-critical metrics. Query failure must not stop rules from reading a confirmed watermark.

Tenant identity comes from mTLS or a server-issued token, never a tenant sample label. Separate limits apply to ingestion, queries, and rules. RBAC controls dashboards and alert configuration, and changes to rules, silences, and routes are audited. The platform observes ingestion latency, log backlog, head watermarks, block age, scanned query points, rule-evaluation delay, pending and firing counts, and notification failures.

Self-monitoring shares the platform's failure domain, so an external black-box canary continuously writes a known metric from an independent environment, waits for its rule to fire and its notification to arrive, then resolves it. An external dead-man check uses an independent notification path if expected heartbeats stop. This detects a completely silent monitoring platform.

Step eight: validate promises with load and fault injection.

Steady-state load must reach at least 667,000 samples per second, followed by a twofold burst. Then generate a tenfold new-series rate and verify that the offending tenant is isolated while other tenants and rule evaluation still meet their SLOs. Correctness tests cover duplicate batches, conflicting values at one timestamp, the late window, counter resets, histogram merges, rollup recomputation, and equal results before and after block compaction.

Fault tests kill a collector, one zone's log replica, a head node, a query shard, a rule evaluator, and a notification provider. They verify that acknowledged samples remain, lease overlap deduplicates, rule state recovers, and notification retry cannot grow without bound. Finally, measure canary latency across ingestion, rule hold duration, group wait, and delivery. Confirm 60 seconds at p99 or identify the stage that consumed its budget.

Strong Sample Answer

“I would first multiply 50,000 instances by 200 series to get 10 million active series. Dividing by 15 seconds gives roughly 667,000 samples per second, or 57.6 billion per day. Capacity planning must include both sample chunks and label indexes. Active- series and new-series-rate budgets prevent labels such as user_id from exhausting the platform.

The main path uses service discovery and sharded pull collectors; short jobs use a push gateway. After authentication, label normalization, and quota checks, samples enter a cross-zone replicated log by tenant and series fingerprint. Only a committed log write is acknowledged. Recent data enters a head, then freezes into immutable compressed blocks and a label inverted index in object storage. Background jobs produce one-minute and one-hour rollups; averages are rebuilt from sum/count.

A query coordinator prunes blocks by time, labels, and resolution, with budgets for series, points, memory, and deadlines. A separately provisioned rule engine evaluates fixed timestamps and preserves inactive, pending, and firing state. Alert managers deduplicate by fingerprint, then group, inhibit, silence, and route. Missing data follows an explicit rule policy, and notifications use at-least-once delivery.

I would load test 667,000 samples per second, a twofold burst, and a tenfold new-series surge. Then I would inject duplicate and late samples, a zone failure, a failed query shard, and a failed notification provider. A canary in an independent environment continuously exercises ingestion through notification, verifying the 60-second p99 and detecting complete platform silence.”

Common Mistakes

• Calculating only samples per second → Label indexes and active series may exhaust memory first → Also budget 10 million active series and the new-series rate.
• Allowing arbitrary user_id labels → Every value creates another series → Apply label policy, cardinality budgets, and quarantine.
• Acknowledging on request receipt → A process or zone failure can lose acknowledged data → Ack only after replicated-log commit.
• Treating missing as zero → Collection failure appears to be a business drop to zero → Preserve stale or unknown state and let rules choose a policy.
• Averaging averages → Buckets with different counts produce a biased result → Retain sum/count and recompute.
• Promising every query in two seconds → Unbounded labels and time ranges have unbounded cost → Scope the SLO and enforce query budgets and async paths.
• Letting alert rules accept partial results → One missing shard may look like recovery → Require a complete watermark or enter unknown state.
• Notifying for every instance → A large outage creates an alert storm → Group incidents and use inhibition, silences, and rate limits.
• Claiming exactly-once notification → A timeout cannot prove whether delivery happened → Use at-least-once delivery with stable idempotency keys.
• Monitoring the platform only with itself → A total outage produces no signal → Add an independent black-box canary and dead-man path.

Follow-Up Questions and Responses

Follow-up 1: Why not let every client push directly?

Long-running services fit pull collection: the platform knows the target set and frequency, distinguishes a measured value from a failed collection, and attaches trusted resource labels consistently. Short jobs, restricted networks, and existing OTLP clients fit push. Both entrances eventually use the same validation and durable-ingestion contract, avoiding two query semantics.

Follow-up 2: How do you control cardinality without blocking legitimate workloads?

Measure both stock and flow: limit active series per tenant and metric, and limit new series per minute. Offer a cardinality preview before rollout. Warn near the threshold, then drop a configured dangerous label, quarantine the metric, or reject new series according to tenant policy. Retain drop counts and representative diagnostics, but do not copy sensitive raw labels into ordinary logs.

Follow-up 3: How do late samples affect alerting?

Rules read a confirmed watermark at a fixed evaluation timestamp and may add a small delay for normal lateness. Older samples arriving after that watermark can update historical queries, but should not retract a human notification by default. If the product needs corrections, emit a versioned correction event. The late window, maximum clock skew, and conflict rule must be documented and tested.

Follow-up 4: Why not store percentiles directly?

The p95 values from several instances or time buckets cannot be averaged or passed through another p95 operation to obtain the global p95. Store mergeable histogram buckets or sketches, merge the distributions at query time, and then calculate the percentile. Incompatible bucket boundaries or sketch parameters require separate results or an explicit migration.

Follow-up 5: How do highly available rule engines avoid duplicate notifications?

Both evaluators may compute and send the same alert instance. The instance fingerprint is stable across the rule and result labels. Alert managers use replicated state to deduplicate, group, and route by fingerprint, state, and receiver. Evaluation need not become a singleton. A partition may still produce a rare duplicate, so receivers retain the same idempotency key.

Follow-up 6: What happens when object storage is unavailable?

Continue writing new samples to the replicated log and bounded head. Pause block upload, compaction deletion, and acceptance beyond safe watermarks. Protect alert-critical metrics with tenant quotas. If recovery cannot happen within log retention, reject new writes explicitly and notify operators; never claim that all data remains accepted.

Follow-up 7: How would global multi-region queries work?

Each region collects and alerts independently so a wide-area partition cannot silence local alerts. A global coordinator reads published immutable blocks or controlled query endpoints and returns each region's watermark and partial status. The small set of truly global alert metrics is aggregated regionally first, then replicated as low-cardinality results into a separate rule domain.

Follow-up 8: How do you find which stage broke the 60-second alert target?

For canary samples, record source time, observed time, log-commit watermark, rule-evaluation time, pending interval, grouping wait, and notification receipt. Break the end-to-end histogram down by stage and verify final arrival externally. A rule's five-minute for clause is a product condition, not platform detection latency, so report platform processing and configured hold duration separately.