System Design Interview: How Would You Design a Global Content Delivery Network?

Problem and Applicable Scenarios

Design a multi-tenant pull content delivery network with 50 points of presence (PoPs). At peak it receives 2 million GET and HEAD requests per second, and the average cacheable GET response is 256 KiB. Customer origins can safely accept 20,000 fetches per second in total. The CDN serves content-hashed assets, images, video segments, and mutable documents at stable URLs.

The target is a p99 time to first byte below 50 milliseconds for a cached response under normal conditions, 99.99% request availability, and application of each purge at 99% of healthy PoPs within 60 seconds. Complete propagation and lagging PoPs must remain observable. These values are interview assumptions, not claims about any provider.

This design applies when users are geographically distributed, origin distance dominates latency, repeated objects can be reused, and origin bandwidth or compute is limited. A private internal service with one region and little reuse may need a regional reverse proxy instead. The interview scope builds the core CDN mechanisms; buying a managed CDN remains a valid production choice after the requirements, security boundary, operating cost, and provider failure model are compared.

What the Interviewer Is Evaluating

The first signal is whether the candidate separates the control plane from the data plane. Tenant onboarding, origin configuration, certificates, cache rules, and purge commands need durable management workflows. Serving requests must continue from last-known-good configuration when that management path is unavailable.

The second signal is correctness of the cache boundary. A cache key that omits a representation dimension can leak or corrupt responses across languages, encodings, devices, or tenants. A key that includes every cookie and header fragments the cache until almost every request misses. The answer must define which requests are eligible, which dimensions vary the representation, and which private responses bypass shared storage.

The third signal is origin protection. With 50 PoPs, one newly popular object can create 50 simultaneous cold fills even before retries. Request coalescing at each tier, regional shields, bounded origin concurrency, and retry budgets solve different parts of that path. No failure mode may silently convert 2 million edge requests per second into origin traffic.

The fourth signal is invalidation correctness. A purge that only deletes current bytes can race with an older fill and allow stale content to reappear. Strong answers use an ordered generation or tombstone, make delivery idempotent, and measure both the common propagation SLO and the long tail.

Finally, the candidate should quantify throughput, state explicit consistency and staleness contracts, cover failure and security boundaries, and propose tests that prove user-visible behavior. A vendor-product list does not provide that proof.

Clarifying Questions Before Answering

• Does the CDN own the source objects? No. It is a pull CDN; customer origins remain authoritative.
• Which methods are cacheable? Start with GET and HEAD. Unsafe methods pass through and are never put in a shared cache by this baseline.
• Which content is private? Requests with authorization or user-specific cookies bypass shared caching unless a tenant supplies a reviewed, explicit partitioning policy.
• How fresh must mutable content be? Each route defines a TTL and any bounded stale window. A purge targets urgent changes; it is not a substitute for an authorization or revocation check.
• Is every query parameter meaningful? No. The tenant allowlists parameters that change the representation and can drop known tracking parameters after canonicalization.
• Are byte ranges required? Yes for large media. The key and metadata must distinguish complete objects from validated ranges, and the origin must provide stable validators or versioned URLs.
• How is a nearby PoP chosen? DNS and/or anycast route the user to a reachable PoP. BGP path selection is not a guarantee of geographic proximity, so health and measured latency remain necessary.
• What happens when the control plane fails? Existing traffic continues with a signed last-known-good configuration; unsafe changes fail closed and queue for later application.
• Can stale data be served during an origin outage? Only for routes with an explicit stale-if-error allowance and a bounded maximum age.
• What does purge completion mean? The 60-second target covers 99% of healthy PoPs; the system separately records every acknowledgement, laggard, retry, and probe result.

30-Second Answer Framework

“I split control and data planes. DNS and anycast steer to a healthy PoP; the edge selects a signed tenant configuration, canonicalizes an allowlisted key, and checks RAM then SSD. Misses coalesce at the edge and regional shield before a budgeted origin fetch; stale serving follows explicit limits. Immutable assets use versioned URLs. Mutable URLs use an ordered purge generation and tombstone, and fills compare generations before publication so old bytes cannot return. I verify request and byte hit rates, origin load, p99 hit TTFB, stale age, purge lag, and outage behavior.”

Step-by-Step Deep Dive

Step 1: Quantify the traffic and origin budget

Using the average GET size as a planning bound for the peak mix, response bandwidth before protocol overhead is:

2,000,000 requests/s × 256 KiB × 8 = 4.19 Tb/s

If that peak were sustained for a day, it would represent about 45.3 PB of edge response bytes. Across 50 PoPs the simple average is 40,000 requests/s and 10.5 GB/s per PoP, but real traffic is geographically and temporally skewed. Capacity planning therefore uses measured per-PoP peaks, headroom, object-size percentiles, and failure redistribution; the average is only a baseline.

The origin limit is 1% of edge request peak:

20,000 / 2,000,000 = 1%

That does not mean the edge hit-rate target alone is 99%. Shield hits, uncacheable routes, fills, revalidation, and retries all consume the same origin budget. The origin-facing scheduler needs per-tenant, per-origin, and global concurrency and request-rate limits.

Step 2: Separate control and data planes

The control plane stores tenants, domains, origin identities, certificates, cache policies, canonicalization rules, stale limits, signed-URL keys, and configuration versions. A validated change is committed durably, compiled into a signed snapshot, and distributed through a versioned stream. PoPs acknowledge applied versions. Certificate private keys use a dedicated secret and key management boundary and stay out of ordinary configuration storage.

The data plane handles TLS, tenant lookup, policy enforcement, request normalization, caching, origin access, and logs. It never synchronously calls the control-plane database on a cache hit. PoPs retain a last-known-good signed snapshot through a control-plane outage. A stale configuration has a bounded lifetime; expired certificates, revoked tenants, and ambiguous security policy fail closed after that lifetime.

Step 3: Route traffic and isolate tenants

DNS can return regional names or anycast addresses; an anycast network can announce the same address from multiple PoPs. Routing chooses a reachable network path, then service health removes a bad PoP and drains it to another location. The design tracks route changes, failover load, and latency because “nearest” is an observed outcome, not a BGP guarantee.

At the edge, SNI and the normalized Host map to one tenant before any cache lookup. The tenant ID is an implicit first component of every cache namespace. Origins accept only authenticated CDN traffic through mTLS, signed requests, private connectivity, or a rotating secret, and should not remain publicly bypassable. Origin addresses and redirects are allowlisted to prevent SSRF.

Step 4: Define cache eligibility and the key

The baseline admits successful GET and HEAD responses only when route policy and HTTP fields allow shared reuse. private, no-store, authorization, user-specific cookies, Set-Cookie, and unsupported Vary values normally bypass storage. Negative responses may be cached only for a short, status-specific TTL so a transient failure does not become a long outage.

A conceptual key is:

tenant_id | canonical_scheme_host | normalized_path | selected_query |
encoding_variant | approved_vary_dimensions | object_generation

Canonicalization happens once before policy, lookup, logging, fill, and purge. Only query parameters and headers that truly change bytes enter the key. Adding Accept-Language is correct when the origin varies by language; adding arbitrary cookies or User-Agent can explode cardinality. A response's Vary must agree with the route's allowed dimensions, or the response bypasses caching.

Freshness follows the tenant policy and HTTP semantics: fresh entries return directly; stale entries revalidate with ETag or Last-Modified; stale-while-revalidate and stale-if-error are used only within explicit bounds. no-cache means revalidate before reuse, while no-store means do not store. Cache policy must not weaken an origin's stricter privacy directive.

Step 5: Build RAM, SSD, shield, and origin fill paths

Each PoP keeps hot metadata and small objects in RAM and a larger admission-controlled SSD cache. Admission and eviction consider request rate, byte size, recency, and fetch cost so one scan of large, cold objects cannot evict the useful working set. The origin stays authoritative; losing an edge cache is a performance event, not data loss.

On a miss, a singleflight table coalesces callers for the exact key. One caller asks a regional shield; followers wait for a bounded time or use an allowed stale entry. The shield repeats lookup and coalescing across many PoPs. Only its elected fill enters the origin scheduler. This second coalescing boundary prevents one cold object from creating an independent origin fetch per PoP.

Every fill has a deadline, maximum size, content-type validation, checksum, tenant byte budget, and retry budget. A retry uses exponential backoff and jitter but still consumes the origin budget. Hedging is limited to idempotent reads and cannot double origin work without a strict cap. Large objects stream to the client while writing a temporary cache entry; the entry becomes visible only after the expected length, validator, and checksum are complete.

Step 6: Make purge and fill races safe

Content-addressed filenames are the default for immutable assets: publishing new bytes creates a new URL, and old URLs can expire naturally. Stable URLs need an explicit purge API that supports an exact object, an approved prefix or tag, and a tenant-scoped emergency purge. Broad purges are rate-limited and require stronger authorization because they can create a global miss storm.

The purge coordinator commits {tenant, selector, generation, issued_at} to a durable ordered log before acknowledging. PoPs apply the event idempotently, advance the selector's minimum generation, delete matching bytes, and retain a tombstone long enough to cover old fills and delayed events. They report the applied generation. Hierarchical fan-out, retries, and regional relays keep one slow PoP from blocking the common path.

Before publishing a filled object, the cache compares the generation captured at fetch start with the current minimum generation. If a purge advanced during the fetch, those bytes are discarded or refetched under the new generation. This comparison prevents an old response arriving after a delete from resurrecting stale content. The same rule applies at edge and shield layers.

The API reports accepted, 99%-propagated, and complete-or-expired states separately. Synthetic probes request the purged key from regions and validate version headers or content hashes. Security revocation still belongs in an authoritative online check or a separately bounded token lifetime; a 60-second cache-purge SLO is not instant revocation.

Step 7: Handle overloads and failures explicitly

• PoP failure: withdraw or stop advertising the route, drain to healthy PoPs, and reserve capacity for redistributed traffic.
• SSD loss: rebuild gradually through admission control; do not warm every object or bypass the shield.
• Shield failure: choose a secondary shield; if direct-origin fallback is allowed, keep the same origin budgets.
• Origin timeout or 5xx: serve bounded stale content only where policy permits; otherwise return an explicit error and avoid retry amplification.
• Control-plane outage: keep serving from last-known-good signed configuration; queue safe changes and reject security-sensitive mutations that cannot be verified.
• Purge-stream delay: retry idempotently, expose lagging PoPs, and bypass or revalidate the affected key when a critical generation is known but bytes are not trustworthy.
• Hot-object surge: coalesce fills, replicate the object across cache processes, protect one process from NIC or lock saturation, and rate-limit abusive tenants.

Step 8: Secure and verify the complete system

Terminate TLS with tenant-scoped certificates and protect origin identity. Enforce request-size, header-count, range-count, and response-size limits. Normalize ambiguous paths and HTTP fields once to prevent request smuggling and cache-key disagreement. Partition quotas, keys, logs, purge rights, and cache namespaces by tenant. Signed URLs or cookies are verified before lookup, and their policy must not accidentally turn a private response into a public object.

Observe request and byte hit rates separately, hit TTFB, miss latency, shield hit rate, origin QPS and bandwidth, coalesced followers, cache-key cardinality, eviction bytes, stale age, purge lag by PoP, configuration version, error rate, and failover load. Logs record a privacy-safe key digest, tenant, PoP, result type, age, generation, upstream tier, and trace ID.

Validation includes a hot-object cold start from all 50 PoPs, a purge racing an intentionally slow fill, duplicate and out-of-order purge events, a poisoned Vary, authorization and cookie bypass, partial range fills, an SSD restart, shield loss, origin throttling, PoP withdrawal, and a control-plane outage. Acceptance checks include cached p99 TTFB below 50 milliseconds in normal conditions, request availability of 99.99%, origin fetches at or below 20,000 per second, and 99% of healthy PoPs applying a purge within 60 seconds without stale resurrection.

High-Quality Sample Answer

“I would start with the capacity boundary. Two million requests per second at 256 KiB is about 4.19 Tb/s of peak response traffic. The origins can accept only 1% of edge request volume, so origin protection is a hard invariant, not an optional optimization.

I separate a durable control plane from the serving data plane. Tenant configuration, certificates, origin identities, cache rules, and purges are versioned and audited. PoPs serve with signed last-known-good configuration instead of querying that database on every request. DNS and anycast route users to a healthy PoP; SNI and Host identify the tenant before a namespaced cache lookup.

The key contains tenant, canonical URL, only the query and header dimensions that change the representation, and a generation. Private, authorized, no-store, and unsafe responses bypass the shared cache. A hit comes from RAM or SSD. A miss is coalesced at the edge, sent to a regional shield, coalesced again, and admitted through per-origin and global budgets. Fresh, revalidated, and bounded-stale paths are distinct.

Immutable assets use versioned URLs. A mutable-URL purge first commits an increasing generation to a durable log. Every tier keeps the minimum allowed generation and a tombstone. A fill checks that generation before publication, so bytes fetched before a purge cannot arrive afterward and restore stale content. I expose 99%-propagated and complete states instead of hiding laggards.

I would prove the design with request and byte hit rates, origin QPS, p99 hit TTFB, stale age, purge lag, and configuration versions. Then I would inject a hot cold miss, purge/fill race, PoP and shield failures, origin throttling, out-of-order purge events, cache-key poisoning, and a control-plane outage, keeping the four stated SLOs as acceptance criteria.”

Common Mistakes

• Calling the geographically closest PoP guaranteed → BGP selects network paths, not straight-line distance → measure latency and health, and design route withdrawal and failover.
• Including every request header in the key → cardinality explodes and most traffic misses → allowlist only representation-changing dimensions and reject unsupported Vary.
• Omitting tenant identity from the namespace → identical URLs can cross tenant boundaries → derive the tenant before lookup and make it an implicit key prefix.
• Deleting bytes on purge without a generation → an older in-flight fill can republish them → advance a tombstone and compare generations before cache insertion.
• Adding only an edge lock → 50 PoPs can still issue 50 origin fills → coalesce again at a shield and retain origin-wide budgets.
• Retrying every failed origin request → retries amplify the outage → use deadlines, bounded budgets, jitter, and route-specific stale or error behavior.
• Using purge for instant security revocation → propagation has a measurable tail → use an authoritative check or bounded credential lifetime for security decisions.
• Reporting only request hit rate → many tiny hits can hide expensive large misses → track byte hit rate, origin bandwidth, size distributions, and fetch cost too.

Follow-up Deep Dive

Follow-up 1: How would you support large video objects and range requests?

Prefer immutable segment URLs and cache complete segments where practical. Validate Content-Range, object length, validator, and generation before combining ranges. Limit range count and amplification, and never let two representations share a partial-object key. For very large objects, align chunks to a controlled grid so overlapping requests can reuse bytes without creating arbitrary fragments.

Follow-up 2: How would you prevent a cache-key poisoning attack?

Canonicalize the URL and HTTP fields once, reject ambiguous encodings, and include every approved input that changes origin bytes. Do not forward an unkeyed header that the origin uses for representation selection. Test conflicting headers, duplicated fields, path encodings, query order, and host normalization at edge and origin so both sides interpret the request identically.

Follow-up 3: Should personalized HTML ever be cached at the edge?

Only with an explicit product and security contract. Safer options cache a public shell and fetch private data separately. If full HTML must be cached, partition by a bounded, verified identity or cohort, prevent shared reuse, define logout behavior, and test cross-user isolation. An arbitrary session cookie in a public cache key is both risky and destructive to hit rate.

Follow-up 4: How would you choose between one global shield and regional shields?

A single shield maximizes miss consolidation but can add distance and concentrate failure. Regional shields reduce latency and blast radius but may fetch the same object from origin several times. Choose from origin location, cacheability, regional demand, acceptable latency, and origin budget; then test shield failover and document where the selected topology stops fitting.

Follow-up 5: How would you roll out a new cache-key policy?

Compile it as a new configuration version, shadow-compute old and new keys, and compare cardinality, hit rate, privacy classification, and origin load without serving from the new key. Roll out by tenant and PoP, keep a rollback version, and warm only proven hot objects. A key change creates a cold-cache event, so it must enter the same origin budgets as ordinary fills.

Follow-up 6: Is multi-CDN required for 99.99% availability?

Not automatically. One provider can meet the target if the measured failure model, PoP redundancy, route withdrawal, origin design, and operations support it. Multi-CDN reduces some provider risks but introduces DNS or steering consistency, duplicated configuration, purge coordination, log normalization, certificate handling, and a common origin-shield problem. Adopt it only after testing that the added control plane improves the stated availability target.