System Design Interview: Design a Reliable Webhook Delivery Platform

Problem and Scope

A SaaS platform emits events such as invoice.paid, order.shipped, and user.disabled. Customers register HTTPS endpoints and subscribe each endpoint to selected event types. Design the outbound platform from the committed business event to the customer's HTTP response. Endpoint administration, delivery history, secret rotation, and manual replay are in scope. The customer's internal processing after it acknowledges the request is outside the platform's control.

Use these interview assumptions:

• The product creates 50 million business events per day. Each event matches four endpoints on average, producing 200 million logical deliveries per day.
• Average load is about 579 events and 2,315 new deliveries per second. A ten-times peak is about 5,787 events and 23,148 new deliveries per second.
• If retries add 10% more attempts, the peak dispatch path should sustain about 25,463 attempts per second.
• Under a normal peak, 99% of eligible new deliveries should start their first attempt within 10 seconds. Retries have a 24-hour deadline, and delivery history remains queryable for 30 days.
• With a 1.5 KB immutable event payload, 300 bytes of delivery metadata, 250 bytes per attempt, and 1.1 attempts per delivery, logical storage is about 190 GB per day or 5.7 TB for 30 days. This excludes indexes, replicas, compression, and object-store overhead.

These are sizing inputs, not industry benchmarks. Billing, arbitrary payload transformation, inbound webhooks, and the customer's application design are out of scope. The key contract is at-least-once delivery: duplicates and out-of-order arrival are possible, while silent loss inside the promised retention and retry boundary must be detectable and recoverable.

What Interviewers Evaluate

The first signal is whether the candidate defines the identifiers and guarantee precisely. One eventid represents a committed business fact. One deliveryid represents that event going to one endpoint and stays stable across automatic retries and manual redelivery. A unique constraint on (eventid, endpointid) prevents fan-out replay from creating a second logical delivery. It does not prevent the same HTTP request from reaching the customer twice when a worker loses the response.

The second signal is the transaction boundary. Publishing directly after a business database commit creates a dual-write gap: the transaction may commit and the process may crash before publishing. A transactional outbox, change-data-capture stream, or an equivalent durable event source closes that gap. Fan-out then materializes delivery state before dispatch, so the system can answer which endpoints were selected, which attempt ran, and what remains due.

The third signal is failure isolation. A slow endpoint must not occupy every connection. A tenant with thousands of failing destinations must not consume the retry budget of healthy tenants. Queue partitions alone do not provide fairness; the design needs bounded per-endpoint concurrency, tenant quotas, retry scheduling, and a circuit-breaker or pause state.

The fourth signal is security across two directions. Recipients need an HMAC signature over the exact bytes plus authenticated delivery metadata. The sender must also treat a customer-controlled URL as an SSRF surface, restrict schemes and destinations, revalidate DNS results, constrain redirects, and isolate egress. A strong answer connects these controls to secret rotation, replay protection, auditability, and incident response.

Questions to Clarify Before Answering

• What is success? This design treats any 2xx response as endpoint acknowledgment. A 3xx, timeout, connection error, 408, 429, or 5xx is not success. Acknowledgment does not prove that the customer's later business processing succeeded.
• Which events and payload version are promised? Each event type needs a documented schema and versioning policy. Retries send the same immutable payload bytes; they must not rebuild old events from current database state.
• What ordering is required? Default delivery is unordered. If a customer needs order for one aggregate, include objectid and monotonically increasing objectversion, or offer an opt-in ordering key that accepts head-of-line blocking. Global order is neither required nor affordable.
• How long may the platform retry and replay? Here, automatic retries stop after 24 hours and logs remain for 30 days. A replay after the automatic window uses the original event and delivery identity but creates a new attempt record.
• Can endpoints point anywhere? The product accepts public HTTPS endpoints only. Private, loopback, link-local, multicast, reserved, and cloud-metadata destinations are rejected for IPv4 and IPv6.
• What data may leave the platform? Subscription authorization and payload minimization are part of fan-out. Sensitive fields are omitted or represented by a resource reference when the integration can fetch them through an authenticated API.

30-Second Answer

“I would commit each business event through an outbox, publish it to a durable event log, and have a fan-out service resolve active subscriptions. It creates one delivery row per (eventid, endpointid) before placing work on a queue. Workers claim attempts with leases, apply per-endpoint and per-tenant limits, sign the immutable bytes together with the stable delivery ID and current attempt timestamp, and send over HTTPS. A 2xx completes the delivery; retryable failures use exponential backoff with full jitter until the 24-hour deadline, while permanent failures stop. Because a timeout after send is ambiguous, the contract is at least once and customers deduplicate by delivery ID. I would add signed secret rotation, SSRF-safe URL validation, delivery logs and replay, endpoint circuit breakers, and reconciliation that detects outbox, fan-out, or queue gaps.”

Step-by-Step Deep Dive

Start at event creation. In the same local transaction that changes business state, write an outbox row containing eventid, tenant, type, schema version, object identity and version, occurrence time, and a reference to canonical serialized payload bytes. An outbox relay publishes to a partitioned durable log. The relay can publish twice, so downstream consumers deduplicate by eventid. If the business data spans systems, the authoritative producer owns the event; the webhook service should not reconstruct facts by polling mutable tables.

The control-plane APIs can be small and explicit:

POST /v1/webhook-endpoints
PATCH /v1/webhook-endpoints/{endpoint_id}
POST /v1/webhook-endpoints/{endpoint_id}/rotate-secret
GET  /v1/webhook-deliveries?endpoint_id=&status=&cursor=
POST /v1/webhook-deliveries/{delivery_id}/replay

Endpoint creation authenticates the tenant, accepts an HTTPS URL and event-type filter, validates the destination, and returns a signing secret once. A challenge delivery can prove ownership, but a successful challenge does not make future DNS answers safe. Secret rotation keeps current and previous versions active during a bounded overlap and exposes which signature version was used. Updating a URL creates an auditable configuration version; it does not silently rewrite historical attempts.

Use four durable records:

Endpoint(endpoint_id, tenant_id, url, status, event_types,
         current_secret_version, previous_secret_version, config_version)

Event(event_id, tenant_id, type, schema_version, object_id,
      object_version, occurred_at, payload_ref, payload_hash)

Delivery(delivery_id, event_id, endpoint_id, endpoint_config_version,
         status, attempt_count, next_attempt_at, expires_at, lease_version)

Attempt(attempt_id, delivery_id, attempt_number, started_at, finished_at,
        http_status, latency_ms, error_class, response_digest)

The fan-out service reads an event, loads authorized active subscriptions for that tenant and event type, then inserts delivery rows in bounded batches. The database enforces uniqueness on (eventid, endpointid). Only after rows exist does it enqueue delivery_id values. If enqueueing fails, a sweeper finds due PENDING rows without active queue work. If the fan-out process crashes midway, replay repeats the lookup and uniqueness constraint fills only the missing rows. A stored subscription snapshot or endpoint configuration version makes later audits explainable.

New work and retries should not share one unbounded FIFO. A scheduler selects due rows into time buckets, then applies weighted tenant fairness, endpoint token buckets, and bounded endpoint concurrency. Healthy endpoints continue while a slow endpoint reaches its own in-flight cap. Repeated failures open a circuit and move later attempts outward, but the delivery remains visible and eligible for probe attempts. A tenant-wide quota prevents millions of bad endpoints from consuming the fleet. Retry work may borrow idle capacity, but cannot make the oldest new-delivery age miss its SLO.

A worker atomically claims a delivery with a lease and increasing leaseversion. It reads the stored payload bytes, creates the attempt timestamp, and signs a canonical string such as deliveryid.timestamp.payload_bytes with HMAC-SHA256. It sends the exact signed bytes and headers containing event type, delivery ID, event occurrence time, attempt time, schema version, and signature version. The consumer verifies the raw bytes using constant-time comparison, checks timestamp tolerance, and deduplicates the stable delivery ID. Each retry gets a new attempt timestamp and signature while keeping the delivery ID and event bytes.

The response policy must be deterministic. Any 2xx marks the delivery SUCCEEDED. Do not follow 3xx automatically. Treat 408, 429, 5xx, connection resets, and timeouts as retryable, and honor a bounded Retry-After on 429 or 503; most other 4xx responses are permanent for that configuration. TLS and DNS failures may begin as retryable but open the endpoint circuit quickly. Use exponential backoff with full jitter, a maximum delay, a maximum attempt count, and the 24-hour absolute deadline. A timeout after the request left the worker is an unknown outcome: retrying may duplicate customer processing, so the platform must never advertise exactly once.

Manual replay creates another Attempt for the same logical Delivery; it does not mint a new business event. The operator sees the original immutable payload, endpoint configuration used, all response classes, and whether automatic retries are still active. Authorization is rechecked before replay, and the endpoint's current active secret may sign the new attempt. If the product instead promises byte-for-byte historical authentication, it must retain the required historical key under a defined key-retention policy.

Customer-supplied URLs need a dedicated egress boundary. Parse with one well-tested URL implementation, allow HTTPS and approved ports, reject credentials in URLs, resolve all A and AAAA answers, and block private, loopback, link-local, reserved, multicast, and metadata ranges. Validate again at connection time or pin the validated address to reduce DNS-rebinding and time-of-check/time-of-use gaps. Disable redirects, or re-run the complete policy for every hop without forwarding secrets. Put workers in an egress network that cannot reach control planes or internal services, and cap connection time, total request time, response bytes, and decompression.

Capacity follows the fan-out, not just event count. Fifty million events times four subscriptions produce 200 million deliveries per day. At 1.1 attempts each, attempt history is 220 million rows per day. The assumed raw sizes give 50M × 1.5 KB = 75 GB, 200M × 300 B = 60 GB, and 220M × 250 B = 55 GB, totaling about 190 GB per day. Keep current due-state indexes small, partition history by time and tenant hash, and archive immutable payloads and old attempts to cheaper storage. Measure event-to-fan-out lag, oldest new and retry ages, first-attempt latency, success by endpoint and status class, retry amplification, circuit state, tenant throttling, and replay outcomes.

Finally, reconcile every durable boundary. Compare committed outbox rows with published event IDs, events with the expected subscription snapshot and delivery count, due delivery rows with scheduler leases, and terminal counts with attempt history. Fault-injection should crash the relay after publish, crash fan-out halfway, duplicate queue messages, kill a worker after the remote endpoint processes the POST but before the local success write, return synchronized 429 responses, and make one tenant's endpoints hang. Acceptance is expressed as durable counts, bounded queue age, fair recovery, and explainable duplicates—not merely a successful demo request.

Strong Sample Answer

“I would give each committed business fact an immutable eventid, and each event-endpoint pair a stable deliveryid. The producer writes the event to an outbox in its business transaction. A relay publishes to a durable log, and fan-out materializes deliveries under a unique (eventid, endpointid) constraint before queueing them. That lets a replay repair missing work without creating a second logical delivery.

At the stated peak, fan-out creates about 23,148 new deliveries per second and the dispatch path plans for about 25,463 attempts per second including retries. A scheduler separates new and retry work, applies weighted tenant fairness, per-endpoint concurrency and token buckets, then lets workers claim attempts with versioned leases. One slow customer therefore consumes only its own allowance. Repeated failures open an endpoint circuit while probes and the 24-hour retry deadline remain visible.

Every attempt signs delivery_id, attempt timestamp, and the exact immutable payload bytes with HMAC-SHA256. The customer checks the signature with a constant-time comparison, rejects stale timestamps, and deduplicates the stable delivery ID. A 2xx succeeds. 408, 429, 5xx, network errors, and timeouts retry with exponential backoff and full jitter; most other 4xx responses stop. A worker can crash after the customer processed a request but before recording success, so I promise at least once and document that consumers must be idempotent.

The control plane provides endpoint and subscription management, bounded-overlap secret rotation, delivery logs, and replay. Endpoint URLs pass HTTPS, IP-range, DNS-rebinding, redirect, timeout, and response-size controls inside an isolated egress network. I would prove the design by reconciling every durable boundary and injecting duplicate publication, partial fan-out, lost acknowledgments, mass 429 responses, and a tenant full of hanging endpoints. The pass criteria are the first-attempt SLO, no unexplained delivery gaps, bounded retry amplification, and fair recovery for healthy tenants.”

Common Mistakes

• Publish after committing business data → a crash between the two operations loses the webhook event → write a transactional outbox or consume an equivalent durable change stream.
• Create a fresh delivery ID for every retry → the recipient cannot deduplicate one logical delivery → keep delivery_id stable and create separate attempt records.
• Promise exactly-once HTTP delivery → a lost response after remote processing leaves the sender unable to know the outcome → offer at-least-once delivery and require idempotent consumer handling.
• Rebuild payloads on retry → current database state changes the historical event and invalidates old signatures → store canonical immutable event bytes and their schema version.
• Use one FIFO for every endpoint → slow destinations occupy connections and delay healthy customers → bound per-endpoint work and schedule with tenant fairness.
• Retry every non-2xx immediately → permanent failures waste capacity and synchronized retries amplify incidents → classify errors and use exponential backoff with full jitter and a deadline.
• Follow redirects from customer URLs → a public URL can redirect workers to internal services → disable redirects or fully revalidate every hop in isolated egress.
• Sign parsed JSON → reserialization changes bytes and causes valid signatures to fail → sign and verify the exact raw body plus authenticated ID and timestamp metadata.
• Treat a dashboard replay as a new event → downstream customers may apply the business fact twice under a new identity → replay the existing delivery and record a new attempt.

Follow-up Questions and Answers

Follow-up 1: Why can the platform not guarantee exactly-once delivery?

Suppose the endpoint commits its work and returns 200, but the connection closes before the worker reads the response. Retrying may repeat the customer's side effect; not retrying may lose an event that never arrived. The sender has no atomic transaction with an arbitrary customer server. A stable delivery ID, recipient-side idempotency, and reconciliation make at-least-once delivery manageable, but they do not turn two databases and a network into one exactly-once commit.

Follow-up 2: How do you prevent one broken endpoint from delaying everyone else?

Give each endpoint a small in-flight limit and token bucket, then schedule across tenants with weighted fairness. Timeouts release leases and schedule delayed retries instead of sleeping inside workers. Consecutive failures open a circuit, so only controlled probes run while healthy endpoints use the fleet. Track both endpoint and tenant retry amplification; an attacker with many endpoints must still remain inside a tenant-wide budget.

Follow-up 3: What ordering guarantee would you offer?

The default is no delivery order. Include occurredat, objectid, and a monotonic object_version so consumers can discard stale updates or fetch current state. If a paid tier requires ordering for one object, partition that subscription by an ordering key and allow only one active sequence per key. A failed earlier item then blocks later items on that key, so the product must expose the latency and availability tradeoff rather than claiming global order.

Follow-up 4: What happens during signing-secret rotation?

Create a new version, retain the previous version for a bounded overlap, and identify the accepted versions in headers or documentation. During overlap, either include signatures from both keys or let consumers try both securely. New attempts use the active key, while immutable payload bytes and delivery IDs remain unchanged. Audit the actor, time, endpoint configuration version, and final retirement. A compromised key may require immediate retirement and customer-visible replay guidance instead of an overlap.

Follow-up 5: How do you recover after fan-out was only partly completed?

Replay the durable event through fan-out. The unique (eventid, endpointid) constraint makes completed inserts no-ops and creates only missing deliveries. Reconciliation compares the event's stored subscription snapshot or configuration version with materialized rows. If subscription semantics say “configuration at event time,” preserve that snapshot; using today's subscriptions during repair could send the event to a destination that was not entitled to it when the event occurred.

Follow-up 6: Should manual replay reset the 24-hour deadline?

Automatic retry and operator replay are separate contracts. Automatic work stops at the original deadline. A replay within the 30-day history window creates a new attempt only after authorization and endpoint-state checks, and the UI clearly marks it manual. It should not silently reactivate the automatic schedule. For security or deletion events, policy may forbid replay after a shorter business deadline even though logs remain visible.