Design a Multi-Channel Notification System

Problem and applicable scenarios

Design a shared notification platform used by authentication, ordering, social, and marketing services. It supports mobile push, SMS, email, and in-app messages. Transactional messages include OTPs and payment results; bulk messages include event reminders and marketing outreach. A caller may send immediately or schedule a notification. Users may opt out by channel and notification type, and marketing messages must respect quiet hours in each user's time zone.

Use these interview assumptions:

• The system creates one billion channel-delivery tasks per day. One business notification sent by both SMS and email counts as two tasks.
• Average throughput is 1,000,000,000 / 86,400 ≈ 11,600 tasks per second. Estimate a campaign peak at 20 times the average, or about 232,000 tasks per second.
• The availability target for durable API acceptance is 99.99%, with p99 acceptance latency below 200 milliseconds.
• For OTPs, p99 time from acceptance to submission to a channel provider is below two seconds. Marketing work may be smoothed across 15 minutes. Neither target promises when a device actually displays the message.
• If the durable envelope for one delivery task averages 1 KB, logical writes are about 1 TB per day and 30 TB over 30 days, before indexes, replicas, receipts, and compression.

These values drive partitioning, backlog, and isolation decisions; they are not provider performance claims. Template authoring, audience-selection algorithms, billing, and A/B testing are out of scope. This question targets senior backend, platform, and system design roles. Its core challenge is preserving priority, recoverability, and truthful observability across external channels with different semantics.

What the interviewer is evaluating

The first signal is whether the candidate defines “sent successfully.” An API 202 means the platform durably accepted work. A successful provider response commonly means that the provider accepted a request. Device receipt, operating-system display, and user open are later states. Firebase's Sends metric can mean a message was queued or passed to APNs, while a successful APNs response describes the request. Collapsing all of these into delivered corrupts both operational metrics and incident response.

The second signal is whether priority is backed by resource isolation. A shared queue with a priority field can still have its disk, consumer connections, and provider quota occupied by a 50-million-user campaign. A strong design separates transactional and bulk queues, consumers, and protected channel budgets while allowing bulk traffic to borrow idle capacity. OTP traffic also needs its own ceiling; a “critical” label must not bypass every protection.

The third signal is precise delivery-guarantee language. A standard queue can deliver a task more than once, so workers should process a stable delivery_id idempotently. If a network timeout occurs after a provider accepted the request, internal deduplication cannot undo that external side effect. Without a provider-side idempotent send API, the platform can reduce duplicate probability, record an uncertain outcome, and choose retry or failover based on message risk. It cannot promise end-to-end exactly-once delivery.

The fourth signal is a closed failure loop. Permanent failures stop retrying and deactivate invalid endpoints when justified. HTTP 429, 5xx responses, and network failures use jittered backoff. Expired tasks terminate. A dead-letter replay keeps the original delivery_id and rechecks expiry and opt-out state. Receipts can be duplicated and arrive out of order, so the system stores raw events before deriving current state through channel-specific transitions.

Questions to clarify before answering

• What does “delivered” mean to the business? The controllable OTP target should end at provider acceptance because offline devices, carriers, and user permissions are outside the platform. If the business requires “read,” it needs a supported channel receipt or client event, with coverage disclosed.
• Who assigns notification type and priority? The server owns a controlled type catalog that maps type to priority, allowed channels, template, and opt-out rules. Allowing callers to send critical=true lets every team compete for the emergency lane.
• Which messages may bypass quiet hours or opt-outs? Only transaction types approved for that exception may do so. Marketing preferences are checked immediately before channel queueing, so a user who opts out after scheduling is excluded.
• Is cross-channel fallback allowed? Replacing failed SMS with push changes cost, reachability, and user expectations. Configure a fallback graph by notification type and distinguish explicit rejection from an unknown result; failover after an unknown result can contact the user twice.
• What ordering is required? Password-reset messages for one user may need ordering, while ordinary social alerts usually do not need global ordering. Preserve order only on keys such as (userid, notificationtype) to avoid serializing the system.
• What are the retention and privacy boundaries? Message bodies may be sensitive. Keep template versions and minimal parameters in queues, encrypt selected fields, apply retention limits, and never log OTPs, full phone numbers, or email bodies.

30-second answer framework

“I would separate durable acceptance, provider acceptance, device delivery, and user read into different states. The ingress atomically writes the notification and an outbox, then returns 202. Recipient fan-out is asynchronous, and policy is checked near dispatch for preferences, quiet hours, expiry, and deduplication. Each channel has separate transactional, normal, and bulk queues and consumers, with provider capacity protected for transactional traffic and idle capacity loaned to bulk work. Workers process at-least-once tasks under a stable delivery_id, classify permanent, rate-limit, and transient failures, and use jittered exponential backoff for transient cases. Provider callbacks are appended as events and folded through a channel state machine, so an out-of-order sent event cannot regress delivered. I would test a marketing surge alongside OTPs, duplicate tasks, and reordered callbacks to prove the two-second SLO and backlog recovery.”

Step-by-step deep dive

The ingress API does not synchronously call an SMS or push provider. It authenticates the caller, validates a controlled notification type, fixes the template version, validates minimal parameters, and writes a notification plus an outbox record in one database transaction:

~~~text POST /v1/notifications { request_id, notification_type, recipients | audience_id, template_version, template_params, schedule_at, expire_at }

202 Accepted { notification_id, accepted_at } ~~~

requestid deduplicates caller retries, notificationid identifies one business notification, and delivery_id identifies one user-channel delivery. They cannot be one identifier because a notification may fan out to many users and channels. An outbox relay publishes only after the ingress transaction commits, closing the database-commit/message-publish gap. For a 50-million-recipient campaign, ingress stores an audience-snapshot reference and cursor; it does not create 50 million rows inside one request or transaction.

A fan-out service reads the snapshot in partitions and creates small planning batches. Near send time, a policy service loads the notification-type rules and current user preferences, then evaluates opt-out, quiet hours, channel availability, frequency policy, expiry, and fallback order. Quiet hours use the user's IANA time zone and handle daylight-saving transitions. If no zone is known, the product uses an explicit default instead of silently using server time. A filtered task receives SUPPRESSED plus a machine-readable reason so support can explain why it was not sent.

The core data can be split into three records:

~~~text Notification( notificationid, tenantid, notification_type, templateversion, scheduleat, expire_at )

Delivery( deliveryid, notificationid, user_id, channel, trafficclass, state, provider, providermessage_id, attemptcount, nextattemptat, stateversion )

DeliveryEvent( deliveryid, providereventid, eventtype, providertime, receivedat, rawpayloadref ) ~~~

Delivery is the materialized current view; DeliveryEvent preserves provider receipt facts. Raw content and personal data live in controlled storage, with only a reference in the event table. When a provider supplies providereventid, enforce uniqueness. Otherwise, hash normalized receipt fields for deduplication. Verify webhook signatures before writing events and preserve unknown fields compatibly; a new provider field must not break valid callbacks.

At minimum, distinguish these states:

~~~text ACCEPTED -> PLANNED -> QUEUED -> SENDING -> PROVIDER_ACCEPTED | v DELIVERED -> READ

Terminal side states: SUPPRESSED, EXPIRED, FAILED_PERMANENT ~~~

The diagram expresses business stages, not guaranteed callback arrival order. A provider can deliver delivered before sent, and duplicate webhooks are common. The handler appends an idempotent event, then updates the projection with a transition table keyed by channel, current state, and new event. A late sent cannot regress an SMS already at DELIVERED; a channel with read receipts may advance from DELIVERED to READ. An unmapped event remains in the raw table and raises an alert rather than forcing a guessed state.

The send data plane separates queues by channel and traffic class, such as sms.transactional, sms.bulk, and push.transactional. Each group has independent backlog-age metrics, consumer concurrency, and retry queues. Suppose an SMS provider contract allows 10,000 requests per second. The scheduler could protect 3,000 per second for transactional traffic, let bulk traffic borrow unused capacity, and reclaim it when transaction traffic rises. These are configuration examples; real values come from provider contracts and load tests. Per-tenant fair scheduling prevents one campaign from consuming the bulk budget, while aging prevents normal work from starving forever.

Separate queues require more topics, connections, and operational configuration than a single queue with priority fields, but they isolate disk backlog and consumer resources. At smaller scale with one channel, a shared queue plus weighted fair scheduling can be simpler. When transaction SLOs and bulk deadlines differ by orders of magnitude, physical isolation is easier to prove. In either design, one channel scheduler enforces the provider's actual quota. Individual consumers must not each assume that they own the full quota.

A channel worker receives an at-least-once task and atomically claims an attempt under deliveryid. If the delivery is already terminal, it acknowledges the queue task. Otherwise it checks expireat, renders the template, calls the provider, and stores the response. A duplicate queue task cannot create a second delivery record. An external timeout still has three possible outcomes: the provider did not receive the request, received it but its response was lost, or has an unknown processing result. Reuse a provider idempotency key when one exists. Without that feature, mark the attempt UNKNOWN and reconcile through a provider query or receipt. A critical notification may retry after the business explicitly accepts duplicate risk; marketing can usually wait for reconciliation or expiry.

Error classification determines the next action:

• Invalid parameters, invalid templates, and confirmed invalid device tokens are permanent. Move to FAILED_PERMANENT and deactivate the endpoint only when the evidence supports it. APNs 410 means the device token is no longer active for the topic.
• HTTP 429, provider 5xx responses, and connection failures are generally retryable. Use exponential backoff, full jitter, a maximum attempt count, and an overall deadline. Retries still pass through channel capacity control so recovery does not create a second spike.
• Authentication or account-configuration failures are platform incidents. Pause the affected provider channel and alert; retrying every message amplifies the failure.
• An OTP or stale reminder that reaches expireat becomes EXPIRED. A dead-letter replay cannot extend its original expiry or assign a new deliveryid to bypass deduplication.

Provider acceptance does not prove device delivery. APNs returns a status and apns-id for each POST; success proves request-level success. Firebase also counts queueing or handoff to APNs as Sends. An SMS provider may offer later queued, sent, delivered, undelivered, and read states, and its webhooks can be out of order. The public API therefore returns a layered state plus status_reason. Reports separately calculate acceptance, provider-acceptance, observable-delivery, and observable-open rates. If a channel does not expose a state, report unknown rather than counting it as success or failure.

For scale, partition notification and delivery tables by notification_id or time, and scale work queues by channel, class, and partition key. Work that needs per-user ordering uses a stable user partition key; unordered bulk tasks can hash more evenly. Keeping the full 30 TB, 30-day logical envelope in expensive online indexes is unnecessary. Keep recent delivery state online, compress and archive older events, and retain only indexes required for compliance and support. Capacity planning separately adds replicas, indexes, receipt amplification, and retry writes.

The decisive acceptance test is “a bulk surge cannot delay OTPs.” Sustain about 232,000 campaign tasks per second, then add transaction traffic. Verify that OTP p99 from acceptance to provider submission remains below two seconds and that the bulk backlog drains within its 15-minute window. Inject 429 and 5xx responses and confirm backoff does not synchronize into retry waves. Deliver the same queue task more than once and verify that it reuses the original delivery_id. Send callbacks in delivered -> sent -> delivered order and verify that state never regresses. Also cover opt-out immediately before send, daylight-saving transitions, expired dead letters, and unknown provider outcomes. Those failure-path results make “reliable” testable.

High-quality sample answer

“I would define four separate outcomes: durable platform acceptance, provider acceptance, observable device delivery, and user read. The API promises only the first. It writes the notification and outbox, then returns 202. Fan-out is asynchronous, and the latest opt-out, quiet-hour, channel-availability, and expiry rules are evaluated near send time before assigning a stable delivery_id.

One billion channel tasks per day averages about 11,600 per second, with a 20-times campaign peak of about 232,000 per second. OTPs must reach the provider within two seconds, while marketing can be smoothed over 15 minutes. I would therefore separate queues and consumers by channel and transactional, normal, and bulk class, protect provider capacity for transactions, and let bulk borrow only idle capacity. Each tenant also gets a fair share.

Workers assume at-least-once consumption. Duplicate queue work claims the same delivery_id. Permanent errors stop, while 429, 5xx, and network failures use jittered backoff bounded by an overall deadline. A provider timeout may have produced an external side effect. I reuse a provider idempotency key if available; otherwise I mark UNKNOWN and reconcile instead of claiming end-to-end exactly-once.

Receipts are immutable events folded through a channel transition table, so a late sent event cannot overwrite delivered. My rollout test combines a marketing peak with OTP traffic, then injects duplicate work, 429 responses, provider timeouts, out-of-order callbacks, and a last-minute opt-out. The key metrics are queue age per class, provider-submission latency, failure class, UNKNOWN count, and attempted state regressions.”

Common mistakes

• Recording delivered when the API returns 202 → only durable acceptance is complete → separate ACCEPTED, PROVIDER_ACCEPTED, DELIVERED, and READ.
• Putting all work in one priority queue → bulk backlog still shares disk, consumers, and provider quota → isolate resources by channel and traffic class, with controlled borrowing.
• Letting critical traffic bypass all limits → abnormal OTP traffic can overload the platform and provider → give transaction traffic its own cap, alerts, and tenant fairness.
• Claiming the user receives exactly once because the queue is at least once → a provider call may succeed while its response is lost → make internal work idempotent by delivery_id, reconcile unknown external results, and disclose duplicate risk.
• Retrying every error immediately → permanent errors waste capacity and 429/5xx failures create a retry storm → classify permanent, transient, throttling, and platform configuration failures; back off transient cases with jitter.
• Overwriting the current state directly → an out-of-order callback can change delivered back to sent → persist idempotent events first, then update the projection with a channel transition table.
• Freezing user preferences at scheduling time → a later opt-out still receives marketing → recheck preferences and compliance rules near channel queueing.
• Assigning a new ID during dead-letter replay → deduplication is bypassed and stale notifications can be sent → reuse the original delivery_id and recheck opt-out and expiry.
• Using global ordering for simplicity → unrelated users block one another and one partition caps throughput → preserve only the local order that a user-notification type actually requires.

Follow-up questions and responses

Follow-up 1: Fifty million users want the campaign at 9:00 a.m. local time. What changes?

During fan-out, build time buckets by IANA time zone and local date, then produce rate-controlled batches ahead of their windows. Add jitter within each target window rather than releasing every task at the hour. Define product behavior for nonexistent or repeated local times during daylight-saving changes, such as moving to the next valid instant and sending no more than once per day. “9:00 a.m.” should mean a provider-submission window because device display remains outside platform control.

Follow-up 2: The provider returned success but never sent a delivery receipt. What is the state?

Keep PROVIDERACCEPTED; do not promote it to DELIVERED. After the channel's observation window, the system may expose DELIVERYUNKNOWN for operations, but unknown must not be counted as failure. If the provider offers a query API or aggregate report, reconcile asynchronously and disclose that data's delay and coverage.

Follow-up 3: What if the delivered callback arrives before sent?

Append both events idempotently. The projection may advance from PROVIDER_ACCEPTED or SENDING to DELIVERED; the later sent enriches audit history without lowering state. If the same provider later emits a documented revocation or error, model that provider-specific transition explicitly instead of guessing order from a global integer.

Follow-up 4: Can two SMS providers fail over automatically?

Failover is reasonable after explicit rejection, a failure before connection, or a provider-wide circuit breaker. A timeout after send may mean the primary already delivered, so immediate failover raises duplicate-SMS risk. OTP can fail over when cost and security owners explicitly accept that risk, while marketing usually waits for a query, receipt, or expiry. Configure the decision by notification type rather than globally.

Follow-up 5: How do you prove that priority isolation works?

Use a closed load test that combines three pressures: a sustained bulk backlog, a hot tenant, and repeated provider 429 responses. Acceptance requires OTP provider-submission p99 below two seconds, no bulk work on transaction consumers, aggregate provider rate within configuration, and bulk recovery within the 15-minute deadline. Then fail one transaction-consumer partition and confirm the remaining instances take over protected capacity. Average throughput alone cannot prove isolation.