Prompt and setting
Producers send CloudEvents with retries, versioned payloads, and uneven traffic. The gateway must validate envelopes, isolate tenants, preserve delivery evidence, and make at-least-once behavior explicit to consumers.
What the interviewer tests
- Turning a protocol envelope into clear ingestion and delivery contracts.
- Choosing idempotency scope, retry state, and durable handoff boundaries.
- Designing tenant isolation, schema evolution, and operational evidence.
Clarifying questions before answering
- What peak events per second and payload sizes must each tenant support?
- Are producers allowed to retry indefinitely, and is ordering required per source or subject?
- Which consumers need at-least-once delivery, and can they deduplicate by source plus ID?
- Are schemas centrally registered, and how long must raw events be retained for replay?
30-second answer framework
I would terminate TLS at an authenticated edge, validate the CloudEvents envelope and tenant quota, then durably append the raw event before acknowledging. A partitioned queue fans out to consumers using at-least-once delivery; (tenant, source, id) is the deduplication key when the producer guarantees its stability. Consumer acknowledgements, retry schedules, dead letters, schema versions, and per-tenant limits make loss and duplication visible rather than implicit.
Step-by-step deep dive
1. Ingest and validate
Accept structured or binary HTTP binding, enforce size and content-type limits, and validate required attributes such as specversion, type, source, and id. Authenticate the tenant and reject an invalid envelope before durable work. Preserve the exact received bytes and headers for audit and replay.
2. Choose the durable boundary
Write an immutable event record and an outbox or log offset in one durable operation before returning success. A queue-only acknowledgement risks losing events if the queue publish is not committed; a database-only design can bottleneck high-volume fan-out. State the latency and durability trade-off.
3. Deduplicate without hiding retries
Use a tenant-scoped (source, id) key with a retention period covering producer retries. Store the payload digest and schema version; the same key with different bytes is a conflict requiring quarantine. Keep delivery attempts separate from the logical event so retries remain observable.
4. Deliver and retry
Consumers pull from partitions or receive pushed deliveries with a lease. A successful acknowledgement advances the attempt; timeouts and transient failures schedule exponential backoff with jitter. Permanent failures move to a tenant-scoped dead-letter stream with replay controls and audit records.
5. Evolve and operate
Validate schema versions at ingress, route incompatible events to quarantine, and support consumer capability declarations. Measure accepted, rejected, duplicate, delayed, retried, dead-lettered, and replayed events by tenant and source. Enforce quotas, encryption, retention, and access controls without logging secrets or unrestricted payloads.
High-quality sample answer
“I would authenticate each tenant at the edge, validate the CloudEvents envelope, enforce size and quota limits, and append the exact event and headers durably before acknowledging. A partitioned log then delivers at least once. The deduplication key is tenant plus source plus ID when the producer guarantees ID stability; a changed digest for the same key is quarantined. Consumers acknowledge attempts, transient failures back off with jitter, and permanent failures go to a replayable dead-letter stream. Schema versions, per-tenant metrics, retention, and audit logs make delivery semantics explicit.”
Common mistakes
- Acknowledge before durable append → a crash loses the event → ack after the chosen durable boundary.
- Deduplicate by ID globally → tenants or sources can collide → scope the key and verify the digest.
- Promise exactly once → retries and consumer effects remain possible → state at-least-once and require idempotent consumers.
- Drop incompatible schemas → debugging and replay become impossible → quarantine with versioned evidence.
Follow-up questions and responses
Can the gateway guarantee ordering?
Only within a defined scope, such as a source and subject partition. Preserve sequence metadata, route the same key to one partition, and document that retries or parallel consumers can delay later events.
What if a producer reuses an ID for different payloads?
Compare the stored digest and reject or quarantine the conflict. Never silently overwrite the first event; alert the producer because the deduplication contract is broken.
How do you support tenant-specific replay?
Keep immutable event records with authorization-bound replay tokens, a new attempt ID, rate limits, and an audit trail. Replays must pass schema and deduplication checks and must not bypass consumer isolation.