System design interview: Design a durable workflow engine with human approval

Prompt and context

Design a workflow engine for customer-defined processes of up to 20 steps. One step requires human approval and may wait for days; activities can fail, be delivered twice, or succeed while the response is lost. The system must recover from failures and support timeouts, cancellation, audit, and version upgrades. Explain the state model, scheduling, idempotency, callback security, scale, and tradeoffs.

This system-design question fits backend, platform, and infrastructure roles. The focus is durable execution semantics, not a drawing of a simple queue. The 20-step limit and multi-day wait are interview assumptions; clarify throughput, latency, tenant isolation, data sensitivity, recovery targets, and retention obligations first.

What the interviewer is testing

The interviewer is looking for a separation between workflow state, activity execution, and external side effects; an explicit acknowledgment that activities may run at least once; and idempotency keys tied to business invariants. Human approval needs an unforgeable, expiring, single-use credential. The answer should also define event history, snapshots, timers, cancellation, versions, and operational repair.

Questions to clarify first

• What are per-tenant concurrency, workflow-start rate, and maximum wait time?
• Are activities functions, containers, or external HTTP services, and which side effects are irreversible?
• How are approvers authenticated, authorized, and replaced? Is quorum approval required?
• Can retries repeat an activity, and can the business service accept an idempotency key?
• How are definitions published, frozen, and migrated? Do running instances follow a new version?
• What audit fields must be retained, and who may read or alter them?
• Are cancellation, pause, manual retry, or step skip normal product operations?

30-second answer framework

“I would model a workflow as a durable state machine: store definition versions, runs, activity attempts, an event log, and a current snapshot separately. The scheduler delivers retryable activity tasks; workers report results with an activity ID, attempt, and idempotency key, and the state machine deduplicates them. A human approval gets a short-lived one-time token bound to the run, step, tenant, and approval version. Timers, timeouts, and cancellation are durable events. I would scale with partitioned queues and tenant quotas, then validate recovery with replay, audit, and fault injection.”

Step-by-step answer

Define the execution model first. A workflow definition is an immutable version containing step types, input mappings, timeouts, retry policies, and compensation rules. A run references one frozen version, so publishing a new definition does not silently rewrite history. Rebuild state from an append-only event log; use snapshots to accelerate reads. Event appends need a sequence or version condition so concurrent writers cannot overwrite each other.

Do not call an external service inside the database transaction. Commit an ActivityScheduled event, let a scheduler enqueue the task, have a worker acquire a lease and call the service, then commit ActivitySucceeded, ActivityFailed, or ActivityTimedOut. Accept results only for the expected run, step, and attempt. Late or duplicate results are audit evidence, not state transitions.

A minimal state model is:

Delivery is at least once; a queue acknowledgement is not business completion. Messages carry run ID, step ID, attempt, definition version, and an idempotency key. A lease prevents two consumers from working at once, while a fencing token or conditional write rejects a stale worker. Lease expiry permits redelivery, but duplicate external effects depend on the activity contract.

Handle idempotency at several layers. State-machine events use a unique (run, sequence) key. Activity results use (run, step, idempotencyKey). Payments, email, or writes to a business service must accept that same key or enforce a business uniqueness constraint. If a call succeeds and the response is lost, a retry must return an already-completed result or a safe duplicate. Do not claim end-to-end exactly once. For non-idempotent effects, use human review, compensation, or a no-retry policy.

An approval URL is not authorization. Generate high-entropy random material, store only its hash, and bind the token to tenant, run, step, action, and expiry. On callback, validate the token, signature or login identity, CSRF protection, single-use state, and current workflow step. Recheck approver authorization at submission time. Approvals and rejections become tamper-evident audit events; an expired callback is invalid.

Persist timers. Store fireAt and a generation when creating a timeout or wait. A scheduler scans a time index or bucket, claims a due timer with a conditional update, and deduplicates repeats with generation and state conditions. Restarts recover from storage. Completing approval or cancellation marks an old timer obsolete; an in-memory sleep must not hold a worker for days.

Cancellation, pause, and repair are explicit state-machine commands. Cancellation records intent and prevents work that has not started; an in-flight external effect cannot be magically undone, so wait for its result or run compensation. Admin skip, retry, and input edits require authorization, a reason, old state, new state, and an audit event. Never edit a snapshot directly, or replay will produce a different result.

Isolate versions per run. A new definition gets a new digest; existing runs keep the old version by default. A migration must define compatible state and inputs, obtain any required approval, and append a WorkflowMigrated event with both versions. Workers accept only the definition version they execute, preventing an old task from advancing a new state machine incorrectly.

Scale by partitioning on tenant or run ID. Give hot tenants concurrency quotas and isolate worker pools by activity type and priority. Append events to partitioned storage, keep snapshots and indexes in an online database, encrypt sensitive payloads, and enforce retention. Add backpressure, dead letters, lease metrics, and rate limits so one runaway workflow cannot consume global capacity.

Observability must serve runtime and business operators. Record the current step, wait reason, attempts, queue delay, timer delay, approval dwell time, retry amplification, compensation result, and version. Traces can carry run, step, and attempt identifiers, while sensitive inputs stay in controlled audit storage. The operations view should explain the next action; “waiting” is not “failed.”

Inject failures: crash after event commit, duplicate messages, lease expiry, successful activity with lost response, replayed approval, duplicate timer, database outage, queue backlog, interrupted version deployment, and exhausted tenant quota. Define observable outcomes: old state does not regress, a side effect is not repeated without a contract, a token cannot be consumed twice, and recovery eventually continues or enters an explicit human state.

High-quality sample answer

“I would build a durable state machine rather than let workers remember workflow state. Definition versions are immutable and each run pins one version; the event log is the source of facts and the snapshot accelerates reads. The state machine commits a scheduling event, the queue delivers at least once, and the worker reports with run, step, attempt, and an idempotency key. Duplicate or stale results are audit entries, not invalid transitions.

An approval creates a one-time token bound to tenant, run, step, and action; only its hash is stored and it expires. The callback checks identity, permissions, CSRF, token state, and the current step, then conditionally consumes the token and appends an audit event. Approval, rejection, timeout, and cancellation are events, not direct snapshot edits.

Timers persist fireAt and generation. A bucketed scheduler claims them with conditional writes; restarts recover them and duplicate fires are deduplicated. Leases prevent concurrent workers and fencing tokens reject stale writes. Payments or email without a proven downstream idempotency contract cannot be exactly once; use downstream keys, uniqueness, compensation, or human handling.

Runs do not silently follow new definitions; migration appends an event with both versions. Partition by tenant and run, isolate hot-tenant quotas, and use worker pools by activity type. Layer append-only events, online snapshots, encrypted audit data, and retention. Fault injection must cover crashes, redelivery, lost responses, callback replay, duplicate timers, and interrupted version deployment.”

Common mistakes

• Treat a queue acknowledgement as completion → a worker may crash after the effect and receive the message again → deduplicate with durable state and idempotent results.
• Claim exactly once → local transactions cannot guarantee distributed side effects → state at-least-once delivery and downstream idempotency, compensation, or human handling.
• Treat an approval URL as authorization → leakage or replay can grant access → bind identity, tenant, step, action, expiry, and single-use state.
• Sleep in memory for days → restart and scaling lose the wait → persist timers and wake through the scheduler.
• Edit snapshots directly → replay produces a different result → use authorized state-machine commands and audit events.
• Share one queue across tenants → a hot tenant starves others → partition, quota, prioritize, and apply backpressure.
• Let new definitions affect old runs → running workflows become unexplainable → pin versions and make migration explicit.

Follow-up questions and answers

Follow-up 1: The activity succeeded but the worker crashed before writing the result. Can a retry charge twice?

If the downstream service supports an idempotency key, retry with the same key and map the completed result. Otherwise, do not retry blindly: query downstream state, enter human handling, or compensate. The state machine can make its own events consistent, but it cannot create exactly-once payment semantics by itself.

Follow-up 2: An approver clicks approve twice. What happens?

Allow one successful token consumption with a conditional update or unique constraint from pending to approved. The second request returns already handled or expired and does not advance the workflow. Both requests and identities remain in the audit trail.

Follow-up 3: How do you support “any two approvers”?

Persist an approval inbox and policy version. Record one deduplicated decision per approver, including the authorization snapshot. Advance only when the quorum is reached; rejection, revocation, and approver replacement are explicit commands.

Follow-up 4: An operator needs to skip a step urgently. What is the safe path?

Define authorized roles, skippable steps, and safety conditions first. Issue a reasoned StepSkipped command with old state, new state, operator, and approval. If skipping violates a business invariant, reject it and offer compensation or termination instead of editing the database.

Follow-up 5: How do you stop a tenant from creating an unbounded workflow?

Limit steps, branch depth, activity concurrency, history size, timer count, and total runtime per tenant. Validate definitions statically and meter execution. Exceeding a quota pauses or rejects work with an operator-visible reason instead of allowing unbounded growth.