1. Question and Context
A platform team needs to change firewall rules, payment limits, or service routing. The requester cannot approve their own change, approval must target an exact version, and a failed execution must not leave a partial update. The goal is a verifiable, auditable, recoverable workflow.
2. What the Interviewer Is Evaluating
- Whether you separate proposal, approval, execution, and rollback states.
- Whether approvers are independent and the approved content cannot be silently replaced.
- Whether you handle concurrent approvals, duplicate requests, timeouts, revocation, and permission changes.
- Whether you provide small-scope rollout and stable recovery while controlling risk.
NIST SP 800-128 requires configuration changes to be reviewed by an authorized person independent of the requester. Google SRE emphasizes code review for configuration versions and continuing to serve the previous configuration when a new one fails checks. Put those principles into data and state transitions.
3. Clarifying Questions Before You Answer
- Which resources and fields are high risk, and do environments or tenants differ?
- Is approval one-person, any-of-many, or a threshold across roles and people?
- Is execution a full switch, a batch rollout, or target-instance confirmation?
- Is rollback the last known-good version or a version chosen by the requester?
4. A 30-Second Answer Framework
Use immutable proposal, policy match, approval isolation, executor, rollback, and audit.
I freeze every change as an immutable version, and a policy selects independent approvers from the resource, risk, and environment. The approval records the version digest and policy version, and the requester cannot satisfy the approval threshold. An idempotent executor applies the change in batches with health checks. Any failure stops further rollout and restores the last verified version, while every state and decision goes to append-only audit storage.
5. Step-by-Step Deep Dive
Step 1: Define Entities and the State Machine
A proposal contains resources, diff, author, version digest, target environment, and expiry. An approval contains approver, role, decision, time, policy version, and proposal digest. An execution record contains batch, target, result, and rollback version. States can be DRAFT, PENDING_APPROVAL, APPROVED, EXECUTING, SUCCEEDED, FAILED, REVOKED, or EXPIRED; only server-side rules may transition them.
Step 2: Make Approval Independent and Version-Bound
The policy service computes required roles, people, self-approval prohibition, and approval lifetime. The diff shown to an approver must hash to the execution version; any proposal edit invalidates approval and starts a new review. Re-check permissions at approval and execution so a later role revocation cannot be bypassed.
Step 3: Control Release with an Idempotent Executor
Generate an idempotency key for each proposal and target, and record external acknowledgements. Apply small batches after syntax, dependency, and safety checks, then observe health signals. Retry only unknown outcomes; never blindly repeat a non-idempotent operation. A queue or workflow engine handles timeouts, retries, and concurrency limits.
Step 4: Roll Back Safely and Audit
Save the last verified version before release. On failure, stop later batches and restore that version. Record the operator and reason for rollback; high-risk emergency rollback may require its own approval. Audit at least the proposal digest, approval chain, execution batches, configuration versions, failure reason, and rollback result, with deletion and mutation restricted.
6. High-Quality Sample Answer
I would split the system into a proposal API, policy service, approval service, execution queue, configuration adapters, and audit storage. The proposal is immutable and contains the resource diff, environment, author, and expiry. The policy service calculates required roles and approver count by risk, excluding the author.
>
Approvers see the version digest and diff, and the approval binds to the proposal hash and policy version. Any edit returns the proposal to pending approval. Once the threshold is met, the executor creates an idempotency key from proposal ID and target ID, runs static checks, then applies small batches while reading health signals. A duplicate request returns the existing execution result; an unknown result goes to review.
>
Each proposal keeps the last verified version. If a batch fails, later batches stop, the old version is restored, and the reason is recorded; an emergency high-risk rollback still needs independent approval. Audit events are append-only and record who approved which digest, when, which targets ran, and whether rollback occurred. This enforces separation of duties and prevents an approved configuration from being silently replaced.
7. Common Failure Modes
- Binding approval only to a resource ID instead of the exact diff and version digest.
- Letting the author self-approve through a second role.
- Writing approval directly into the configuration store without execution state or acknowledgements.
- Expanding the rollout after failure or retrying non-idempotent operations indefinitely.
- Rolling back without recording version, authorization, and audit evidence.
8. Follow-Up Questions and Responses
Follow-up 1: What if the author leaves after approval?
The decision is bound to an immutable proposal and does not require the author to be online. Execution uses the service account and current policy; revoking the author does not erase the valid audit chain.
Follow-up 2: How do you prevent two approval flows from executing the same resource?
Use a resource-and-environment lock or lease, then re-check the current configuration version before execution. On conflict, the newer proposal recomputes its diff and approval.
Follow-up 3: Can an emergency change bypass approval?
Define a constrained break-glass path with two-person authorization, least privilege, short lifetime, post-change review, and complete audit. It must not become the normal shortcut.