Prompt and scope
A multi-tenant platform must audit every administrator access and change to user data. Design a verifiable tamper-evident audit log and explain writes, hash chains, external anchoring, queries, retention, and recovery.
Append-only records and hash chains make later edits detectable, but they do not prove an event was truthful when first written. The question tests threat modeling, evidence, and operations; immutable object storage alone is not a complete design.
What the interviewer evaluates
Evaluate event fields for actor, target, action, time, source, and result; reliable coupling to business transactions; segmented and anchored hash chains; tenant isolation, indexes, key permissions, retention, deletion, and a usable verifier.
30-second answer framework
“I would define an immutable canonical event and publish it through an outbox after the business transaction commits. An isolated writer groups events by tenant and time, builds hash chains, and periodically anchors each chain head in an independent append-only store or transparency ledger. Queries use rebuildable indexes, never mutate evidence. A verifier recomputes chains and compares anchors. Integrity comes from the chain; event truth still comes from business authorization and transaction records.”
Step-by-step deep answer
Step 1: Define the threat model and evidence goal
Assume an application or database administrator may delete, reorder, or rewrite records, and the anchor service may be unavailable. Decide whether the goal is to detect missing, changed, or reordered events, or to prove that a subject acted. An audit log does not replace access control.
Step 2: Design canonical events
Include event ID, tenant, actor and identity source, action, resource, before and after digests, request ID, event time, server sequence, result, and policy version. Canonicalize JSON and similar fields so serialization order does not change hashes. Store only digests or controlled references for sensitive values.
Step 3: Couple business and audit writes
The business transaction writes an outbox; a reliable consumer emits the event after commit. Do not rely on best-effort logging after a business write. Retries are idempotent by event ID; failures go to an isolated queue and alert. Audit latency may be bounded, but loss cannot be silent.
Step 4: Build segmented hash chains
Store the previous hash, canonical event, and current hash, such as H(previous || event || metadata). Roll segments by tenant, date, or size and record segment start, end, and sequence. Cross-shard order uses server sequence and receive time rather than client clocks alone.
Step 5: Anchor externally and manage keys
Periodically write each segment head to an isolated append-only medium with anchor time, segment ID, and signature. Keep signing keys in a controlled key service; rotation and revocation are themselves audited. An independent anchor prevents a database writer from recomputing and replacing an entire chain invisibly.
Step 6: Isolate queries and permissions
Build actor, resource, action, and time indexes in a search index or read replica. Indexes are rebuildable; evidence is not updateable. Enforce tenant, role, and purpose checks, and include a verification digest in exports. Every read, export, or verification is a new audit event.
Step 7: Handle retention, deletion, and privacy
Set retention, WORM, or object-lock policies by regulation. For personal-data deletion, use cryptographic erasure, field redaction, or irreversible digests while retaining proof that deletion occurred; never silently rewrite history. Align backup, cache, and anchor retention.
Step 8: Verify, recover, and monitor
The verifier checks chain continuity, sequence, canonical hashes, signatures, and external anchors. Run periodic sample and full verification, monitoring gaps, lag, duplicates, anchoring failures, and verification time. Recover by replaying outbox and append-only segments from the last trusted anchor, isolating unprovable ranges.
Trade-offs and boundaries
One chain versus segmented chains
A single chain simplifies queries but complicates recovery and concurrent writes. Segments enable tenant isolation, parallel verification, and retention management, at the cost of segment anchors and ordering metadata.
Hash chain versus signed log
Hash chains are inexpensive and detect edits; signatures add cross-organization verification and key-management cost. Combine them when external proof is required.
Integrity versus truth
A chain proves relationships between records and anchor consistency, not that event content was true. Truth still depends on identity, authorization, transactions, and independent evidence.
Failure drills and evolution
A middle record is deleted
Delete one object from a segment and verify a sequence gap and chain break are reported with the segment and anchor location.
A database administrator recomputes hashes
Rewrite a segment and replace its head; the independent anchor must reject the new chain while the old chain remains retrievable for investigation.
The anchor service is unavailable
Disconnect it and verify segments enter a pending-anchor state, events remain durable, and anchors are added in order after recovery with an alert.
Common mistakes and follow-ups
Mistake 1: Only writing to immutable storage
Ask whether the application or administrator can forge or omit an event before storage, and how it is coupled to the business transaction.
Mistake 2: Ordering only by client time
Ask how clock drift and replay affect order; use server sequence and receive time.
Mistake 3: Updating evidence to fix an index
Ask how query performance is preserved; indexes should be rebuildable while evidence stays append-only.
Mistake 4: Ignoring deletion and privacy rules
Ask how deletion preserves verifiable fact without retaining the original personal data.
Mistake 5: Equating chain consistency with event truth
Ask who proves that the actor really performed the action; authorization and independent business records are needed.
Extended follow-ups and reference answers
What does an external anchor solve?
It prevents someone with primary-database write access from silently recomputing and replacing the entire chain later. The anchor also needs independent permissions and retention.
How do you support multi-tenant queries?
Partition by tenant and authorize indexes; record every query, export, and verification as another audit event, denying cross-tenant access by default.
How do you handle an unprovable range?
Isolate and mark the gap, preserve original objects and anchor evidence, and notify security. Recovery must not automatically label an unknown range complete.