Prompt and scope
Your company builds thousands of container images daily from GitHub Actions and self-hosted runners. Design a service that produces and verifies traceable build provenance for every image, allowing deployment only when organizational policies pass. Cover data model, issuance, verification, trust roots, key rotation, compromised runners, offline deployment, rollback, and observability.
Public supply-chain security hiring guidance uses a realistic system-design prompt: produce verifiable provenance for every container image across thousands of services and mixed runners. SLSA separates producing, distributing, and verifying provenance; Sigstore documents identity, issuer, signature, and artifact-digest checks. The problem is a trust chain with operational policy, not a signature field on a registry record.
What the interviewer is testing
- Whether you distinguish artifact digest, provenance claim, signature, transparency log, and admission policy.
- Whether untrusted source, dependencies, build configuration, and runners appear in the threat model.
- Whether claims bind to an immutable artifact rather than a mutable tag or unrelated text.
- Whether verification failure, key rotation, revocation, offline mode, and rollback have explicit semantics.
- Whether you estimate write, query, cache, retention, and audit cost and expose policy versions.
Clarifications to ask first
- Is the goal deployment prevention or audit evidence? Assume admission is a hard gate and audit evidence is stored separately.
- Are artifacts only containers? Start with OCI images and leave an extension point for binaries and packages.
- Are all builders online? Assume restricted or offline environments need preloaded roots and bundles.
- Is identity tied to a person, repository, workflow, or builder? Assume workflow identity and managed builders, not an email address alone.
- How long must evidence be retained? Ask for compliance and audit duration before sizing storage and indexes.
30-second answer
I would separate four boundaries: a builder emits provenance containing source revision, dependencies, parameters, and builder identity; an issuer binds the claims to an immutable artifact digest and records them in a transparency log; a verifier checks the signature chain, identity, digest, policy version, and time window; an admission controller returns allow, quarantine, or deny. Source, dependencies, and runners are untrusted inputs, so a successful CI run is not proof by itself. Policy versions and failure reasons are traceable, while offline environments use pinned roots and bounded-age bundles.
Step-by-step deep dive
Step 1: Map trust boundaries and threats
List source control, dependency resolution, build configuration, runners, registry, issuer, and deployment controller. Attackers may alter dependencies, steal runner credentials, move tags, forge claims, or replay old evidence. The claim to prove is which trusted builder produced which digest from which inputs; provenance does not prove the code is vulnerability-free.
Step 2: Define provenance and artifact binding
Include source revision, builder identity, entry point, locked dependencies, parameters, step digests, build time, and output digest. The digest is the authorization key; a tag is only a discovery alias. Record schema versions in claims, signatures, and decisions so field meaning cannot change silently.
artifact_digest -> provenance_digest -> signature -> log_entry
policy_version + identity + builder + time_window -> admission_decisionStep 3: Design issuance and transparency
The builder submits a claim. The issuer validates workflow identity and claim shape, then signs or returns a verifiable bundle. The signed payload must cover the artifact digest and critical claims. A transparency log helps detect anomalous issuance but does not replace deployment policy. High-volume writes may land in immutable storage first, with asynchronous indexes by digest, repository, and identity.
Step 4: Implement deployment verification
The verifier resolves an immutable digest, checks the chain, trust root, certificate identity, issuer, claim integrity, and log evidence, then applies policy. Example rules allow only main-branch builds from managed runners with current dependency checks. Return allow, quarantine, or deny with policy version and reason code, not a bare Boolean.
Step 5: Handle keys, identities, and runner compromise
Prefer short-lived workflow identities or managed keys with restricted audiences and issuance permissions. Rotation must not invalidate every historical artifact; retain old roots for their validity and revocation windows. If a runner is compromised, revoke its identity, freeze affected workflows, mark related claims, and block new admission. Re-verify or roll back deployed artifacts by environment risk.
Step 6: Cover replay, rollback, and offline verification
Claims include build time, version, and policy window; reject expired or digest-mismatched evidence. A rollback still checks that the old digest meets the current or explicitly compatible policy. Offline environments preload roots, revocation snapshots, and bundles and record snapshot age; an expired snapshot isolates the deployment instead of pretending current verification occurred.
Step 7: Size capacity, retention, and failure modes
Estimate artifact volume, average claim size, signature writes, verification QPS, and deployment bursts. Keep claims in inexpensive immutable storage and recent digests in a hot index. If issuance is down, stop or quarantine new artifacts. If verification is down, production fails closed; low-risk environments may use a short, pre-approved cache window with an explicit exception record.
Step 8: Add observability and migration
Record digest, policy version, trust-root version, reason code, and latency without storing unnecessary source or secrets. Monitor provenance coverage, signature failures, identity anomalies, policy denials, cache hits, log delay, and revoked runners. Shadow-test policy upgrades before staged rollout; every denial should be replayable and explainable.
High-quality sample answer
I would split the system into build evidence, issuance and transparency, deployment verification, and admission policy. Builders produce provenance binding source revision, dependencies, parameters, builder identity, and output digest. The issuer verifies workflow identity, binds the signature to the immutable digest, and records the event. Before deployment, the verifier checks the chain, certificate identity, issuer, claim integrity, trust root, time window, and policy version, then returns an explainable allow, quarantine, or deny. Tags never authorize. Short-lived identities, managed runners, key rotation, and revocation address credential risk; offline environments use bounded-age roots, revocation snapshots, and bundles. Claims live in immutable storage with digest indexes. Failures use fail-closed, quarantine, or a limited cache by environment, with complete policy and decision evidence.
Common mistakes
- Signing only a mutable image tag.
- Verifying mathematical signature validity without identity, issuer, or claim checks.
- Collapsing SBOM, provenance, signature, and vulnerability scan into one field.
- Trusting every CI runner and ignoring revocation blast radius.
- Rotating keys by invalidating all history or accepting revoked identities forever.
- Failing open whenever the verifier is unavailable.
- Keeping only pass/fail without policy version, reason code, or replayable evidence.
Follow-up questions and answers
How can you prove that a builder did not lie in its claim?
Provenance proves what a trusted build process asserted, not that every step was honest. Reduce assumptions with isolated builders, least privilege, reproducible or comparable builds, independent logs, and policy constraints. High-risk artifacts can require second-party review or additional attestations.
What if the registry is compromised?
Deploy by digest and verify the signature; changing a tag or registry metadata does not change the signed digest. Keep evidence and bundles in an independent immutable store. Freeze new releases, compare log entries and digests, revoke affected identities, and re-verify deployed environments after an incident.
How do you support several builders and vendors?
Use one claim schema and an identity-mapping layer for GitHub Actions, self-hosted runners, and vendor builders. Adapters normalize claims; one verifier still checks digest, identity, time, and policy. Vendors should not get separate bypass rules.
Teams say verification slows releases. What do you trade off?
Measure verification latency, cache hits, and failure reasons first. Optimize indexes and parallel reads without lowering the trust boundary. Offer an auditable short cache to low-risk environments, keep strong verification in production, and use shadow mode to distinguish real defects from policy mistakes. Every exception needs an owner, expiry, and automatic cleanup.