Prompt and context
A multi-tenant platform's read-only API key is committed to a public repository. The key may have been copied, and callers are spread across 200 services that cannot all be shut down immediately. Design detection, alerting, revocation, impact analysis, rotation, and verification to minimize the attack window without taking every healthy tenant offline.
This is a lifecycle and incident-handling question for backend, security-engineering, and platform-engineering roles. The read-only scope, 200 callers, and inability to shut everything down immediately are interview assumptions, not industry benchmarks. Focus on treating exposure as an observable security event, moving detection earlier, propagating revocation, bounding impact, and orchestrating recovery. You do not need to design a complete cryptographic key-management platform.
What the interviewer is testing
First, can you distinguish “the text looks like a key” from “the credential is valid”? A regex or provider fingerprint creates a candidate; a restricted validation path, status lookup, or internal metadata must confirm it without printing the secret.
Second, can you connect prevention, detection, and response? Pre-commit or push protection reduces entry into a repository, historical and public-repository scans catch misses, the key service revokes, request logs support impact analysis, and deployment systems rotate callers.
Third, can you define revocation semantics? Changing a database status does not make an edge cache reject immediately, and it does not invalidate a static credential already copied by an attacker. A strong answer names a propagation target, cache policy, and downstream rotation order.
Finally, can you make a bounded security-versus-continuity choice: revoke the key rather than the whole tenant, allow a short dual-key window for routine rotation but not by default after exposure, and make every step auditable, retryable, and reversible?
Questions to clarify first
- What credential is it: read-only API key, write key, cloud credential, or user token? Scope changes blast radius and ordering.
- Where was it exposed: public repository, private repository, logs, chat, or an artifact? Is history mirrored or still accessible?
- How can validity be confirmed without making a risky business call?
- Does revocation mean rejecting new requests within seconds, or also invalidating the old credential in a downstream system?
- Do the 200 services share deployment, configuration, sidecar, or secrets-manager paths? Which cannot update automatically?
- Can the tenant tolerate a short dual-key window? Which writes must stop immediately, and which reads can degrade safely?
- Which evidence must be retained, and who can approve an exception or break-glass action?
A 30-second answer
“I would treat the candidate as potentially exposed and never echo the secret from the scanner. I would identify the credential, scope, tenant, and last use, revoke or reduce high-risk access immediately, and make every gateway meet a measurable propagation target. Then I would use request logs to bound the time window, routes, and suspicious sources; create a replacement through the secrets manager; roll it out in batches; and retire the old key after observing usage. Push blocking, historical scans, request auditing, and rotation drills form the durable loop. Only a provably invalid false positive should enter an audited exception path.”
Step-by-step answer
Step 1: Establish incident states and an evidence boundary
Model detected → triaged → contained → rotated → verified → closed. Each transition records an incident ID, public credential identifier, tenant, source, scope, time, and owner. The scanner stores hashes, fingerprints, and locations, never the complete secret; analysts use a permissioned internal validation path.
Candidates can come from pre-commit scanning, repository history, public-repository events, CI logs, artifact scans, or provider notifications. GitHub push protection blocks pushes containing detected secrets and creates alerts for bypasses. It reduces new history entries but does not replace historical scanning or runtime monitoring.
Step 2: Confirm validity and calculate blast radius
Use prefix, length, checksum, or provider rules for local filtering, then call a validation endpoint that does not expose the secret. The response should be valid, invalid, revoked, or unknown plus safe metadata. Do not let a scanner use production write privileges; if validation is necessary, use a read-only, low-cost, isolated tenant and an audit marker.
After confirmation, read scopes, tenant, creator, environment, expiry, last use, and the dependency list for 200 services. Query request logs between discovery and revocation. Separate normal sources from unknown networks, unusual regions, unexpected routes, denial spikes, and high-value resource reads. Logs contain only the public key ID, tenant, and correlation ID, never an Authorization header or secret.
Step 3: Contain first, then plan migration
Revoke high-privilege, write, or money-moving keys first. Treat a read-only key as exposed even when no abuse is visible. Write authoritative revocation state, publish invalidation events, and use a short gateway TTL as a fallback for lost events. Set a target such as “all gateways reject new requests within five seconds,” then test message loss, node restart, and partitions.
Do not give an exposed key a long grace period for the convenience of 200 services. A non-exposed routine rotation can use a dual-key window; a suspected leak is revoked first. If necessary, a low-risk read endpoint may return a safe degraded result for a short period. The degradation must not disclose more data or look like a successful write.
Step 4: Orchestrate an auditable batch rotation
Generate an independent replacement per caller, with no more privilege than the old key; prefer short-lived or workload identity. Deliver it through a shared secrets manager or deployment configuration in batches: a small canary, then service groups. Each service follows an idempotent state machine such as prepared → deployed → observed → old-revoked; retries must not mint unlimited keys or reactivate the old one.
Stripe recommends rotating immediately after exposure, even when you cannot prove that someone saw the key. Restricted keys and source-IP controls reduce blast radius. OWASP also recommends recording creation, use, rotation, deletion, purpose, and ownership metadata, and using short-lived or dynamic credentials where possible.
Step 5: Verify closure, not just deployment
Run four checks: every gateway rejects the old key consistently; the new key can access only the allowed tenant and routes; logs contain no old key ID or suspicious source; and health, business metrics, and error budgets for all 200 services recover. For services that cannot update automatically, name an owner, deadline, and temporary isolation policy instead of treating “new configuration sent” as completion.
Before closing, retain an irreversible evidence summary, timeline, permission changes, approvals, and representative suspicious requests. Destroy the secret material according to policy. The review should explain why detection did not happen earlier, why callers shared a key, which logs lacked a key ID, and whether revocation met its target. Convert the answers into testable actions.
Model high-quality answer
“I would treat the read-only key as exposed. The scanner would never echo it; it stores a fingerprint and repository location, and a restricted validation endpoint confirms the key ID, tenant, scope, and state. I would revoke high-risk credentials immediately, write authoritative state, publish invalidation events, and require every gateway to reject new requests within five seconds, tested under message loss, cache behavior, and node restart.
I would query request logs between discovery and revocation to identify the 200 callers, routes, source networks, and suspicious reads. Logs would contain only the public key ID, not the Authorization header. I would generate an independent least-privilege key per service, deliver it through the secrets manager in canaries and batches, and observe old and new key traffic. An exposure gets no long grace period; dual keys are for routine rotation.
Verification means more than a successful deployment: every gateway rejects the old key, the new key enforces tenant and route authorization, old-key traffic reaches zero, and service errors return to budget. I would preserve the incident timeline and safe evidence, destroy secret material, and add pre-commit blocking, historical scans, runtime anomaly alerts, independent keys, and rotation drills. That shortens the attack window without taking the whole tenant or every service offline.”
Common failure modes
- Logging the complete key after a match → creates a second leak → keep only fingerprints, key IDs, and locations.
- Letting regex decide exposure → false positives and unknown formats misroute response → confirm through restricted validation and provider metadata.
- Waiting for abuse before revoking → a static key may already be copied → treat exposure as potential compromise, contain first.
- Deleting a database row and stopping → gateway caches and downstream systems can still accept the key → measure propagation, invalidate caches, and verify downstream rotation.
- Banning the entire tenant → blast radius expands and recovery becomes harder → bound by key, scope, route, and time window.
- Giving the exposed key a long dual-key window → the attacker keeps access → reserve dual keys for non-exposed routine rotation.
- Switching every service at once → one bad configuration causes a fleet outage → use canaries, batches, idempotent state, and rollback.
- Treating delivery as completion → a service may not reload and still uses the old key → test old-key rejection, new-key authorization, and business metrics.
Follow-up questions
Follow-up 1: The scanner cannot prove a candidate is valid. What now?
Treat it as potentially exposed to shorten the valid window. Add evidence with a non-secret-returning validation endpoint, repository context, and internal key metadata. If false-positive cost is high, an authorized security reviewer can approve a time-bounded exception with a reason; the scanner must not self-approve.
Follow-up 2: A legacy system cannot hot-reload a new key. How do you rotate?
Prepare a new deployment or short dual process, validate the new key, shift traffic, then revoke the old one. If that is impossible, isolate the service’s permissions and egress, set a short deadline, and audit the manual step. Legacy constraints do not justify leaving an exposed key valid indefinitely.
Follow-up 3: Revocation takes longer than five seconds. Keep serving or shut everything down?
Tier by scope and business risk. Writes, payments, and high-value routes fail closed when state is stale; low-risk reads may use an explicitly labeled short degradation. Increase isolation and throttling, repair invalidation or cache paths, then decide whether a broader block is necessary.
Follow-up 4: The attacker already read data with the old key. What next?
Preserve evidence, bound the time, tenant, routes, and data, notify affected parties, and assess breach-reporting duties. Revocation and rotation stop further use; review exports, caches, asynchronous jobs, and downstream copies for secondary exposure.