System Design Interview: Design a Real-Time Payment Fraud Detection Platform

Prompt and Applicable Context

Design a real-time payment fraud detection platform. It sits before payment authorization and returns one of four actions: ALLOW, STEP_UP, REVIEW, or BLOCK. A step-up asks for additional authentication. A review can delay fulfillment or another reversible business action, but the fraud service itself does not capture money.

Assume 1,000 requests per second on average and 5,000 per second at peak. Each request is about 1.5 KB. The synchronous decision has a p99 below 100 milliseconds and a 99.99% monthly availability target. Operations can manually review at most 10,000 cases per day. Decision records must be searchable for seven days and raw events archived for 400 days. These numbers are interview assumptions, not claims about a real product or legal retention requirements.

The payment service sends tokenized payment-instrument, account, merchant, device, network, amount, currency, and event-time attributes. The fraud platform must not receive a raw card number or security code. It owns risk decisions, rule and model versions, online risk features, review cases, and feedback labels. Payment authorization, 3DS execution, disputes, and the money ledger remain owned by their respective systems.

What the Interviewer Evaluates

The first signal is whether the candidate defines a decision contract instead of optimizing model accuracy in isolation. Fraud loss, false blocks, added authentication friction, review capacity, latency, and availability all constrain the policy. A model score is evidence; a versioned policy converts evidence into an action.

The second signal is time correctness. Velocity features must include the current attempt without counting an idempotent retry twice. Historical training features must contain only information available when the original decision occurred. Chargebacks and analyst outcomes arrive later, so an unlabeled transaction cannot immediately become a negative example.

The third signal is a bounded synchronous path. It performs identity and schema checks, reads a small batched feature snapshot, evaluates rules and one production model, applies policy, persists a replayable decision, and returns. Global graph traversal, large joins, model training, and case analytics belong off the critical path. Graph or batch jobs publish compact risk features that the synchronous service can read.

The fourth signal is explicit degradation. A stale feature is not the same as a safe feature, and a missing model is not a score of zero. The action changes according to feature freshness, amount, affected entity, and available fallback. A strong answer specifies when to allow, step up, review, or block during each dependency failure.

The final signal is falsifiability. Every decision records the request digest, feature provenance, rules hit, model and policy versions, score, action, reason codes, and latency. Historical replay, point-in-time feature checks, duplicate and out-of-order events, hot-key load, dependency failures, adversarial traffic, and shadow or canary comparison can then test the design.

Questions to Clarify Before Answering

• Where is the platform placed? This design makes a synchronous decision before authorization. A purely asynchronous

detector could find rings and support investigation but could not prevent the current payment.

• What actions are available? The prompt provides four actions. REVIEW is restricted by the 10,000-case daily budget;

it cannot be a convenient answer for every uncertain score. The payment product defines what is reversible while a case waits.

• What identifiers are supplied? Require a stable decision_id, tokenized instrument ID, account ID when present,

merchant ID, and device ID. Raw card data is outside the boundary. Missing optional identity becomes an explicit feature, not a fabricated value.

• How fresh must each feature be? A ten-minute velocity rule cannot use an hour-old counter. Each feature group has a

maximum age and fallback policy. Profile features may tolerate minutes; critical velocity state may tolerate only seconds.

• How and when do labels arrive? Analyst disposition is an early but fallible signal. A later confirmed dispute is a

stronger label. Label source, observation time, maturity state, and version must be stored so corrections do not rewrite history invisibly.

• Is cross-entity graph detection synchronous? No global traversal is required under the 100-millisecond budget.

Offline or streaming graph jobs publish compact entity-risk or neighborhood-risk features. An incident-specific lookup can be added only after measuring its latency and availability.

• What is the fail posture? There is no universal fail-open or fail-closed answer. Low-value known customers may be

allowed under a rules fallback, while high-value new-device payments may require step-up or block when critical features are unavailable.

• What are the privacy constraints? Retention, geographic processing, investigation access, and feature legality must be

confirmed with security, privacy, and compliance owners. This answer minimizes identifiers and audits access but does not invent a jurisdiction-specific rule.

30-Second Answer Framework

“I would start with a versioned decision contract: one decision_id, immutable request digest, four actions, a 100-millisecond p99 budget, and a hard review-capacity constraint. The synchronous service batch-reads fresh online features, obtains idempotent per-entity velocity counts that include the current attempt, evaluates hard rules and a model, then applies a policy and durably records all provenance before returning. Events also feed a stream processor and an offline store. Training uses point-in-time features and versioned, matured labels. Global graph work stays asynchronous and publishes compact risk features. Missing or stale dependencies trigger an amount- and identity-aware degradation matrix, not a silent zero score. Shadow replay, canaries, hot-key tests, and failure injection verify both fraud outcomes and customer friction.”

Step-by-Step Deep Dive

Step 1: Turn the prompt into budgets and invariants

At 1,000 requests per second, the platform sees 86.4 million decisions per day. A 1.5 KB request produces about 129.6 GB of raw ingress per day before replication and indexing; the 5,000-per-second peak is about 7.5 MB per second. If a compact decision record averages 2 KB, seven hot days are about 1.21 TB before indexes and replicas. These are sizing anchors, not precise storage forecasts. Compression, schema overhead, and indexes must be measured.

The review budget is the sharper product constraint: 10,000 cases are only about 0.012% of 86.4 million daily decisions. The policy therefore needs priority and admission control. When the queue is full, it must move the lowest-priority review band to ALLOW with a guardrail, STEP_UP, or BLOCK according to expected loss and friction; it cannot create an unbounded backlog.

Keep review quota in the decision service's authoritative database. A conditional daily counter, optionally divided into reserved risk bands, is claimed by decision_id in the same transaction that creates the decision and review case. The low review volume does not justify a separate distributed quota system. If the claim fails, policy evaluates the declared overflow action before committing, so concurrent servers cannot admit more than the daily limit.

An illustrative 100-millisecond budget is 8 milliseconds for authentication and validation, 25 for batched online features and velocity state, 10 for rules, 20 for model inference, 15 for policy plus durable decision write, and 22 for network and tail-latency headroom. Each stage gets a timeout shorter than its budget. The invariants are:

one decision_id identifies one immutable request digest
an idempotent retry returns the original decision and does not increment velocity twice
every returned action has a persisted policy, model, rule, and feature provenance record
missing or expired critical features cannot be interpreted as normal values
review admissions never exceed the configured operational capacity
training features contain only values available at the historical decision time
labels retain source, observation time, maturity state, and version
raw payment card data never enters the fraud platform

Step 2: Define the API and decision record

Keep the synchronous interface narrow:

POST /v1/risk/decisions              create or replay a decision
GET  /v1/risk/decisions/{decision_id} read the immutable result and current case status
POST /v1/reviews/{case_id}/disposition record an analyst outcome
POST /v1/feedback                     ingest a dispute or trusted fraud outcome

The create request contains decisionid, eventat, amount and currency, tokenized entity IDs, merchant and channel, and current request attributes. The response contains action, stable reason codes, optional step-up type or review case ID, and decisionversion. A unique key on decisionid stores a normalized request hash. The same ID and hash replay the original response; the same ID with a different payload returns a conflict.

Store separate responsibilities:

• risk_decisions: identifiers, request hash, event time, score, action, reason codes, feature snapshot and freshness,

rule bundle, model, policy versions, latency, and degradation state;

• review_cases: decision reference, priority, queue state, assignee, disposition, and timestamps;
• feedback_labels: decision reference, label, source, observed time, maturity state, confidence, and version;
• policy_bundles: signed immutable rule, threshold, review-budget, and fallback configuration;
• outbox_events: the committed decision event awaiting asynchronous publication.

Persist the decision and outbox event in one local transaction before returning. The event stream is at least once, so downstream consumers deduplicate by decision_id and event version. The payment service treats the immutable result as advice for the named request; it cannot reuse the result for a different amount or instrument.

Step 3: Separate the synchronous path from learning pipelines

The synchronous data flow is:

payment service
  -> decision API
  -> idempotency lookup
  -> batched online feature + velocity read
  -> hard rules
  -> model inference
  -> versioned action policy
  -> decision store + outbox
  -> ALLOW | STEP_UP | REVIEW | BLOCK

The asynchronous flow consumes decision, authentication, payment-result, review, and dispute events. A stream processor deduplicates them, applies event-time windows, and updates online entity features such as attempts in ten minutes, distinct merchants in one hour, amount deviation, device age, and recent failed authentication. The raw immutable events and feature history also enter an offline store for analysis and training.

This follows the useful feature-store split: the online store keeps the latest values for low-latency serving, while the offline store holds historical time-series values for training and materialization. They must share feature definitions, types, entity keys, and transformation tests. They do not need to share a storage engine. A single database serving both arbitrary historical scans and predictable low-latency lookups couples two conflicting workloads.

Rules and model are complementary. Hard policy constraints, trusted blocklists, and high-confidence velocity limits are explicit and fast. The model combines weaker signals and interactions. The final policy maps rule results, score, freshness, amount, identity confidence, and review capacity to an action. A model-only design is hard to operate during incidents; a rules-only design is a valid first version but becomes brittle as behaviors change.

Step 4: Make velocity features include the current attempt exactly once

A stream-updated counter may lag behind the request that is being scored. Two concurrent card-testing attempts could both read the same old count. For a small set of critical velocity rules, use a partitioned risk-state service with an idempotent operation such as:

observe(entity_type, entity_id, window, decision_id, event_at, value)
  -> count, sum, distinct_estimate, state_version, freshness

The service orders updates per entity key, stores decision_id in the window's deduplication state, includes the current attempt, and returns the resulting count. A retry therefore returns the same observation instead of incrementing again. State partitions are replicated, checkpointed, and rebuilt from the event log. High-cardinality approximate distinct features can use bounded sketches, while exact block thresholds use exact counters.

One transaction touches account, instrument, device, IP prefix, and merchant. A global atomic transaction across every entity would damage latency and availability. Query them in parallel with per-entity idempotency. If a subset times out, record which group is unavailable and let policy degrade; never replace the missing value with zero. Route and capacity test known hot entities separately because an attacked instrument or IP can concentrate traffic on one partition even when total QPS is normal.

Event-time processing must handle duplicates, late events, and idle partitions. Watermarks describe event-time progress; they do not make late data disappear. Define an allowed lateness per feature, emit corrections with a higher feature version, and monitor watermark delay. Online decisions keep the exact snapshot they saw. A later correction improves future decisions and offline analysis but does not pretend that the earlier service knew the corrected value.

Step 5: Enforce freshness and a deliberate degradation matrix

Each feature group returns computed_at, source event time, version, and maximum allowed age. The feature service derives FRESH, STALE, MISSING, or ERROR; the policy consumes that state directly. The model should receive trained missing indicators only for expected sparse data, not for an outage disguised as ordinary nulls.

Use a concrete failure matrix:

The exact cells are business decisions, but they must be versioned and tested. Blanket fail-open can convert an outage into loss; blanket fail-closed can convert it into a customer and revenue outage. If both model and critical velocity state are missing, the policy can use amount, trusted identity, merchant risk, and authentication availability, but it must surface a distinct degraded reason and page operators.

Step 6: Build point-in-time training data and mutable label history

For each historical decision, use its decision_at as the boundary. A feature row is eligible only when the underlying event occurred and became available by that time. Point-in-time joins must account for late arrival; recomputing last month's seven-day count from today's corrected warehouse would leak information the online service did not possess.

Labels have a lifecycle:

UNOBSERVED -> PROVISIONAL_ANALYST_OUTCOME -> MATURE_CONFIRMED_OUTCOME
                                      \-> CORRECTED_VERSION

The exact maturity policy depends on the payment and dispute process. Store every label version rather than overwriting an analyst decision when a later dispute arrives. Training selects a declared label definition and maturity cutoff. Evaluation uses forward time splits, entity-aware checks for duplicates or linked cases, and a final untouched time window. Offline and online feature parity tests replay the same raw events through both implementations and compare values at the decision boundary.

Monitor more than aggregate precision. Measure fraud loss or prevented loss, false-positive value and rate, authorization and step-up completion, review yield and queue age, score calibration, feature drift, label coverage, and performance by merchant, channel, geography, payment method, amount band, and identity state where such slicing is lawful. A threshold that looks good globally can silently block one segment.

Step 7: Keep graph detection off the critical path

Fraud rings connect accounts, devices, instruments, addresses, merchants, and network identifiers. Traversing the complete graph for every payment conflicts with the latency and dependency budget. Streaming and batch jobs instead compute compact features such as risky-neighbor count, shared-device fan-out, component risk, and time since connection to a confirmed bad entity. The online store serves the latest version with freshness metadata.

This creates a known detection delay. For a newly discovered campaign, the operator can deploy a narrow signed rule or blocklist while the graph pipeline catches up. A future synchronous graph query is justified only if replay shows material incremental prevention, its p99 fits the remaining budget, and its outage has an explicit fallback. Graph evidence should also produce investigator-readable paths or reason codes rather than an unexplained score.

Step 8: Roll out, secure, observe, and verify the system

Rules, models, features, and policy thresholds have independent immutable versions but one signed policy bundle pins the combination used for a decision. A new bundle first replays mature historical traffic, then runs in shadow without changing actions, then receives a small canary slice. Promotion compares fraud loss, false positives, approval, step-up completion, review admission and yield, latency, feature freshness, and fallback rate. Rollback changes the active bundle pointer; old decisions remain reproducible.

The service accepts tokenized IDs, encrypts sensitive attributes in transit and at rest, applies least-privilege access, audits investigator and policy changes, redacts logs, and separates policy approval from deployment. Data deletion and retention jobs operate by documented identifier mappings and produce auditable outcomes. Training exports are access- controlled and cannot include review notes or future outcomes as features.

Verification includes:

• replay the same decision_id with the same and different payloads;
• send duplicate, late, and out-of-order events across watermark boundaries;
• compare offline and online features at historical decision times;
• load test 5,000 requests per second plus hot entity keys and a cold-cache start;
• kill the model, online store, stream processor, one state partition, and review system independently;
• exhaust review capacity and verify the configured admission policy;
• shadow and canary a deliberately stricter rule, then roll it back;
• probe feature poisoning, identifier fan-out, reason-code leakage, unauthorized policy changes, and replay abuse.

Operational dashboards separate system health from decision quality. System signals include endpoint latency, errors, dependency timeouts, feature age, stream lag, state-rebuild progress, fallback actions, and queue age. Outcome signals are recomputed on mature labels and always tagged with the policy and model versions that produced them.

High-Quality Sample Answer

“I would first fix the contract. We score 1,000 payments per second on average and 5,000 at peak, return one of four actions within 100 milliseconds at p99, and may admit only 10,000 reviews per day. That means review is a scarce action, model accuracy is not the sole objective, and the policy must price fraud loss against false blocks and step-up friction.

The payment service calls POST /v1/risk/decisions with a stable decision ID, tokenized entity IDs, event time, amount, and request attributes. A unique decision ID stores an immutable request hash. Same request means replay; changed payload means conflict. The service batch-reads versioned online features and idempotently observes the current attempt in critical per-entity velocity windows. It then evaluates hard rules, one model, and a signed policy bundle. The decision, reason codes, feature freshness, rule hits, model and policy versions, and outbox event commit before the action returns.

The event stream updates online aggregates and stores raw history offline. Training performs point-in-time joins at the original decision time and chooses only label versions mature by a declared cutoff. Graph jobs remain asynchronous and publish compact neighbor-risk features. Every feature group carries age and availability. If a model or critical counter fails, policy selects a rules fallback, step-up, bounded review, or block using amount and identity risk; it never treats an outage as a zero-risk value.

I would release a bundle through historical replay, shadow traffic, and canary traffic. I would compare fraud loss, false positives, approval and step-up completion, review yield, latency, feature freshness, and fallbacks by meaningful slice. Duplicate requests and events, late data, hot keys, dependency outages, review saturation, and adversarial feature inputs are explicit tests. That makes the platform low-latency, capacity-aware, and able to explain and reproduce any decision.”

Common Mistakes

• Mistake: optimize AUC or accuracy and choose one threshold. Why it fails: those metrics omit loss magnitude,

legitimate-customer friction, review capacity, and operational failures. Fix: define an action policy with cost and capacity constraints, then monitor outcome and experience metrics.

• Mistake: read a stream-updated counter and assume it includes the current attempt. Why it fails: concurrent attacks

can observe the same stale count, and retries may double count. Fix: make critical velocity observation per-entity and idempotent, and record its state version and freshness.

• Mistake: use NULL or zero for an unavailable feature. Why it fails: an outage becomes an ordinary low-risk input.

Fix: carry availability and age into policy and apply a tested degradation matrix.

• Mistake: query the global fraud graph synchronously. Why it fails: unpredictable traversal and a large dependency

surface break the latency SLO. Fix: publish compact graph features asynchronously and justify any online lookup with measured incremental value.

• Mistake: label every non-disputed transaction as legitimate immediately. Why it fails: outcomes are delayed and the

newest negatives have incomplete observation windows. Fix: version labels and train only on a declared mature window.

• Mistake: recompute historical features from today's warehouse. Why it fails: late and corrected data can leak future

knowledge. Fix: use event and availability times for point-in-time joins and retain the served snapshot.

• Mistake: send every uncertain case to review. Why it fails: 10,000 reviews cover only about 0.012% of daily traffic.

Fix: prioritize by expected avoidable loss and enforce queue admission and overflow behavior.

• Mistake: deploy a rule or model directly to all traffic. Why it fails: a technically available service can still

cause mass false blocks. Fix: historical replay, shadow evaluation, canary slices, signed versions, and fast rollback.

Follow-Up Questions and Responses

Follow-up 1: Two concurrent attempts on the same instrument both see a count below the limit. What changes?

Move the critical rule from a passive cached aggregate to an idempotent per-entity observation. The state partition orders updates for that instrument, records each decision ID once, includes the current attempt, and returns the new count and version. Other entity features can remain eventually consistent. Load testing must include a single hot instrument because uniform QPS tests will not expose this partition bottleneck.

Follow-up 2: The model service is down for fifteen minutes. Do you allow or block?

Use the versioned failure matrix. Hard blocklists and critical velocity rules continue. A low-value payment with trusted identity may use rules-only allow; a new-device or high-value payment may step up, enter the bounded review queue, or block. All fallback actions carry a degradation reason. Monitor both dependency recovery and the business effect; do not silently serve a default score.

Follow-up 3: Chargeback labels take weeks, but a new campaign starts today. How do you adapt?

Use leading signals such as authentication failure, analyst disposition, merchant reports, and concentrated shared-entity patterns for investigation, while keeping their provenance separate from mature labels. Deploy a narrow reversible rule through shadow and canary stages. Retrain only when the chosen label definition has sufficient mature coverage; otherwise the newest apparently legitimate examples bias evaluation.

Follow-up 4: Review demand rises to 50,000 cases per day while capacity remains 10,000. What happens?

Rank cases by expected avoidable loss, evidence quality, amount, and time sensitivity; reserve capacity for required segments and admit only the top 10,000. The remaining band follows a preapproved step-up, allow, or block policy. Track queue age and analyst throughput. Adding messages to a queue without an executable service level merely hides the overload.

Follow-up 5: Why not make the online feature store the only source for training too?

It retains latest values for low-latency reads and usually cannot reconstruct what was known at millions of historical decision times. Training needs time-series history, point-in-time joins, backfills, and large scans. Use shared feature definitions and parity tests across separate online and offline stores rather than forcing one engine to serve conflicting workloads.

Follow-up 6: A graph job finds a fraud ring after some payments were allowed. Can you rewrite those decisions?

No. Preserve the original action and exact provenance. Add a new finding or label version with its observation time, take permitted downstream action, update online entity risk for future decisions, and include the case in replay. Rewriting the old decision would erase what the service actually knew and break audit and model evaluation.