System Design Interview: Design a Centralized Logging System

Problem and Scope

Design a centralized logging system shared by multiple engineering teams. One hundred thousand service instances, containers, and batch jobs produce 1 million events per second at steady state, averaging 600 bytes of raw data per event. A large incident can sustain 3 million events per second for 15 minutes because failed requests, retries, and stack traces all increase together. The p99 target from event creation to search visibility is 30 seconds. Common queries over the last 15 minutes, with tenant, time, service, environment, and severity filters, should finish under 2 seconds at p95.

Users need filters by time, service, host, severity, and traceId, plus bounded full-text search, downloads, and alerts. Log classes have different retention and reliability requirements. Debug logs can have short retention and may be sampled during an emergency. Operational errors need durable retention. Security audit logs cannot use the ordinary degradation policy. The system also needs tenant quotas, field redaction, access auditing, lifecycle management, and historical restoration.

The scale, latency, retention, and reliability targets are interview inputs, not public commitments from a logging vendor. The scope does not include implementing an entire search engine or rewriting a message broker during the interview. It also does not require copying a company's internal architecture. Public 2026 system-design prompts still ask candidates to design centralized logging with burst handling, search, retention, and tenant isolation. OpenTelemetry's stable log data model, its Collector troubleshooting guidance, and Elastic's documentation on data tiers and mapping explosion provide primary evidence for the important engineering boundaries.

What the Interviewer Is Evaluating

First, can the candidate separate durable acceptance from immediate searchability? The gateway can acknowledge an event after committing it to a replicated durable stream. The search index may lag by tens of seconds. If success requires a write to the search cluster, index maintenance and scaling directly backpressure every producer.

Second, does the candidate recognize correlated failure during an incident? The time when logs are most valuable is also when application errors, retry storms, and search-cluster pressure often rise together. A complete answer includes local disk buffering, a central durable queue, backpressure, finite capacity, priority-based shedding, and measurable drop counts. Saying only “add Kafka and Elasticsearch” does not close the failure path.

Third, does the indexing plan understand cost and cardinality? Dynamically indexing every arbitrary JSON attribute lets high-cardinality fields create mappings, dictionaries, and shard overhead. A sound design indexes stable, frequently queried, well-typed fields. Other attributes can still be stored without becoming indexed by default.

Fourth, does tenant isolation cover both ingestion and queries? Tenant identity must come from authenticated credentials, not a tenantId supplied inside the log body. One high-volume tenant, broad regular expression, or field explosion must not exhaust shared queues, indexing threads, caches, or query concurrency.

Finally, do capacity, delivery semantics, security, and validation form one coherent system? The candidate should calculate raw throughput and burst backlog, admit that at-least-once delivery creates duplicates, explain why audit and debug logs need different policies, and use end-to-end canaries to prove that losses are never silent.

Clarifying Questions Before Answering

• Which logs must not be lost? Audit and security events require durable retention. Debug logs may be sampled or

dropped under an explicit degradation policy.

• Where is the success boundary? The central gateway acknowledges after committing the event to a durable stream

replicated across availability zones, without waiting for search-index refresh.

• May logging block the producer? Ordinary application logs must not synchronously block business requests. A node

agent uses asynchronous batches and a finite local disk.

• What are the primary query patterns? Structured filters with a tenant and time range, exact traceId lookup, and

full-text search over a bounded window.

• Is global ordering required? No. Preserve approximate order within one source instance and use event and observed

timestamps to help order events across sources.

• What are the retention assumptions? Debug logs have 24 hours in the hot index and 7 days in raw archives.

Operational logs have 7 days hot and 90 days archived. Audit logs have 30 searchable days and 365 days in immutable archives.

• Do we need active writes across regions? The primary design collects into a regional pipeline. A unified control

plane and disaster-recovery queries can read archives.

• Do we index every field? No. Stable top-level fields and allowlisted attributes may be indexed. Arbitrary

high-cardinality attributes are stored only by default.

• How are sensitive values handled? SDKs and node agents prevent or redact them first, and central processors check

again. Secrets and tokens should not enter logs.

• How broad may a query be? Require a time range, cursor pagination, scan budgets, concurrency limits, and

cancellation. Large historical exports use asynchronous jobs.

30-Second Answer Framework

“Applications write to stdout or an asynchronous SDK. A node agent batches, redacts, and uses a finite disk spool. A regional gateway authenticates the workload, binds the tenant, applies quotas, and validates the format before writing to a durable stream replicated across availability zones. That commit is the acknowledgment boundary. Processors normalize events, write raw records to object storage, and write approved fields plus selected text to a hot search index. The query service requires tenant and time predicates and routes windows across hot, warm, and archive storage. Steady raw traffic is about 600 MB/s or 51.84 TB/day. A 15-minute 3x burst creates about 1.08 TB of backlog above steady processing. Delivery is at least once with eventId deduplication. During overload, preserve audit and error logs first, sample debug logs first, and expose every drop as an alertable metric.”

Step-by-Step Deep Dive

Start with six invariants:

1. An ordinary business request must not synchronously depend on the remote logging platform.
2. The central system acknowledges an event only after it reaches replicated durable buffering.
3. Tenant identity comes from authentication, and ingestion, storage, search, and export cannot cross tenant boundaries.
4. Audit logs do not use the sampling and shedding policy of debug logs.
5. Arbitrary user attributes are not dynamically indexed by default, so high-cardinality fields cannot grow the schema

without a bound.

1. Drops, delays, parse failures, and redaction failures are measurable and never silent.

Step 1: Define the collection path and acknowledgment boundary.

Application stdout / asynchronous SDK
  -> node agent or sidecar: batch, compress, redact, local disk spool
  -> regional ingestion gateway: authenticate, bind tenant, quota, mandatory redaction, format and size limits
  -> durable stream replicated across availability zones
  -> normalization and routing processors
       -> raw object-storage archive
       -> hot search index
       -> alerts and streaming subscriptions
  -> query coordinator -> hot / warm / archive readers

Applications normally write to stdout so the node agent remains decoupled from business processes. An asynchronous SDK is reasonable for explicitly structured events, but it still writes to an in-memory queue or local agent. The agent batches by bytes and time, limits individual event size, persists its position, and uses a finite disk spool during network failure. When the disk fills, it follows a policy by log class instead of blocking forever or consuming unbounded disk.

The regional gateway derives tenant, service, and environment from mTLS identity, workload identity, or short-lived credentials, overriding spoofed fields in the body. It applies rate limits, decodes compression, performs basic schema checks, enforces a maximum event size, and applies mandatory secret and token redaction before data enters the central durable buffer. It acknowledges only after the event is replicated into a durable stream across availability zones. Search, archives, and alerts are separate consumers, so one failed consumer does not become a direct dependency of every producer.

Step 2: Use a common data model with controlled extension points.

LogEvent(
  eventId, timestamp, observedTimestamp,
  tenantId, service, environment, instance,
  severityNumber, severityText, body,
  traceId, spanId, schemaVersion,
  attributes, sensitivityClass
)

timestamp is when the event occurred, while observedTimestamp is when the collection system first saw it. If source clocks drift, the latter still exposes ingestion order and delay. traceId and spanId connect logs with distributed traces. severityNumber supports normalized comparison while severityText preserves the producer's wording. schemaVersion lets processors upgrade parsing rules safely.

Stable top-level fields have explicit types and indexes. attributes retains extra structured data, but only registered allowlist fields may enter index mappings. Unknown attributes can use a flattened object, key-value column, or raw body. If the same field changes from a number to an object, the processor quarantines it as a parse failure or writes a versioned field. One bad deployment must not break an entire shared index.

The agent can generate eventId from the source instance, boot epoch, and local sequence. Delivery is at least once, so an agent retries when it does not receive an acknowledgment. Consumers and index writes use eventId idempotently. Deduplication windows are finite, and archives may retain duplicates. Queries and aggregates should understand this semantic instead of claiming expensive, brittle end-to-end exactly-once delivery.

Step 3: Plan partitioning and tenant isolation.

The durable stream scales through virtual partitions. A routing key can hash (tenantId, sourceInstance), preserving approximate order for one source while spreading a tenant across multiple partitions. Partitioning only by tenantId makes a large tenant a hotspot. Pure random partitioning loses source-local order. Large tenants can receive dedicated partition pools while small tenants share a pool, and the control plane can rebalance the assignment without changing the event format.

Each tenant has quotas for ingest bytes, event rate, burst tokens, local and central backlog, indexed field count, hot storage, query concurrency, scanned bytes, and export jobs. Rejections and degradation are recorded per tenant. During system overload, the priority order can be:

1. Preserve audit and security events.
2. Preserve errors and critical operational events.
3. Sample repetitive warnings, informational logs, and debug logs according to declared policy.
4. Reject new low-priority broad queries and exports.

A shared hot index works for small tenants, but every document, cache key, and query plan must include a trusted tenantId. Large or highly regulated tenants can use isolated indexes and encryption boundaries. The design should neither create many empty daily shards for every small tenant nor place every tenant in one unlimited shared index.

Step 4: Separate the raw archive from the search index.

Object storage holds normalized raw events in large compressed columnar files partitioned by region, tenant, date, hour, and optionally service. It is the low-cost long-term source for compliance exports, historical scans, and rebuilding a damaged hot index. Writers aggregate events into suitably sized objects rather than creating one object per log line.

The hot search index stores only recent, searchable data. Stable fields use inverted or column-oriented indexes, while full-text indexing of the body can vary by log class. High-cardinality request IDs, user IDs, or arbitrary tags remain stored fields when they are not frequently queried. traceId is also high cardinality, but it has a clear exact-lookup use case, so it can use a dedicated exact field with bounded retention rather than establishing a precedent for all dynamic fields.

A lifecycle controller moves data by class from hot to warm, cold, frozen, or archive tiers. Hot storage has more compute and replicas for write and low-latency search. Older data costs less and accepts slower access. Retention deletion must cover indexes, archives, caches, exports, and legal holds. Deleting one search index is not a complete deletion workflow.

Step 5: Build a controlled query path.

The query API derives tenantId from the session and requires startTime, endTime, and a service or another selective condition by default. The coordinator examines the time range and log class, then routes the request to hot indexes, warm storage, or an asynchronous archive scan. Example interfaces are:

POST /logs/search
  { startTime, endTime, services, severities, traceId, query, cursor, limit }
  -> { events[], nextCursor, partial, scannedBytes }

POST /logs/exports
  { startTime, endTime, filters }
  -> { jobId }

Results use the stable sort key (timestamp, eventId) and cursor pagination instead of deep offsets. Interactive queries have limits on scanned bytes, returned rows, execution time, and concurrency. When a limit is reached, the response returns an explicit partial marker rather than silently omitting data. Broad regular expressions, 90-day full-text searches, and large exports enter an asynchronous queue and can be canceled. Metadata for common filters may be cached, but query results cannot be reused across tenants without complete isolation.

The p95 target below 2 seconds applies to selective filtered queries over the last 15 minutes on a healthy cluster. An unbounded full-text scan over all retained data does not share that SLO. Defining query classes is more credible than promising that every query is fast.

Step 6: Calculate capacity.

Steady raw throughput is:

1,000,000 events/s × 600 bytes = 600 MB/s
600 MB/s × 86,400 s = 51.84 TB/day

Traffic rises from 1 million to 3 million events per second for 15 minutes. If downstream processing can sustain only the steady rate, it must absorb this additional backlog:

(3,000,000 - 1,000,000) × 600 bytes × 900 s = 1.08 TB

The burst produces 1.62 TB in total during those 15 minutes, but 0.54 TB is the baseline that steady processing handles at the same time. Real buffering also needs replication, batch overhead, recovery capacity, and safety headroom, so 1.08 TB is not a hardware-purchase figure.

If only 20% of logs enter a full-text or structured hot index and remain there for 7 days, the raw input before index overhead is still:

51.84 TB/day × 20% × 7 = 72.576 TB

Actual index size depends on fields, compression, shards, and replicas and must be measured by load testing. Ninety days of raw archives is 51.84 TB × 90 = 4.6656 PB, or about 4.67 PB before compression. This scale makes tiering, class-based retention, compression, and removal of low-value logs more important than simply enlarging the search cluster.

Derive partition count from sustained bytes per partition, events per partition, and required replay speed. Capacity tests must combine a lost availability zone, consumer catch-up, and a 3x burst while ensuring the oldest-message age eventually recovers. Counting events alone is insufficient because stack traces can change average size sharply.

Step 7: Handle backpressure, log storms, and downstream outages.

Every layer has finite capacity: SDK memory, agent disk, gateway connections, stream retention, processor concurrency, index write queues, and query threads. The system propagates pressure through retry-after, smaller batches, paused consumers, and priority queues. When the agent disk approaches its limit, it samples debug logs according to policy first and records local counters by dropped class. Audit events use a separate reserved pool or trigger an explicit business failure policy.

OpenTelemetry Collector troubleshooting guidance notes that an unavailable destination or an undersized Collector can both cause drops. Sending queues and retries cover temporary failures, but an oversized queue can also create memory pressure. “Enable retries” is therefore not a reliability plan by itself. Monitor queue utilization, oldest event age, rejections, permanent failures, and process memory, and alert before capacity is exhausted.

If the index cluster is unavailable, the durable stream continues accepting data, and raw-archive and index consumers advance independently. During recovery, index consumers catch up with tenant fairness and replay limits so they do not overload the newly restored cluster again. When indexing lags, the UI displays the search freshness watermark and missing interval. Users must not interpret “no search result” as “no log exists.”

Step 8: Close the security, privacy, and deletion loop.

The best sensitive-data control is to avoid generating the value. SDKs offer structured-field allowlists. Agents redact common token, password, cookie, and personal-data patterns. The gateway applies mandatory rules before the central durable stream, and later processors perform semantic schema-based redaction and route failed events to quarantine. A raw body cannot bypass security merely because it goes only to an archive.

Transport uses mTLS or short-lived workload identity, and data is encrypted at rest with environment or regulated-tenant key boundaries. RBAC restricts tenant, environment, service, fields, and time ranges. Queries, exports, retention changes, and legal holds produce immutable access-audit events. Highly sensitive fields can use field encryption or be removed completely; hiding them in the UI is not sufficient.

Deletion is a traceable workflow across hot indexes, warm tiers, object partitions, caches, and export copies. When audit retention conflicts with a privacy-deletion request, product and legal policy must define precedence, exceptions, and legal holds. The system executes that explicit policy; it cannot use “logs are immutable” to avoid a deletion obligation.

Step 9: Prove the system with metrics and fault injection.

Core metrics include p50/p95/p99 from creation to gateway, gateway to durable stream, stream to archive, and stream to search visibility; queue utilization, oldest-event age, retries, and drops at every stage; tenant ingest, throttling, field cardinality, and cost; index rejections, mapping growth, scanned bytes, query timeouts, cancellations, and cache hits; redaction failures, authorization denials, and archive restoration success.

An end-to-end canary writes a structured event with a unique ID from every region each minute and verifies durable acceptance, search visibility, object archive presence, and eventual retention deletion. Count reconciliation compares agent sent, gateway accepted, stream committed, archive written, and index success totals. Explained duplicates are allowed. Unattributed differences are not.

Fault tests include a 3x log storm for 15 minutes, a 30-minute search outage, object-storage throttling, loss of one availability zone, a hot tenant, a high-cardinality field attack, an invalid schema, source clock skew, sensitive-data injection, an exhausted agent disk, duplicate batches, stream replay, and rebuilding a hot index from archives. The critical assertions are that audit events remain recoverable according to policy, every ordinary-log drop has evidence for tenant, class, time, and count, and users and alerts can see degraded search freshness.

High-Quality Sample Answer

“I would first separate durable acceptance from search visibility. Applications write to stdout or an asynchronous SDK, and a node agent handles batching, redaction, and finite disk buffering, so a logging outage does not synchronously block ordinary business requests. The regional gateway derives the tenant from workload identity, enforces quotas and format rules, then commits the batch to a durable stream replicated across availability zones. That is the central acceptance point. Search indexes and object archives consume asynchronously.

Events have stable top-level fields for event time, observed time, tenant, service, environment, instance, severity, body, traceId, spanId, schema version, and sensitivity. Arbitrary attributes are stored by default. Only stable, well-typed fields with real query value enter the index, preventing high-cardinality attributes from causing mapping and index growth. Delivery is at least once. The agent retries with a stable eventId, and index writes are idempotent, although archives may retain identifiable duplicates.

Object storage keeps replayable long-term raw records, while the hot search tier keeps only recent, searchable data. Queries require a trusted tenant and time range, route across hot, warm, and archive tiers, and limit scanned bytes, concurrency, and result size. Broad historical full-text searches and exports become asynchronous jobs. Small tenants share indexes and large tenants may be isolated, but ingest, backlog, field, and query quotas always apply by tenant.

At steady state, raw ingest is 600 MB/s or 51.84 TB/day. A 15-minute 3x burst adds about 1.08 TB of backlog if downstream processing remains at steady capacity. Indexing only 20% for 7 days still produces about 72.6 TB before index overhead, while 90 days of raw archives is about 4.67 PB before compression. Selective indexing, class-based retention, and tiering are therefore mandatory.

During failure, the durable stream absorbs short backlog and archive and index consumers recover independently. Every buffer is finite. The system preserves audit and error events first, samples debug events first, and exposes drops, oldest-message age, and the search freshness watermark to monitoring and the UI. Finally, I would run a unique canary through producer, stream, index, and archive and inject 3x traffic, downstream outages, high-cardinality fields, and disk exhaustion to prove there is no silent loss and that indexes can be rebuilt from archives.”

Common Mistakes

• Synchronously calling a remote logging API from the application → a logging outage harms business requests

→ write to a local asynchronous agent with finite buffering and explicit degradation.

• Acknowledging only after a search-cluster write → index maintenance backpressures every producer

→ acknowledge after committing to the replicated durable stream.

• Dynamically indexing every JSON field → high cardinality creates mapping explosion and uncontrolled cost

→ index only stable allowlisted fields.

• Using one stream partition per tenant ID → a large tenant becomes a single-partition hotspot

→ use virtual partitions from tenant and source, with dedicated pools when justified.

• Claiming global order after random partitioning → clocks and parallel sources cannot provide it

→ preserve source-local order and keep an observed timestamp.

• Using exactly once to hide duplicates → lost responses and replay still happen

→ use at-least-once delivery, stable eventId, and idempotent writes.

• Assuming an unlimited queue prevents loss → disk, memory, and retention eventually fill

→ set capacity limits, monitor oldest age, prioritize classes, and retain drop evidence.

• Sampling all logs uniformly during a storm → audit evidence is discarded too

→ define reliability and degradation separately by log class.

• Promising every query finishes in 2 seconds → a 90-day full-text scan harms the interactive cluster

→ separate selective hot queries from asynchronous historical scans.

• Treating an empty UI result as proof that no log exists → index lag hides incident evidence

→ show watermarks, partial results, and missing windows.

• Deleting only the hot index → archives, caches, and exports still contain sensitive values

→ use an auditable deletion workflow across all copies.

• Monitoring only whether the Collector process is alive → a healthy process may still drop data

→ reconcile stage counts and monitor queues, rejects, and permanent failures.

Follow-Up Questions and Responses

Follow-up 1: Why not let agents write directly to the search cluster?

That is reasonable for a small internal tool because it has fewer components and direct search freshness. At this scale, search-shard changes, mapping failures, and write rejection would propagate to 100,000 sources. A durable stream creates an acknowledgment boundary, absorbs bursts, lets consumers recover independently, and supports replay. It costs extra operations, duplicates, and asynchronous delay. If measured scale is small, choose the direct path instead of adding a broker merely for the diagram.

Follow-up 2: What if one field suddenly has millions of distinct values?

The schema registry records indexable fields, types, owner teams, and cardinality budgets. Processors use approximate distinct counts to watch cardinality. When a field crosses its limit, they stop creating new index structures, retain the value as an unindexed attribute, and notify the tenant. If mapping explosion already occurred, first block new fields and repair upstream formatting, then rebuild healthy fields into a new index. Splitting the same unbounded field set across more indexes does not solve its growth.

Follow-up 3: What happens when both the durable stream and local spool are full?

A finite system cannot guarantee an infinite burst. The agent reserves capacity by log class: audit events use the strongest channel, errors outrank informational events, and debug logs are sampled or dropped first. Every drop is counted locally and reported after recovery. If a business operation requires a particular audit event to be durable, the operation can explicitly fail or use a local transactional outbox. An ordinary debug call must not block the entire application forever.

Follow-up 4: How do you rebuild the search index from object storage?

Archive objects include schema version, time partition, tenant, checksum, and a manifest. A rebuild job selects a tenant and window, reads and verifies manifests, transforms records with the current mapping, and writes to a new index idempotently by eventId. It compares counts, time boundaries, and canaries before atomically switching an alias or route. Rebuild traffic has a separate quota so it does not starve real-time indexing.

Follow-up 5: How do you stop one tenant's query from harming everyone?

The scheduler maintains tenant token buckets for concurrency, CPU time, scanned bytes, and returned rows and uses weighted fair queuing. Interactive queries outrank exports, and expensive work can be canceled or moved asynchronous. Shared cache keys include tenant and permission summaries. Large or highly regulated tenants may receive isolated index pools, but isolate only after measuring the bottleneck so small tenants do not create excessive tiny shards.

Follow-up 6: Why do audit logs need a separate policy?

Audit logs exist to prove who did what and when. Sampling, mutable content, and ordinary administrator deletion would defeat that purpose. They need stricter schemas, authenticated provenance, integrity checks, immutable archives, restricted queries, and access auditing. They still follow privacy deletion and legal-hold policy, so “immutable” means ordinary paths cannot alter them, not that an authorized compliance workflow can never act.

Follow-up 7: How would you extend the design across regions?

Each region first writes local events into a local durable stream and archive so business requests do not wait across continents. Events carry region and global tenant IDs, and the control plane distributes schemas, quotas, and retention rules. A global query coordinator fans out by time and region and marks partial results when a region is unavailable. If regulation requires residency, raw logs remain local and only approved indexes or aggregates cross regions. Disaster recovery restores from local or compliant replicated object archives.

Follow-up 8: How do you prove the system is not silently losing logs?

Each batch records source count and checksum. Every stage emits accepted, duplicate, rejected, permanently failed, and success counts. A unique canary continuously crosses agent, gateway, durable stream, index, and archive. Reconciliation allows duplicates explained by eventId and drops explained by an explicit quota policy. Any unattributed difference is an incident. Then stop indexing for 30 minutes and recover it, verifying that message age falls, canaries fill the gap, and the UI freshness watermark recovers instead of merely checking that the process restarted.