Prompt and context
A lakehouse job reads Parquet from object storage. A small number of files are corrupted after cross-region replication, but the current pipeline only retries the whole file. Design a page-level CRC plan: define the checked bytes, explain compression boundaries, preserve reader compatibility, isolate bad pages, and choose acceptance metrics. This is a data question about columnar-file integrity.
What the interviewer evaluates
- Distinguish page CRC from file-level checks and know the field is optional.
- State the checksum scope and compression boundary precisely.
- Design a rollout when older readers ignore CRC fields.
- Prevent silent partial reads and separate isolation, replication recovery, and rewrite.
- Measure correctness, localization time, and CPU/I/O overhead.
Clarifying questions to ask
- Do the target writer and reader versions write and verify page CRCs?
- Is corruption introduced by object storage, replication, or a local cache?
- Are compression, encryption, or page indexes enabled, and can the reader locate a column chunk?
- Must every row be recovered, or can the partition be rebuilt from upstream data?
- Could retries keep caching the same bad object version?
A 30-second answer
“I would first verify page-CRC support against the Parquet format and the exact reader versions, recording file version, row group, column chunk, page type, and object version. A verification failure should quarantine the object rather than skip rows; recovery can try a matching replica or rewrite from upstream. Older readers may ignore CRCs, so they remain readable but cannot claim integrity verification. I would inject corruption into data pages, dictionary pages, headers, and replication tails, then compare detection rate, localization time, retry cost, and result equality.”
Deep-dive answer
Step 1: Establish the format boundary
Parquet page headers contain an optional 32-bit CRC field for page types such as data and dictionary pages. Confirm that the writer populates it and that the reader verifies it; a writer-only configuration change does not make downstream reads safe.
write page bytes -> compute CRC32 -> persist header and payload
read page -> read header -> verify payload CRC -> decodeInclude object version, row group, column chunk, and page ordinal in diagnostics so a retry can prove it read the same bytes.
Step 2: Fix compression and checksum ordering
The format specification and implementation define the exact bytes covered by CRC. Keep a writer/reader version matrix and never invent a checksum over decoded logical values. A decompression failure and a CRC mismatch are different failure classes and need separate metrics.
Step 3: Isolate corruption safely
On failure, fail the file read and put the object into a quarantine queue with URI, version, page location, and error type. Do not silently omit the page. Read a matching replica when available; if all copies fail, rewrite the partition or replay upstream data instead of retrying the same bytes forever.
Step 4: Handle old readers
An old reader may ignore the CRC field and return rows, so successful decoding does not prove integrity. During rollout, maintain a reader capability matrix. Strict readers validate critical workloads; old readers are labeled as lacking integrity verification, and a sampled sidecar check can expose differences before migration.
Step 5: Relate CRC to encryption and page indexes
Encryption authentication, compression, and CRC ordering must follow the target implementation. A CRC is not an authentication tag, and a page index does not validate payload bytes. Complete the format-required integrity check before decoding or pruning, while retaining column-chunk coordinates for diagnosis.
Step 6: Build corruption-injection tests
Flip one byte in a data page, then separately damage a dictionary page, page header, file tail, and replicated object version. Cover compressed pages, null-heavy pages, empty pages, and large pages. Verify the strict reader’s location and record the behavior of old readers.
Step 7: Define repeatable acceptance
With the same object version, concurrency, and cold cache, compare CRC on and off for CPU, bytes read, p95, detection rate, and localization time. Results for intact inputs must match exactly; injected corruption must fail and enter quarantine. Alert on checksum failures, repeated retries, and recovery success rate.
Model answer
“I would begin with the Parquet format and target-reader implementation to confirm that page CRC is optional, which page types carry it, and the exact byte range covered. Reads verify CRC before decompression and decoding; decompression and checksum errors are separate metrics. A bad page fails and quarantines the file, then recovery uses the same object version from another replica or rewrites from upstream. It must not skip the page or endlessly retry a bad object.
Older readers may ignore CRC, so I would publish a capability matrix and migrate critical jobs to strict readers. Byte-flip, page-header, dictionary-page, and replication-truncation tests measure detection, localization, CPU, bytes read, recovery rate, and result hashes before expanding the rollout.”
Common mistakes
- Treating CRC as authentication → CRC detects random corruption, not tampering → add authenticated encryption or signatures when needed.
- Changing only the writer → downstream readers may ignore the field → test the version matrix.
- Skipping a bad page → creates silent missing rows → fail and quarantine.
- Retrying one object forever → amplifies cost → pin the version and cap retries.
- Testing only whole-file damage → misses page and header boundaries → inject layered corruption.
- Mixing decode and checksum errors → obscures recovery → separate metrics and runbooks.
Follow-up questions and responses
Follow-up 1: Does CRC prevent malicious tampering?
No. CRC detects accidental corruption. Use authenticated encryption, signatures, or trusted-storage verification for tamper resistance.
Follow-up 2: Do old readers break compatibility?
They generally remain able to decode the format, but they cannot claim that CRC was verified. Critical jobs should use strict readers.
Follow-up 3: Can you reread only the bad page?
Try a range read or a matching replica, but validate the complete result. If no verified copy exists, rewrite or replay upstream data.
Follow-up 4: Why record object versions?
An object may be overwritten during retry. Version identifiers tie the error and recovery action to one byte sequence.
Follow-up 5: How do you control overhead?
Benchmark CPU, throughput, and p95 with realistic page sizes and concurrency, then canary high-risk partitions before broad rollout.