Prompt and scope
The table is append-heavy and partitioned by day, but account_id is not clustered. Equality queries often match far below 1% of rows. Explain how Bloom filters can skip row groups or pages without changing results, how to verify reader support, and when their metadata and write cost outweigh the savings. The capacity and selectivity are interview assumptions, not universal benchmarks. The core skill is probabilistic file-layout optimization, so this belongs to data.
What interviewers assess
Strong answers distinguish a probabilistic membership test from an exact index: a negative result proves absence, while a positive result only keeps the unit as a candidate. They identify the filter granularity and on-disk location used by the target implementation, account for nulls and encoding, and preserve a normal-read fallback. They also propose a control experiment with identical snapshots, cache state, and reader versions.
Questions to clarify first
- Which engine writes and reads Parquet Bloom filters, and which versions are deployed?
- Is the predicate equality-only, or are
INlists and normalized keys required? - Are filters attached per column chunk, row group, or page in this implementation?
- What is the distinct-value distribution and expected false-positive rate?
- Can old readers ignore the metadata while returning identical results?
- Are files immutable, or will compaction and rewrites add ongoing CPU cost?
30-second answer framework
“I would first confirm end-to-end writer and reader support and inspect sample footers for filter offsets and sizes. I would enable filters only for high-selectivity equality columns, choose a target false-positive rate from measured distributions, and keep an index-disabled control. During a canary, I would compare row-group/page reads, bytes, CPU, latency, filter bytes, and exact result equality. Positive tests still read the candidate; only a proven negative may skip it. If support is missing or scans are broad, the fallback remains ordinary Parquet filtering.”
Step-by-step answer
Step 1: Establish capability and granularity
Read the Apache Parquet Bloom-filter specification and implementation matrix for the exact libraries in use. Verify that writers persist filters and readers consult them for the predicate type. Record the filter’s offset/length and the data unit it protects; do not assume every engine uses the same granularity.
Step 2: Choose columns and size filters
Estimate distinct values per protected unit and query selectivity. Size filters from an explicit false-positive target, then benchmark memory, footer growth, and write CPU. A filter that is too small produces many positives; an oversized filter can dominate metadata I/O without improving broad scans.
Step 3: Preserve probabilistic semantics
For a queried value, a negative membership result can safely skip the protected unit. A positive result means “may exist,” so the reader must apply the exact predicate after decoding. Never use a Bloom filter to return an empty result directly, and test null handling and key normalization separately.
Step 4: Roll out with controls
Write filters for one partition or file cohort while retaining an equivalent cohort without filters. Run the same snapshot, workload, concurrency, and reader build against both cohorts. Include point lookups, long IN lists, missing keys, hot keys, and low-selectivity queries.
Step 5: Define acceptance and rollback
Track protected units tested, negatives, false positives, units read, bytes read, filter bytes, CPU, p50/p95 latency, and write throughput. Compare complete result sets, counts, and aggregates with filters enabled and disabled. Roll back writes or disable consumption if results differ, metadata overhead rises, or the skip ratio is immaterial.
Model answer
“Bloom filters are useful when equality predicates are selective and values are scattered. I would verify the deployed Parquet writer and reader support, inspect filter offsets and granularity, and size filters against a measured false-positive target. On a canary partition, I would compare an identical no-filter cohort under cold and warm cache controls. A negative membership test may skip the unit; a positive test must still execute the exact predicate. I would require identical results plus lower units and bytes read, while checking footer growth, CPU, write throughput, and p95 latency. Unsupported readers continue with ordinary reads, so deployment is capability-aware and reversible.”
Common mistakes
- Treating “may contain” as exact → matching rows can be discarded → decode and evaluate the predicate after positives.
- Assuming all readers support filters → metadata is ignored or behavior differs → test a versioned writer/reader matrix.
- Sizing from table-wide cardinality → local units have different distributions → measure distinct values per protected unit.
- Testing only point lookups → broad scans may pay overhead → include low-selectivity negative controls.
- Comparing different snapshots → result and cache effects are confounded → hold snapshot, resources, and workload constant.
- Skipping null/normalization tests → semantic edge cases are missed → test nulls, casing, encoding, and
INlists.
Follow-up questions
Follow-up 1: Can a false positive change correctness?
No. It only causes extra reads. Correctness fails only if an implementation treats a positive as proof or a negative as valid despite malformed metadata.
Follow-up 2: When is a Bloom filter not worth writing?
Full scans, low-selectivity predicates, tiny files, and readers that ignore filters usually gain little. Compare filter bytes and write CPU with measured skip savings.
Follow-up 3: How do you validate a missing key?
Use a key absent from the snapshot and verify many protected units return negative, then confirm the complete query result is empty with filters both enabled and disabled.
Follow-up 4: What if a reader lacks support?
It should ignore the optional metadata and perform ordinary row-group/page filtering. Keep compatibility tests and avoid making filter presence a correctness prerequisite.