Prompt and scope
An Iceberg table is organized by date and tenant. Queries frequently filter a low-cardinality status column, yet planning still inspects many data files. The team wants to store indexes or statistics that do not fit directly in manifests in Puffin files and let the planner use them selectively. Explain snapshot binding, stale-statistic protection, missing statistics, concurrent commits, and a proof that acceleration does not sacrifice correctness.
Capacity, selectivity, and file counts are interview assumptions, not universal benchmarks. This question fits data engineering, lakehouse storage, query engines, and data platform roles. Its core skill is table-format metadata and planning, so it belongs to data.
What interviewers assess
First, do you understand Puffin’s boundary? Puffin is a file format for indexes and statistics about an Iceberg table; blob metadata describes its contents. It does not replace table snapshots or manifests.
Second, can you separate optional optimization from correctness metadata? A reader may ignore statistics and still read correctly. Pruning may remove only candidates proven not to match.
Third, can you bind statistics to snapshots and scope? Every blob needs the applicable snapshot, partition, or data-file range; an update must not blindly reuse an old blob.
Fourth, can you quantify maintenance? Extra files add object-storage I/O, caching, generation jobs, and expiration work; low-selectivity queries may pay planning overhead without benefit.
Fifth, can you design fallback? Missing, unreadable, incompatible, or invalid blobs must send the planner back to manifests and data-file filters, never to an empty result.
Questions to clarify first
- Which Iceberg version and query engine support the target Puffin blob type?
- Are statistics generated per partition, data file, column, or value range?
- How frequently are snapshots committed, rewritten, or time-traveled?
- What generation staleness is acceptable, and is eventual optimization acceptable?
- How are blob compression, checksums, encryption, and object permissions managed?
- How does the planner prove that a candidate is safe to exclude?
30-second answer framework
“I would verify supported Puffin blob types and their Iceberg snapshot association. Statistics are an optional optimization, not a correctness prerequisite; each blob records its snapshot, partition or file scope, and the planner falls back to manifests when it is missing, stale, or unreadable. A generation job uses a fixed snapshot and atomically commits the reference, rather than reusing old statistics for a new snapshot. In a canary I would compare candidate files, planning time, additional Puffin I/O, end-to-end latency, and result validation.”
Step-by-step answer
Step 1: Establish Puffin and snapshot relationships
Puffin stores indexes and statistics that do not fit directly in Iceberg manifests. The StatisticsFile API exposes path, file size, footer size, blob metadata, and the associated snapshot ID. Put these fields in the metadata contract instead of keeping only an object path in an external setting.
Step 2: Define blob contents and scope
Each blob should declare type, version, encoding, covered partitions or data files, generation snapshot, columns, and comparison rules. A low-cardinality status column can use partition counts or a value-range summary; an unsafe high-cardinality summary should remain observational rather than prune files.
statistics = build_from_snapshot(snapshot_id, data_files)
blob = {
type, version, snapshot_id, partition_scope,
covered_files, column_rules, checksum, payload
}
commit_statistics_file(blob)Step 3: Design safe pruning
The planner excludes a candidate only when statistics prove it cannot match. Missing statistics, overlapping value ranges, uncertain null semantics, unknown blob versions, or failed checksums all trigger manifest and data-file filtering. “No statistics” must never mean “no data.”
Step 4: Handle concurrent commits and staleness
The generation job reads a fixed snapshot and, at commit time, verifies that the table can still reference that snapshot or a compatible descendant. New writes, deletes, and partition evolution change the file set; an old blob is valid only for its declared scope. Do not update a Puffin object in place.
Step 5: Plan reads and cache safely
Puffin itself has footer and blob I/O. The planner should use lightweight metadata to decide whether a blob is worth reading, then choose blobs by query columns and partition. Cache keys include table identity, snapshot, blob type, and version. Read errors invalidate the cache and fall back; an error must not be cached as an empty statistic.
Step 6: Budget cost, expiration, and permissions
Track blob bytes, footer ratio, generation CPU, object requests, and pruning hit rate. After snapshot expiration or file rewrites, clean statistics by reference relationships, not only object modification time. Grant least-privilege access to generators and readers, validate checksums, and encrypt sensitive summaries where needed.
Step 7: Run a canary acceptance test
On the same snapshot, compare planning with statistics enabled and disabled: candidate files, scanned bytes, planning p50/p95, extra Puffin I/O, end-to-end latency, and result hashes. Delete a blob, create a version mismatch, and race a concurrent commit to prove fallback returns complete results. Keep a low-selectivity query as a negative control.
Model answer
“I would treat Puffin as an optional planning acceleration layer. A generator reads a fixed snapshot, and blob metadata records snapshot, partition or file scope, column rules, version, and checksum. A new snapshot must not reuse a blob outside that declared scope.
The planner prunes only when statistics safely prove a mismatch. Missing, stale, unreadable, ambiguous-null, or unknown-version blobs fall back to manifests and data-file filters; missing statistics never means an empty result. Cache keys include table, snapshot, type, and version.
In a canary I would compare candidate files, planning time, Puffin I/O, end-to-end p95, and result hashes, while injecting missing blobs and concurrent commits. I would expand only when gains are stable, fallback is correct, and maintenance cost is acceptable.”
Common mistakes
- Treating Puffin as the snapshot source of truth → stale statistics change results → keep it optional with manifest fallback.
- Recording only an object path → scope and snapshot cannot be checked → store complete blob metadata.
- Pruning with stale blobs → new writes or deletes are missed → bind snapshots and covered files.
- Returning an empty result for missing statistics → unknown becomes no match → fall back to normal filtering.
- Overwriting Puffin in place → concurrent readers see mixed versions → write a new file and atomically reference it.
- Caching without snapshot → different snapshots reuse wrong statistics → include version and snapshot in the key.
- Measuring only planning time → scan results can change → compare candidate sets and result hashes.
- Deleting by modification time → retained snapshots may still reference the file → expire by references.
Follow-up questions
Follow-up 1: Why may readers ignore statistics?
Puffin statistics are informational. The table can still be read correctly from manifests and data files, so unsupported or temporarily unreadable blobs should cause transparent fallback.
Follow-up 2: How do you expire statistics?
Confirm that no retained snapshot, branch, or tag needs the blob’s covered range, then clean it according to table-format references. Object modification time alone is insufficient.
Follow-up 3: What if a value range is incomplete?
Treat the unknown portion as potentially matching and widen candidates or fall back completely. Optimization may have false positives, never false negatives.
Follow-up 4: When should statistics be generated during concurrent writes?
Generate from a committed snapshot and bind after metadata commit; temporary files from an in-flight write must not be visible to planning. Recheck compatibility at commit time.
Follow-up 5: When is Puffin not worth it?
Disable or narrow it when selectivity is low, blob I/O exceeds manifest savings, generation is too stale, or maintenance costs exceed gains. Keep a per-table switch.
Follow-up 6: How do you prove the optimization preserves results?
Run equivalent queries on the same snapshot with statistics enabled and disabled, compare complete results, aggregates, candidate files, and boundary samples, then inject missing, corrupt, and incompatible blobs to test fallback.