Prompt and scope
Your lake uses Iceberg tables and is moving from 1.10.1 to 1.10.2. Tables contain equality deletes, v2 deletes, and concurrent writers, while the release includes security fixes. Design upgrade, validation, rollback, and cleanup.
Apache Iceberg records 1.10.2 as released on May 18, 2026, with fixes for equality-delete schema ordering, loading snapshots after commit, unsafe file cleanup, concurrent format upgrades, and dependency vulnerabilities. The interview tests turning release notes into data-correctness evidence.
What the interviewer evaluates
- Separating table spec, engine implementation, Catalog, FileIO, and runtime dependencies.
- Explaining equality deletes, position deletes, delete vectors, and snapshot visibility.
- Designing shadow validation, concurrent-write tests, cleanup protection, and a rollback window.
- Recognizing that a dependency upgrade passing is not proof of historical-query correctness.
- Defining auditable metrics, stop lines, and post-upgrade cleanup gates.
Clarifying questions
- Which engines, Catalog, object store, and Iceberg runtimes are deployed? Are writers multilingual?
- What format versions, delete-file types, partitioning, snapshot retention, and commit rates apply?
- Is this only a client dependency replacement, or do writers, readers, and Catalog services change too?
- Which streaming writers and downstream queries cannot pause?
Version and compatibility matrix
Lock the actual dependency tree and deployed versions. Build a matrix for readers, writers, Catalog, FileIO, object storage, and cleanup jobs. A 1.10.2 fix does not prove that an old engine understands new metadata; use official compatibility guidance plus replay tests. Roll out in read-only readers, offline writers, online writers, and cleanup stages.
In a canary environment, copy production snapshots, delete files, and concurrent commits. Record snapshot IDs, manifest counts, delete-file counts, and table format versions read by each component. Mark unknown compatibility as a blocker instead of treating “it reads” as evidence.
Delete semantics and correctness validation
Create a golden dataset with repeated equality keys, multiple row versions, position deletes, delete vectors, and concurrent updates. Compare pre- and post-upgrade readers on the same snapshot, time-travel queries, and incremental scans. Check row counts, primary-key sets, aggregates, and schema ordering; record rows filtered and unmatched deletes.
Do not validate only the current table: a bad delete can surface after compaction. Retain original manifests, delete files, and snapshot logs, and cross-check with independent SQL or a small exact implementation. On failure, preserve snapshots and do not immediately expire them.
Concurrent commits and snapshot protection
Run append, overwrite, row-level delete, and compaction tests during the upgrade. Inject commit conflicts, Catalog timeouts, and transient object-store 503s. Confirm failed transactions cannot clean files referenced by active snapshots and that successful commits expose one consistent snapshot. Set a dedicated stop line for v2 deletes concurrent with format upgrades.
Cleanup computes candidates from retention, active queries, branch or tag references, and commit time. Candidates enter a delayed queue and are deleted only after a second confirmation. During rollback, forbid irreversible cleanup so readable history remains available.
Deployment, rollback, and governance
Release in batches: read-only services, low-volume writers, then all jobs. Each batch records errors, snapshot commit latency, query-result differences, delete misses, object-store 404s, cleanup candidates, and resource cost. Rollback switches to a compatible client while retaining snapshots created by the new version for review; deleting metadata is not rollback.
Lock and scan transitive dependencies in build artifacts. If 1.10.2 removes or changes a test fixture or runtime artifact, validate packaging, classpaths, and licenses in the build matrix before production. Record manual exceptions with expiry dates.
Failure drills and release gates
Drill equality-delete schema-order changes, concurrent format upgrades, snapshot-load failures, cleanup receiving 503, old readers loading new snapshots, object-store lag, and duplicate commits during recovery. Gates include zero unexplained result differences, delete misses within budget, no active-snapshot files cleaned, readable history after rollback, and passing dependency scans.
If only ordering differs, determine whether ordering was unspecified. If primary-key sets or delete visibility differ, stop expansion immediately. After the retention and observation windows, restore cleanup gradually; disk pressure does not justify skipping evidence retention.
Follow-up questions and reference answers
Why is comparing only the latest row count insufficient?
It misses time travel, incremental reads, delete-file application, and historical cleanup. Compare multiple snapshots, primary-key sets, aggregates, and delete misses on the golden dataset.
How do you prove rollback is safe?
Retain before-and-after snapshots, stop irreversible cleanup, verify old readers can read the retention window, and rehearse recovery after commit conflicts and object-store failures.
How do dependency security fixes enter data validation?
Lock transitive dependencies and scan artifacts, then run classpath, license, and reader/writer matrix tests. A clean security scan does not prove delete semantics.