Prompt and context
Your lakehouse uses both Spark and Trino, and the team wants shared logical views with safe rollback. Based on the Apache Iceberg View Spec, design metadata publication, cross-engine representations, concurrent updates, rollback, and compatibility validation.
The Iceberg View Spec removes view definitions from engine-specific metastore formats. A view contains no data; its definition runs when referenced. View metadata records the schema, versions, SQL representations, and version log. The interview tests a cross-engine contract and publication consistency, not just CREATE VIEW syntax.
What the interviewer evaluates
The interviewer looks for the boundary between a view and a table, atomic metadata replacement, immutable versions, and optimistic commits. Strong answers explain view-uuid, format-version, current-version-id, versions, and version-log; handle Spark/Trino dialect differences, concurrent edits, rollback, schema evolution, cache refresh, and execution permissions.
Clarifying questions
Sharing and execution targets
Ask which engines must read or write the view, whether edits are bidirectional, whether SQL dialects can be translated, and how quickly readers must observe a new version.
Version and rollback policy
Clarify how much history to retain, whether rollback only changes the current pointer, whether approval and audit are required, and whether old views remain executable after a base-table schema change.
Consistency and security boundaries
Confirm that the metadata store and catalog support atomic pointer swaps, conflict detection, permission isolation, and regional visibility. Shared view metadata does not automatically share access to the underlying data.
30-second answer
“I would write every view change to a new self-contained metadata file and atomically replace the catalog's metadata location. The file keeps a stable view-uuid, format version, schema, immutable versions, and a version-log describing current-pointer changes. Each version carries SQL representations tied to engine dialects; Spark and Trino publish only after semantic equivalence checks. Writers use optimistic concurrency and recompute from a new base after a conflict. Rollback points current-version-id at an existing version, with permissions, cache refresh, and base-schema compatibility audited.”
Step-by-step solution
Step 1: Define the view metadata model
Create a stable view-uuid, set format-version to the required value 1, and record the base location, schemas, versions, current-version-id, and version-log. Use properties for comments or maintenance settings, not arbitrary business state.
Step 2: Publish by replacing a complete file
Every update creates a complete metadata file. Commit by atomically swapping the catalog pointer from the old location to the new one. Readers continue using the version they loaded until they refresh the location, so no query sees a half-written definition.
Step 3: Make versions immutable and rollback-safe
A version contains a version ID, schema ID, creation timestamp, summary, representations, and default namespace. Once created it is immutable; any SQL or representation change creates a new version. The version log records changes to current-version-id, so rollback points to an old version instead of rewriting history.
Step 4: Handle SQL representations across engines
A version may contain multiple SQL representations, but only one per dialect, and all must express the same underlying definition. The publisher should parse, compare column types, and compare representative result sets for Spark, Trino, and other engines. An engine without an equivalent representation must reject execution or follow an explicit fallback.
Step 5: Handle concurrency and caches
Writers build from the metadata location they read; an atomic-swap failure means the base changed. Clients refresh on a catalog pointer or metadata-location change instead of relying only on a fixed TTL. Bound conflict retries so automated publishers do not continuously overwrite one another.
Step 6: Connect schema evolution and permissions
The view schema is part of its version. When a base column is deleted, renamed, or retyped, compile and run representative queries in each target engine. Audit permissions for the view definition, base table, and catalog separately; read access to a view must not grant write access to raw data.
Step 7: Validate, roll back, and observe
Before publishing, run cross-engine semantic comparisons, result-schema checks, permission tests, and snapshot-level rollback drills. Record the author, engine version, dialect, commit conflicts, refresh delay, execution failures, and rollback reasons. Bound retained history with maintenance settings such as version.history.num-entries and monitor metadata growth.
Model answer
I would treat the view as a shared, versioned logical object. Creation generates a stable view-uuid and format version 1. Every change creates a complete metadata file containing schemas, versions, representations, and the version log, then atomically swaps the catalog's metadata location. Versions are immutable; rollback only points current-version-id at an existing version. Spark and Trino publish dialect-specific SQL representations after parser, column-type, and result-set checks prove semantic equivalence. Writers use optimistic concurrency and retry from a fresh file after a conflict. Release validation also covers base-schema evolution, cache refresh, permission isolation, auditability, and cross-engine execution after rollback.
Common mistakes
- Mistake: Storing the definition only in one engine's metastore. → Why it fails: Other engines cannot reliably read or modify it. → Fix: Use shared Iceberg view metadata and explicit dialect representations.
- Mistake: Editing the current metadata file in place. → Why it fails: Readers may see partial state and rollback loses a safe boundary. → Fix: Write a complete new file and atomically swap the catalog pointer.
- Mistake: Treating the version log as creation timestamps. → Why it fails: It records current-version-id changes and may include rollbacks. → Fix: Separate version creation time from pointer history.
- Mistake: Freely rewriting representations inside one version. → Why it fails: Representations must express the same definition and versions are immutable. → Fix: Create a new version and run dialect semantic tests.
Follow-ups and responses
Why should view metadata be self-contained?
A reader can parse the schema, versions, and representations from one location and roll back within retained history without depending on an untraceable side table.
What if two engines publish at once?
Writers include the metadata location they read. Atomic swap rejects one commit when the base changed; that writer rereads, merges, and reruns cross-engine validation instead of silently overwriting the other version.
Does rollback destroy history?
No. Rollback adds a version-log pointer change that sets current-version-id to an earlier version. The old versions and prior log entries remain auditable.
What if a base table drops a column?
Treat it as a compatibility gate before publishing a new view version. Compile and run representative queries in every supported dialect. Block publication or mark an engine unsupported instead of discovering the breakage in production.