Prompt and scope
Your team stores table data in object storage while compute expands from Spark to Trino and an in-house service. The old Hive Metastore approach needs multiple client implementations, and concurrent metadata updates can overwrite one another. Design an Apache Iceberg REST Catalog, covering what it owns, snapshot commits, authentication and authorization, and recovery from failures.
The Apache Iceberg REST Catalog specification exposes catalog operations through a language-neutral HTTP API and uses change-based commits to help the server deconflict concurrent updates and retries. It manages namespaces, table metadata, and snapshot references; data files remain in the underlying object store. The interview is about the metadata control plane, not rebuilding a query engine or object storage.
What the interviewer is testing
A strong answer separates the data plane from the metadata control plane, then derives components from client compatibility, commit concurrency, authorization boundaries, and recovery. The interviewer expects a connection between configuration discovery, table loading, updates, optimistic concurrency, snapshot caching, and credential vending.
A weak answer merely adds “a REST service” without explaining how two writers avoid overwriting snapshots or how catalog-issued data credentials, cross-engine authentication, and old clients are handled.
Questions to clarify first
Access patterns and consistency goals
Ask about table size, namespace count, read/write mix, snapshot commit rate, and whether multi-table atomic commits are required. A low-frequency batch workload may need only a simple metadata database. Many engines committing concurrently require explicit conflict detection, retry budgets, and commit-latency targets.
Storage and catalog boundary
Confirm who owns object storage, FileIO, the catalog database, and compute engines. The REST Catalog returns metadata and configuration, but should not proxy large data files through the catalog service. Data credentials can be vended for a table or location with narrow permissions and a short lifetime.
Authentication and governance
Ask whether clients use OAuth2, cloud request signing, or service accounts, and whether tenant isolation, audit, or column and row policies are needed. Catalog authorization controls discovery, reads, and commits; object storage must enforce least privilege again, so the catalog is not the only security boundary.
30-second answer framework
“I would make the REST Catalog a stateless metadata control plane. A client first calls the configuration endpoint, then uses namespace and table APIs to load metadata. The catalog database stores the current metadata location, snapshot references, and commit version; a writer submits a change against an expected version, and the server detects conflicts and returns a retryable result. Authentication uses OAuth2 or cloud signing, while catalog permissions and object-store credentials are separate. Metadata can be cached briefly but must be checked with a version or ETag. When the catalog is unavailable, reads may use a verified old snapshot, but writes must not bypass the commit protocol and edit the root metadata directly.”
Step-by-step solution
Step 1: Define the control-plane model
The catalog needs at least a namespace, table identifier, current metadata location, snapshot references, a version, and audit fields. Table metadata files remain in object storage; the catalog records their locations and commit versions. Clients can load snapshots on demand, and the catalog does not become a large-file transfer path.
Step 2: Design the smallest useful API
The configuration endpoint returns server defaults, overrides, and supported endpoints. Namespace APIs create, list, and manage properties. Table APIs create, load, update, commit, delete, and rename tables. A load response may include table and data-access configuration, after which the client talks directly to object storage. A commit response returns the new version so clients can refresh their cache.
Step 3: Protect commits with optimistic concurrency
A writer reads version V, writes new metadata, and submits “I am based on V and want to switch to location M.” The server atomically updates the catalog only if the current version is still V. If another writer committed first, it returns a conflict; the client reloads, merges its change, and retries. Retries need a cap and jitter so that several engines do not turn one conflict into a commit storm.
Step 4: Handle cache and read consistency
Clients may cache table configuration and snapshot references, but the key must include the complete table identifier and server identity. Prefer ETag, version, or snapshot-reference validation over a time-to-live alone. Reads can use a committed snapshot within an explicit staleness budget; requests requiring the latest branch, a permission change, or read-after-write must revalidate the catalog version.
Step 5: Separate authentication, authorization, and credential vending
The catalog API authenticates with OAuth2, cloud signing, or an enterprise service account. Authorization distinguishes namespace, table, and operation. If the catalog returns temporary object-store credentials, they must cover only the required location and actions, have a short TTL, and be linked to audit records with a revocation path. Clients must not write those credentials to logs or shared configuration as permanent keys.
Step 6: Design failure and recovery paths
When the catalog database is unavailable, verified cached snapshots may serve reads with a visible staleness timestamp. Create, commit, and delete operations must fail and retry later; they must not edit the object store’s root metadata directly. If object storage is temporarily unavailable, the catalog must not report success with a new catalog version pointing at a missing file. Recovery validates metadata locations, snapshot references, and file manifests before writes reopen.
Step 7: Add observability and compatibility evolution
Record request ID, client engine, table identifier, expected and actual versions, conflict count, retry count, and credential scope, but never tokens or keys. Track commit latency, conflict rate, cache hit rate, and stale-read ratio by API, namespace, and engine. New endpoints or fields should use capability discovery and backward-compatible defaults; an unknown field must not change existing commit semantics for old clients.
High-quality sample answer
I would define the Iceberg REST Catalog as a metadata control plane. Object storage holds data files and table metadata files. The catalog database stores the table identifier, current metadata location, snapshot references, version, and authorization audit data. Spark, Trino, and other language clients then implement one HTTP protocol.
The client reads configuration and loads the table. A writer reads version V, writes new metadata, and submits an expected version V. The server checks that version inside a transaction; if it is still V, it atomically switches to the new location. Otherwise it returns a conflict. The client reloads and merges, then retries with exponential backoff and a bounded budget. This is safer than last-writer-wins, which can silently lose another writer’s schema or snapshot change.
Authentication can use OAuth2 or cloud signing. Catalog authorization controls discovery, reads, and commits. If the catalog vends object-store credentials, I scope them to the location and a short TTL. Metadata caches use ETags or versions, so permission changes and read-after-write do not rely on an unchecked stale entry. During a catalog outage I allow reads only with an explicit staleness marker and fail writes. After recovery I verify metadata locations, snapshot references, and file existence. I finish with compatibility tests for concurrent commits, duplicate retries, catalog failure, expired credentials, stale caches, and old-client requests.
Common mistakes
- Mistake: Proxying every data file through the catalog. → Why it fails: The metadata control plane becomes a high-bandwidth bottleneck and couples authorization to data transfer. → Fix: Return metadata and scoped access configuration, then let clients read object storage directly.
- Mistake: Using last-writer-wins for concurrent commits. → Why it fails: A later writer can silently overwrite another writer’s schema or snapshot. → Fix: Submit an expected version, detect conflicts with an atomic conditional update, and retry within a budget.
- Mistake: Relying only on a fixed TTL for freshness. → Why it fails: Read-after-write and permission changes can observe a wrong view before the TTL expires. → Fix: Validate with ETags, versions, or snapshot references and force refresh by request risk.
- Mistake: Editing root metadata directly when the catalog is down. → Why it fails: It bypasses the commit protocol and splits the catalog index from file state. → Fix: Fail and retry writes, then validate locations, snapshots, and manifests during recovery.
Follow-up questions and responses
Follow-up 1: Two writers are based on version V. How do you merge schema changes?
The server rejects the second commit rather than guessing. The client reloads current metadata and checks whether its schema, partition, or property changes conflict with the new state. It creates and submits new metadata only when the merge is safe. A change that cannot be merged automatically becomes an explicit conflict for an operator or job-level decision.
Follow-up 2: What if the catalog cache and object storage disagree?
Treat the catalog version as the commit fact and the object-store metadata location as a verifiable copy. A background checker validates that the location is readable, snapshot references are complete, and manifests resolve. If the catalog points to a missing location, freeze later writes, restore the most recent verifiable version, and keep an audit record. Increasing cache TTL only hides the failure.
Follow-up 3: Why not let every engine talk directly to Hive Metastore?
Multiple language clients would repeat authentication, conflict handling, and feature evolution, with cost growing as engines are added. REST Catalog provides one protocol and capability discovery, while the server centralizes deconfliction, caching, and credential vending. If an organization has a stable Hive deployment and one engine, keeping it may be simpler; migration should be justified by cross-engine compatibility and governance benefits.
Follow-up 4: What if a client logs an object-store credential?
Credentials must be short-lived, least-privilege, and associated with a request ID. Redaction runs in both the client and catalog service. On exposure, revoke or shorten the session, inspect access audit logs, and issue a replacement. For highly sensitive data, a server proxy or remote signing can narrow credential exposure at the cost of direct-read performance.