Prompt and scope
This is a retrieval-system design problem, not a request to name a vector database. The target is semantic search over embeddings with strict tenant isolation, metadata and permission filters, bounded latency, and a measurable freshness and relevance contract. Assume embeddings are produced by an upstream model, documents can be replaced or deleted, and the service must support both batch backfills and continuous updates.
What the interviewer is testing
- Whether you separate ingestion, embedding, index building, query serving, and evaluation.
- Whether you explain why exact search is too expensive and choose an approximate nearest-neighbor (ANN) strategy deliberately.
- Whether filters are applied without silently destroying recall or tenant isolation.
- Whether updates, deletes, model changes, and index rebuilds have explicit visibility semantics.
- Whether you define relevance, recall, latency, cost, and freshness metrics instead of claiming that “similarity” is correct.
Questions to clarify first
- What is the vector dimension, distance function, document count per tenant, and expected query rate?
- Are tenant and ACL filters mandatory hard constraints, or can a result be removed after retrieval?
- Is one-minute freshness required for every write, or only for a subset of hot collections?
- Do we need hybrid keyword plus vector search, reranking, or only nearest-neighbor retrieval?
- Can the embedding model change, and must old vectors remain queryable during migration?
A 30-second answer framework
“I would split the system into an ingestion log, embedding workers, a versioned vector index, and a stateless query tier. Each record carries tenant, ACL, document version, model version, and a tombstone state. Query routing first selects the tenant’s shard or namespace, then performs filtered ANN search and optionally reranks a small candidate set. A mutable delta index handles recent writes while immutable segments are rebuilt in the background; reads merge both and hide stale versions. I would measure recall against an exact or curated gold set, p95 latency, filter miss rate, freshness lag, and cost per query.”
Step-by-step deep dive
1. Define the data and visibility contract
Store documentid, tenantid, ACL attributes, embedding model version, content version, vector, and updated timestamp. A delete is a tombstone with a version, not an immediate assumption that every replica has removed the vector. A query is authorized before retrieval; tenant and ACL predicates are mandatory constraints, while relevance ranking is applied only to authorized candidates.
2. Choose an ANN index and partitioning
Brute-force search costs roughly O(N × D) distance operations for N vectors of dimension D. At 100 million vectors, that is unsuitable for a 150 ms target, so use an ANN index such as HNSW or an inverted-file approach. HNSW favors high recall and fast reads with memory overhead; clustered or quantized indexes reduce memory and cost but add tuning and recall risk. Start with tenant-aware namespaces or shards, then split hot tenants and replicate read-heavy partitions. Do not claim a universal Big-O or recall number; benchmark the chosen library and dimension.
3. Make filtering part of retrieval correctness
A post-filter can return fewer than 20 results when the nearest neighbors belong to another tenant or fail an ACL predicate. Pre-filtering can shrink the candidate space but may make sparse filters expensive. A practical design keeps filterable metadata indexed alongside the vector path, estimates selectivity, and chooses a larger ANN candidate pool or a dedicated filtered segment when needed. Pinecone documents metadata predicates and warns that filtering is part of the search contract; the interview answer should state what happens when fewer than 20 authorized matches exist.
4. Separate fresh writes from compacted segments
Append accepted writes to a durable log and a small mutable delta index. Query both the immutable base segments and the delta, then merge by document version and remove tombstoned IDs. Background compaction builds a new segment, verifies counts and sampled recall, and atomically swaps a manifest. A one-minute SLA is measured from acknowledged write to query visibility, not from embedding-job start. If embedding or indexing is delayed, expose lag and keep the previous version visible rather than pretending the write succeeded.
5. Handle model and schema migrations
An embedding-model change makes old and new vectors incomparable unless the system supports dual indexes or a projection plan. Write model version into every record, backfill a new index, shadow queries against both, and compare recall and latency before switching. Keep the old index until rollback and retention requirements expire. Metadata and ACL schema changes need the same versioned rollout discipline; a vector match must never bypass a newly added permission field.
6. Design the query path and overload policy
The query tier authenticates the tenant, normalizes the query, chooses the model version, and fans out only to relevant shards. It enforces a deadline, bounded candidate count, and cancellation. If a shard times out, return a partial result only when the API marks completeness; otherwise fail closed for security-sensitive searches. Cache embeddings and stable public queries, but never share a cache entry across authorization scopes. Admission control protects index memory and reranking capacity under bursts.
7. Measure relevance, freshness, and cost
Create a labeled query set with relevant and forbidden documents. Compare ANN results with an exact-search baseline on sampled partitions, and report recall@20, precision or nDCG, filter correctness, and authorization leakage tests. Track p50/p95/p99 latency, candidate counts, index build time, write-to-visible lag, tombstone backlog, memory per vector, and cost per thousand queries. Offline metrics catch ranking regressions; online click metrics need guardrails because position bias can make a bad result look popular.
Trade-offs and boundaries
HNSW versus clustered or quantized indexes
HNSW is a strong first choice for read-heavy workloads when memory is available. IVF or product quantization can lower memory and improve scan efficiency at large scale, but requires training, tuning, and recall validation. Choose from update rate, dimension, tenant skew, and hardware budget; do not choose from a product name alone.
Native vector store versus an existing database
A general-purpose database with a vector index is attractive when collections are moderate and joins, transactions, and ACL data must stay together. A dedicated service is justified when vector search dominates capacity, requires specialized ANN indexes, or needs independent scaling. Keep the source-of-truth document and permission records outside the index when the vector store cannot provide the required transactional guarantees.
One index per tenant versus shared partitions
Per-tenant indexes simplify isolation and noisy-neighbor control but multiply overhead. Shared indexes use hardware better, yet require strict metadata filtering and fair scheduling. Use namespaces or partition keys for ordinary tenants and promote very large or regulated tenants to isolated capacity.
High-quality sample answer
“I would start with a durable write log and version every document, ACL, and embedding model. The query tier authorizes the tenant, fans out to relevant shards, runs filtered ANN search, and merges a fresh delta index with immutable segments. HNSW is a read-heavy baseline, but I would benchmark it against clustered or quantized indexes using recall@20 and p95 latency. Rebuilds publish a manifest atomically, model changes use shadow queries, and deletes are versioned tombstones. The service reports filter correctness, write-to-visible lag, authorization leakage tests, memory per vector, and query cost; relevance is a measured contract.”
Common mistakes
- Treating an ANN library choice as the architecture while ignoring ingestion, deletes, and rebuilds.
- Applying ACL filters after retrieval and silently returning fewer or unauthorized documents.
- Claiming fixed recall or latency without naming dimension, index settings, hardware, and workload.
- Replacing an embedding model in place so old and new vectors become incomparable.
- Measuring only clicks and never maintaining an exact or labeled relevance baseline.
Follow-up questions and answers
What if an ACL filter leaves only three matches?
Return three with an explicit totalorcompleteness signal, or return an empty/insufficient response according to the API contract. Never fill the remaining slots with unauthorized or unfiltered results. Increase the candidate pool only within the authorized search path.
How do you rebuild an index without losing writes?
Replay the durable log into the new segment, record a high-water mark, catch up the writes after that mark, validate counts and sampled recall, then atomically publish a manifest. Keep the delta path active until the swap completes; rollback by restoring the prior manifest.
When can you delete old model vectors?
Only after shadow evaluation passes, the new model is serving, rollback and retention windows close, and every query path rejects the old model version. Deleting by time alone is unsafe when delayed jobs or replay consumers still reference it.