Prompt and context
Design a system that places virtual block-device logic in userspace: the kernel exposes a block device while a userspace service handles loop, remote block storage, or qcow2 mapping. It must support high-concurrency I/O, userspace-process crash recovery, permission isolation, and observability.
Linux ublk separates this framework into a control plane and a data plane: /dev/ublk-control manages devices, /dev/ublkb* carries block I/O, and the userspace service fetches requests and commits results through io_uring passthrough. The interview tests request lifecycles, failure semantics, and security boundaries rather than moving file I/O into a process.
What the interviewer is testing
Show the boundaries among control, the kernel block layer, io_uring, and the userspace backend. Explain the one-to-one relationship among queues, tags, buffers, and completions; choose between per-I/O and batch modes; define requeue, fail, and replay semantics when the server exits; and handle zero-copy, privileges, container isolation, and metrics.
Questions to clarify first
Workload and backend
Confirm read/write ratio, I/O sizes, queue count, latency goals, whether the backend is a local file, remote NBD, or copy-on-write format, and whether ordering guarantees are required.
Failure and data safety
Confirm semantics for a userspace crash, network partition, short backend write, duplicate write, and device removal. Establish whether replay can tolerate double writes.
Permissions and deployment
Confirm who may create devices, who may read /dev/ublkc*, whether the service runs in a container, which zero-copy privileges are allowed, and how tenants are isolated.
A 30-second answer
“I separate control and data planes. Control negotiates queues, depth, and features before starting the device; data fetches requests from io_uring by queue and tag and commits results. Each request has an owner, state, and timeout. When the userspace server exits, I quiesce the device and choose requeue, fail, or replay according to backend guarantees. Copying is the default; zero-copy is limited to trusted, authorized backends. Metrics cover queue depth, completion latency, retries, drops, and recovery time, with consistency tests for removal and recovery.”
Step-by-step deep answer
Step 1: Partition the control plane
Expose commands to add, set/get parameters, start, stop, and delete a device. Adding one negotiates nrhwqueues, queue_depth, and maximum I/O-buffer size; parameters freeze before start, after which /dev/ublkb* is exposed. The userspace service stores device IDs and backend-specific information.
Step 2: Design data-plane requests
The block layer assigns a unique tag per queue, and the userspace service associates requests by (queue, tag). A fixed mapped area describes offset, length, operation, and flags. The service receives notifications through io_uring passthrough and commits status and completed bytes back to the kernel.
Step 3: Choose per-I/O or batch mode
Traditional per-I/O commands are easy to reason about, with one daemon owning each tag. Batch mode prepares and commits multiple requests per queue, reduces syscall overhead, and lets tasks share work dynamically. Do not mix the command sets during migration; compare tail latency, CPU, and load balancing under pressure.
Step 4: Define the recovery state machine
Use states such as running, quiescing, recovering, and failed. On server exit, stop dispatching new I/O, wait for or mark in-flight requests, and issue STARTUSERRECOVERY. REISSUE suits backends that tolerate duplicate writes; FAIL_IO makes in-flight and future requests fail explicitly instead of fabricating success.
on_server_exit:
quiesce_device()
if policy == REISSUE:
requeue_inflight()
else:
fail_inflight_and_future_io()
wait_new_server_ready()
end_user_recovery()Step 5: Handle buffers and zero-copy
The ordinary path uses preallocated userspace buffers and kernel copies, giving simpler boundaries. Zero-copy requires registered fixed buffers, backend segment alignment, and a trusted service that fills READ data and reports byte counts correctly. A mistake can expose uninitialized kernel buffers, so restrict privileges and audit lifetime.
Step 6: Build permission and container isolation
Separate privileged control commands from device access. With unprivileged devices, the kernel still checks ownership of the relevant char device; a container should see only its device nodes. The userspace backend must not receive file, network, or KMS permissions beyond its target device.
Step 7: Verify performance and correctness
Measure IOPS, p50/p99 latency, queue depth, CPU, copied bytes, and recovery time with fio or production-like workloads. Inject server crashes, backend timeouts, short writes, device deletion, and duplicate submissions. Verify every request completes once or fails explicitly by policy; test bounds and checksums in both copy and zero-copy modes.
Model answer
I would split a ublk-like system into control plane, kernel block layer, io_uring data plane, and userspace backend. Control negotiates queues and buffers before start; data tracks requests by (queue, tag) and validates status and byte counts on completion. A server crash triggers quiescing and recovery: replay only for a backend that can tolerate it, fail otherwise. Copying is the default; zero-copy is limited to trusted, aligned, authorized services. Release requires tail-latency, fault-injection, permission-isolation, and removal-consistency tests.
Common mistakes
- Mistake: Letting the userspace server close the device directly. → Why it fails: In-flight requests and kernel-queue state remain undefined. → Fix: Stop dispatch, quiesce, then apply a recovery policy.
- Mistake: Enabling zero-copy for every backend. → Why it fails: Buffer lifetime, privilege, and uninitialized-data risks grow. → Fix: Copy by default and gate zero-copy on capability and audit.
- Mistake: Protecting all queues with one global lock. → Why it fails: Multi-queue concurrency becomes serialized. → Fix: Shard state by queue/tag and measure contention.
- Mistake: Measuring only healthy I/O throughput. → Why it fails: Crashes, short writes, and duplicate writes determine consistency. → Fix: Include the recovery state machine and fault injection in acceptance tests.
Follow-up questions and responses
When should you choose batch I/O?
Choose batch when syscall and notification overhead dominate, queues have enough concurrency, and tasks can share work dynamically. Traditional mode is easier to debug when per-I/O ownership matters or concurrency is low.
How does REISSUE avoid corruption?
Enable it only for idempotent or duplicate-detecting backends, using request IDs, write versions, or log deduplication. If idempotence cannot be shown, fail and let the upper layer recover.
How do you limit zero-copy security impact?
Bind buffer registration, unregistration, and device permissions to one trusted service; validate address, length, alignment, and completion bytes; and forbid writable mappings shared across tenants.
How do you preserve consistency during device deletion?
Stop new requests, wait for submitted requests to complete or fail, confirm the userspace queue is empty, then release device nodes and mappings. Record timeouts as traceable failures instead of silently dropping them.