How do you safely ingest untrusted Apache Arrow IPC data?

1. Scenario and threat model

An analytics service receives Arrow IPC streams from external tenants and passes them to Python, Rust, and Java consumers. An attacker may craft inconsistent schemas, huge lengths, out-of-bounds offsets, recursive nesting, malicious dictionaries, or buffers that exceed the receiver's budget. The goal is batch throughput while ensuring a parse failure affects only the current request.

Define the trust boundary first: network bytes, IPC metadata, buffer contents, and business schemas are untrusted; an authenticated tenant is not automatically safe data. The security model should cover the Arrow Columnar Format, C Data Interface, and IPC rather than relying only on one language binding's defaults.

2. State Arrow's security assumptions

The columnar format describes buffers, lengths, offsets, null bitmaps, dictionaries, and nested layouts. Zero-copy reduces copying but can let a consumer read memory regions described by external input. A secure implementation validates metadata against actual buffers before handing data to a compute engine.

Do not equate a valid format specification with validated input. Protocol versions, extension types, and language implementations have different boundaries. Fix the supported versions and type set, and define an explicit policy for unknown fields.

3. Validate schema, lengths, and offsets

The first stage parses only bounded metadata and checks field count, nesting depth, data types, dictionary references, and batch row limits. For each buffer, verify that offset + length cannot overflow and lies within the actual allocation. For variable-length strings and lists, verify monotonic offsets that stay inside the values buffer.

if offset < 0 or length < 0: reject
if offset > buffer_size: reject
if length > buffer_size - offset: reject
if nesting_depth > MAX_DEPTH: reject

Use overflow-safe arithmetic, reject negative or implausibly large integers and schema-incompatible null counts. Never allocate the attacker-declared length before validation.

4. Apply resource budgets and backpressure

Set per-request budgets for metadata bytes, columns, rows, total buffer bytes, nesting depth, dictionary size, and CPU time. Pass the budget into each batch parser; cancel the request and release allocations immediately when it is exceeded.

Use bounded reads and batch backpressure for an IPC stream instead of loading the upload into memory at once. Bound expansion from compression, dictionaries, and repeated references so a small input cannot become a large allocation. Isolate quotas per tenant so one malicious request cannot consume every worker.

5. Decide when to keep zero-copy and when to copy

Keep read-only zero-copy only after proving buffer ownership, lifetime, and bounds, and confirming that downstream code cannot mutate it. Network receive buffers, temporary mmaps, or cross-language FFI memory that may be released, reused, or written must be copied into a controlled arena.

Copying is not a failure; it is the security boundary that turns an untrusted lifetime into a controlled one. Choose per column: share validated numeric buffers, and copy strings, dictionaries, or nested columns when the budget permits. Record copied bytes and latency instead of hiding risk behind “full zero-copy.”

6. Isolate parsing, errors, and compatibility

Run the parser in a constrained worker or process with CPU, memory, file-descriptor, and wall-time limits. Return structured reasons such as unsupported version, schema mismatch, offset out of bounds, or budget exceeded; never echo raw payloads or internal addresses.

Use allowlists for extension types, unknown metadata, and future fields. For compatibility, convert to an internal schema before entering the query engine. Before upgrading Arrow 25 or language bindings, run the same malicious corpus as differential tests and compare error classes, peak memory, and results.

7. Observe, fuzz, and respond to incidents

Record tenant, format version, schema hash, rejection reason, input size, parse duration, peak memory, copied bytes, and cancellations; hash payloads instead of logging sensitive content. Alert on each rejection class and distinguish a misconfigured client from a sustained attack.

Fuzz random schemas, offsets, null bitmaps, dictionaries, and truncated streams, using AddressSanitizer, MemorySanitizer, or an equivalent tool to detect crashes and out-of-bounds access. Keep minimized reproducers and regress them after fixes. For a production anomaly, isolate the tenant or format version first, then analyze the sample and rotate affected credentials.

8. Rubric and follow-ups

Must explain

• Treat Arrow metadata, buffers, and business schemas as untrusted; validate lengths, offsets, depth, and dictionary references.
• Control memory and CPU with budgets, backpressure, and isolation rather than treating zero-copy as an unconditional goal.
• Provide copy boundaries, structured errors, fuzzing, observability, and upgrade strategy.

Follow-up questions

• A string column has monotonic offsets, but the final value exceeds the buffer. At which layer do you reject it?
• How do you prevent a dictionary or nested list from expanding into an unbounded resource cost?
• How would you prove that Python, Rust, and Java bindings reach the same conclusion for one malicious input?

Scoring guide

An excellent answer connects format validation, resource governance, memory ownership, and runtime evidence: reject impossible layouts, consume batches within budgets, and prove with isolation, fuzzing, and metrics that untrusted input cannot become a service-wide failure.