Data engineering interview: How should Parquet Modular Encryption define key and read boundaries?

Prompt and scope

A data lake has public and sensitive columns in Parquet files. The team wants to encrypt sensitive data and metadata while retaining projection, predicate pushdown, and compression. Design footer encryption, column keys, key custody, read authorization, legacy-reader behavior, and rotation recovery.

This question discusses capabilities in the Parquet Modular Encryption specification; it does not assume every engine or language binding supports the same key configuration.

What interviewer is testing

• Whether you distinguish encrypted footers, encrypted columns, and page-only protection.
• Whether KEKs, DEKs, key metadata, and KMS permissions are separated instead of storing a master key in the file.
• Whether you understand how an encrypted footer affects schema, statistics, predicate pushdown, and legacy readers.
• Whether rotation, revocation, rewrite, and observable decryption failures are designed together.

Clarifying questions

1. Which columns are sensitive, and must schema, row groups, and statistics be hidden too?
2. Which columns do query engines need, and do they support column keys and footer decryption?
3. Is key custody a central service, cloud HSM, or caller-injected key?
4. Are files replicated across accounts or regions, and how do backups obtain decrypt permission?
5. Does rotation target new files, historical rewrites, or immediate revocation?

A 30-second answer

“Parquet Modular Encryption can protect file data and metadata while retaining columnar features, but an encrypted footer hides schema and row-group information from readers without its key. I would separate KMS master keys from file or column keys and authorize readers per column. A legacy reader should report unsupported encryption rather than call the file corrupt. Rotation writes new files with newly wrapped keys and rewrites history under control, while recording key versions and failures. A capability matrix must verify predicate pushdown for each engine.”

Step-by-step design

1. Choose the protection scope

Footer encryption protects FileMetaData, including row groups and column chunks; column encryption can use different keys for sensitive columns. If the footer stays plaintext, a reader may see schema or statistics even when it cannot read encrypted pages. Choose the scope from the threat model.

2. Separate key responsibilities

A KMS or HSM stores key-encryption keys. Writing creates or obtains data-encryption keys and stores protected key metadata in the file. A reader authorization service resolves the metadata and asks KMS to unwrap only permitted keys; the file must not contain a directly usable master key.

3. Evaluate query capabilities

Apache Parquet documents that modular encryption can encrypt and authenticate data and metadata while retaining projection, predicate pushdown, encoding, and compression. A query engine still needs the relevant keys first. An encrypted footer can block a keyless scanner from reading schema and statistics, so test public-only projection, sensitive-column filters, and mixed projections.

KMS KEK -> wraps file/column DEK -> protected key metadata
reader authorization -> unwrap allowed DEK -> decrypt footer/pages

4. Identify files and legacy readers

Encrypted-footer files use different magic bytes: Apache Parquet documents PARE, while plaintext files use PAR1. A legacy reader may classify the encrypted file as unsupported. Mark capability in a registry or probe, rather than waiting for a vague corruption error during a query.

5. Design rotation and revocation

Rotation should first affect new files: new DEKs are wrapped by a new KEK, and old files are rewritten according to retention. Immediate revocation makes KMS refuse unwrapping and reads fail; the platform needs a rewrite queue, key versions, and recovery steps. Rewrite to a temporary file and atomically replace it so an interruption cannot leave a half-encrypted file.

6. Observe failures by layer

Record file ID, footer or column key version, principal, KMS latency, decryption failure type, and queried columns; never log plaintext keys or values. Separate authorization denial, missing key, authentication-tag failure, unsupported legacy reader, and file corruption for alerts and remediation.

Model high-quality answer

“I would choose whether to hide the footer from the threat model. Modular encryption can protect footer, columns, and pages; footer encryption hides schema and statistics, while column keys limit sensitive-column access. KMS holds KEKs, and file or column DEKs are referenced through protected metadata; readers unwrap only after authorization. A capability matrix tests public projection, sensitive filters, and predicate pushdown. PARE versus plaintext PAR1 lets the platform identify legacy-reader risk early. Rotation uses new files, controlled rewrites, and atomic replacement, with key versions and failure classes recorded; revocation uses KMS denial plus a recovery queue.”

Common mistakes

• Write a KEK directly into the file → possession of the file enables decryption → store only protected key metadata.
• Encrypt columns without evaluating the footer → schema and statistics may still leak → choose footer mode from the threat model.
• Assume every engine supports column keys → production queries reveal a capability gap → maintain an engine and binding matrix.
• Overwrite files in place during rotation → interruption leaves unreadable partial output → write temporary output and atomically replace.
• Call authentication failure a corrupt file → investigation follows the wrong path → separate KMS, authorization, tag, and format errors.

Follow-up questions and responses

Does footer encryption disable predicate pushdown?

Not necessarily. After obtaining the footer and required column keys, the format can retain columnar reads and predicate pushdown. A scanner without the footer key cannot read statistics; actual behavior depends on engine encryption support.

Why use column-level keys?

Different principals may be allowed to read different columns. Column keys narrow authorization from the whole file to sensitive columns and reduce exposure when granting access to public columns.

What should a legacy reader do with PARE?

It should clearly report unsupported modular encryption and direct the operator to a compatible reader or decrypt-and-rewrite path. It should not silently treat a PARE file as a corrupt PAR1 file.