Prompt and context
A team has a classification dataset built from online behavior logs and human labels. Several models, offline evaluations, and analytics projects will reuse it, but users do not know how samples were collected, who produced the labels, which groups are represented, how missing values were handled, or whether a new business use is allowed. Design a Data Card and explain how it stays auditable.
This fits data engineering, data science, machine-learning engineering, and data-governance interviews. The test is whether you can turn dataset documentation into evidence for a decision, rather than list columns or produce a polished statistics page. Cover provenance and ownership, intended and prohibited use, quality and representation, privacy and licensing, known risks, versioning, and maintenance.
What the interviewer is testing
A strong answer starts from the decision a reader needs to make and organizes the card around evidence. It separates facts, inferences, and unknowns; reports sample size, time range, collection conditions, labeling protocol, missing-data corrections, and key slices. It names the release approver, records versions, handles new uses and severe findings, and updates the card from reproducibility checks and user feedback.
Questions to clarify first
- What is one data unit: an event, a user, a session, or one annotation?
- Is the dataset for training, evaluation, retrieval, reporting, or a safety decision?
- Which time periods, regions, devices, and channels produced the samples, and what population is the target?
- What are the label definitions, annotator qualifications, disagreement process, and audit rules?
- Does it contain personal data, sensitive attributes, third-party licenses, or purpose restrictions? Who approves and maintains it?
A 30-second answer
“I would state the dataset purpose, audience, and prohibited uses first, then record provenance, collection window, sample count and unit, labeling process, version, and owner. I would validate representation, completeness, consistency, label quality, privacy, and licensing, reporting results by important people and time slices. The card would list unknowns, known biases, and use boundaries. An owner approves each release, and versioned snapshots, reproducibility checks, and usage feedback keep it current. If a new use exceeds the evidence, I would pause approval and run additional validation.”
Step-by-step answer
Step 1: Define uses and decision boundaries
Turn “this dataset is good” into questions that can be answered: who will use it, for what, and what happens when it is wrong. List approved uses, uses requiring additional evidence, and prohibited uses. A dataset for customer-intent classification is not automatically authorized for hiring screening; an offline comparison label may not support real-time fraud decisions.
Step 2: Record provenance, unit, and generation
The card should identify source systems, collection dates, covered versions and regions, the unit of observation, and filtering, deduplication, de-identification, and derivation steps. For every correction, record why, who made it, when, and what was affected. Google’s data-quality guidance notes that instruments, human workflows, and rounding can create a gap between data and reality; the final file alone is not evidence.
Step 3: Layer the quality evidence
Report counts, missingness, duplicates, type and unit checks, label agreement, outlier handling, leakage checks, and rerun results. Attach a window, denominator, and threshold to every metric; “zero missing values” must say which fields were excluded. Preserve reviewed examples and adjudication evidence so another person can reproduce the claim.
Step 4: Test representation and slice differences
Define the target population, then slice by region, language, device, time, lifecycle stage, or another dimension that can change performance. Report slice size, label distribution, and key quality metrics, with uncertainty for small samples. A high overall accuracy cannot hide a critical group with little evidence, and an observed difference is not automatically causal.
Step 5: Describe labels and human decisions
Specify operational definitions, positive and negative examples, boundary cases, annotator background, training, double review, adjudication, and audit rates. If labels come from user behavior, call them proxy outcomes rather than ground truth. Version labels and record semantic changes so training does not mix incompatible meanings.
Step 6: State privacy, licensing, and access
List personal and sensitive fields, de-identification methods, re-identification risks, retention, license sources, sharing scope, and approvals. De-identification does not make every use permissible; combinations of fields can still expose groups. Link the card to access and deletion procedures, and call out unverified licenses or provenance gaps.
Step 7: Bind versions, owners, and reproduction material
Pin each release to a data snapshot, code revision, processing configuration, dependency versions, statistical summary, and checksum. Name a business owner, technical maintainer, and risk contact, with the last review date and triggers for another review. A new use requires a fresh comparison of population, slices, licensing, and known risks; copying the old card is insufficient.
Step 8: Review the card with five kinds of questions
Review accountability, utility, quality, impact, and risks. Google’s Data Cards guidance emphasizes that a card should help readers decide whether the dataset fits their task, what consequences may follow, and what restrictions are needed. Reviewers should point to evidence, unknowns, and owners for follow-up rather than assign one unexplained score.
Trade-offs and boundaries
More detail makes responsible reuse easier, but increases maintenance cost. I would make provenance, use, quality, and risk mandatory across projects, while putting one-off analyses in an appendix. Summary statistics do not replace raw-example review; overall metrics do not replace critical slices; automated checks do not replace human review of label meaning and business consequences.
When evidence is missing, write “unknown” and attach a plan to obtain it. Restrict the use to exploration or defer approval instead of using an unvalidated accuracy number to justify a high-risk decision.
Rollout plan and evidence
Start with a dataset whose consumer set and impact are limited. In week one, register purpose, provenance, observation unit, and owners. In week two, run quality and slice checks. In week three, have data science, privacy, and business representatives review the card, then rerun the statistics from a pinned snapshot and publish a version. The pilot record should include findings, corrections, remaining unknowns, and the next review date.
After release, collect user misunderstandings, data defects, new-use requests, and model-performance discrepancies. If issues are first found downstream, the card lacks evidence or discoverability. If nobody can explain a field or limitation, ownership and terminology remain unclear.
Common mistakes and follow-ups
A column dictionary without use boundaries
Names and types do not say how samples were produced, what labels mean, or which uses could cause harm. Add the target population, approved and prohibited uses, and evidence level.
Hiding slice gaps behind an overall average
Large groups can dominate aggregate metrics. Report critical-slice sizes, distributions, uncertainty, and where evidence is insufficient.
Treating cleaning output as reality
Dropping outliers, imputing missing values, and normalizing units change the dataset. Record rules, impact, and unresolved issues; do not call the processed value ground truth by default.
Copying the old card for a new use
Changing purpose changes risks, target population, and required evidence. Reassess license, representation, label suitability, and consequences, and record the owner’s approval or rejection.
How do you prove the Data Card is useful?
Check whether users find limitations before modeling and make safer choices. Compare where problems are first detected, reproducibility success, approval rework, and high-risk misuse events. Interpret changes with dataset version, usage volume, and review effort; fewer incidents alone do not prove causation.