Source-linked AI summary
Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI
Mahima Pushkarna, Andrew Zaldivar, Oddur Kjartansson
TL;DR
As datasets become more complex, the paper addresses the need for documentation that is transparent, comprehensible, and useful across stakeholders. It proposes Data Cards and supporting frameworks, then examines their development through collaborative studies and case studies. The reported experience shows that Data Cards can reveal opportunities to improve dataset design, while their use is bounded by distinct needs for technical agents and product users.
Problem
Complex datasets require documentation of their origins, development, intent, ethical considerations, and evolution, but practical standardized mechanisms for transparent documentation have had limited success.
Method
The paper proposes Data Cards—structured lifecycle summaries of dataset facts, explanations, rationales, and instructions—and frameworks for organizing, adapting, and evaluating them.
Results
Case studies showed that Data Card creation revealed opportunities to improve dataset design, including investigating unknown values and establishing shared labeling lexicons.
Takeaways & Limitations
Data Cards support informed decision making by making dataset context, provenance, usage, evaluations, and uncertainty more accessible to stakeholders.
Takeaways & Limitations
Data Cards are intended for agents with technical expertise, while product users require alternative transparency artifacts designed for their experiences.
Abstract
from arXiv · showhide
As research and industry moves towards large-scale models capable of numerous downstream tasks, the complexity of understanding multi-modal datasets that give nuance to models rapidly increases. A clear and thorough understanding of a dataset's origins, development, intent, ethical considerations and evolution becomes a necessary step for the responsible and informed deployment of models, especially those in people-facing contexts and high-risk domains. However, the burden of this understanding often falls on the intelligibility, conciseness, and comprehensiveness of the documentation. It requires consistency and comparability across the documentation of all datasets involved, and as such documentation must be treated as a user-centric product in and of itself. In this paper, we propose Data Cards for fostering transparent, purposeful and human-centered documentation of datasets within the practical contexts of industry and research. Data Cards are structured summaries of essential facts about various aspects of ML datasets needed by stakeholders across a dataset's lifecycle for responsible AI development. These summaries provide explanations of processes and rationales that shape the data and consequently the models, such as upstream sources, data collection and annotation methods; training and evaluation methods, intended use; or decisions affecting model performance. We also present frameworks that ground Data Cards in real-world utility and human-centricity. Using two case studies, we report on desirable characteristics that support adoption across domains, organizational structures, and audience groups. Finally, we present lessons learned from deploying over 20 Data Cards.
1 INTRODUCTION
The paper introduces Data Cards as a structured, transparent framework for documenting datasets across their lifecycle. They provide context, explanations, and rationales to support responsible decisions by diverse stakeholders.
- Beyond metadata, Data Cards explain dataset provenance, representation, usage, and fairness-informed evaluations.They capture information and context that cannot be inferred directly from the dataset.
- Data Cards are designed as accessible boundary objects that support informed decisions about data used in ML products, policy, and research.They complement longer-form documentation frameworks such as Model Cards, Data Statements, and Datasheets for Datasets.
- Case studies showed that creating Data Cards revealed opportunities to improve dataset design, including investigating unknown values and aligning labeling lexicons.These discoveries emerged during practical development of Data Cards.
- Data Cards are structured summaries of essential dataset facts needed by stakeholders across the dataset lifecycle.They are presented as a transparency artifact for production and research environments.
- The framework is accompanied by adaptable methods for organizing information, framing questions, and evaluating answers.The paper specifically introduces OFTEn as a repeatable knowledge-acquisition framework for producing transparent documentation.
2 DEVELOPMENT METHODOLOGY
The development methodology combines human-centered, participatory, and collaborative studies to make Data Cards consistent, comprehensive, intelligible, and useful across stakeholder groups. It also identifies practical constraints, including fragmentation during iterative template use and distinct explanation needs for technical agents and product users.
- DEVELOPMENT METHODOLOGY: Over 24 months, the authors iteratively designed Data Cards with dataset and ML teams in a large technology company.The work borrowed from human-centered design, participatory design, and human-computer interaction.
- DEVELOPMENT METHODOLOGY: The authors worked with 12 teams to create 22 Data Cards covering image, language, tabular, video, audio, and relational datasets.The teams included research software engineers, research scientists, data analysts, and data program managers.
- DEVELOPMENT METHODOLOGY: External focus-group feedback from nine participants informed a working definition of transparency and recommendations for teams creating Data Cards.Participants represented UX and HCI research, policy, product design and development, academia, and law.
- DEVELOPMENT METHODOLOGY: Participatory studies and a MaxDiff survey examined stakeholder use cases, information needs, and the relative importance of 31 documentation themes.The internal study included 30 experts spanning sixteen company teams.
- DEVELOPMENT METHODOLOGY: Data Cards are restricted to technical agents, while product users require alternative transparency artifacts designed for their product experiences.The paper distinguishes agents from users and identifies different explanation and control needs.
- DEVELOPMENT METHODOLOGY: The methodology emphasizes comparable documentation, concurrent creation, audience-aware language, and explicit communication of uncertainty.Participants treated known unknowns as important for decision making, while technical and dense language could make transparency artifacts opaque.
3 DATA CARDS
Data Cards provide a flexible, modular framework for documenting datasets across their lifecycle while supporting readers with different information needs. Their layered questions and OFTEn framework organize both explicit dataset facts and the processes, assumptions, and downstream uses that shape responsible decisions.
- Principles: Data Cards are designed as a flexible, modular, and extensible framework for documenting datasets while preserving utility and intent.
- Design and Structure: Blocks combine titles, prompting questions, instructions, and structured answer inputs suited to text, choices, tables, numbers, code, visualizations, tags, links, and demos.
- Scopes: Telescopes provide broad overviews, periscopes support suitability assessments, and microscopes elicit implicit decisions, assumptions, and policies shaping the dataset.
- OFTEn Framework: OFTEn organizes dataset-lifecycle questions around Origins, Factuals, Transformations, Experience, and n=1 example, including feedback from downstream stakeholders.
- Evaluation: The framework establishes a trail for tracking AI-system performance and exposing caveats or limitations relevant to dataset users.
- Design and Structure: Template adaptation could produce identical but inaccurate responses, propagate errors, and fragment forked Data Cards when blocks or questions were modified.
- Evaluation: Reviewers found the evaluation dimensions easier to use for structured and actionable feedback, although some use-case suitability remained difficult to determine.
4 CASE STUDIES
Two case studies show Data Cards functioning as collaborative exercises that organize distributed knowledge and expose assumptions about sensitive attributes and dataset design. The process prompted teams to clarify labeling concepts, investigate unknown values, and improve documentation for downstream users.
- Computer Vision Dataset: A computer-vision fairness dataset contained 100,000 bounding boxes across 30,000 images, manually annotated with perceived gender and age-range presentation attributes.
- Computer Vision Dataset: Dataset authors, internal reviewers, and paper authors contributed complementary lifecycle knowledge, responsible-AI analysis, and human-centered design perspectives.
- Computer Vision Dataset: Creating the Data Card revealed that nearly 40% of perceived age-range labels were unknown and prompted investigation of labeling criteria and bounding-box size.
- Computer Vision Dataset: The investigation attributed high unknown rates partly to 30% of bounding boxes occupying less than 1% of their images, leading to added documentation and visualizations.
- Biographical Dataset: A team developing a geographically diverse dataset recognized that gender, race, age, background, and profession required explicit consideration beyond country-based geographic proxies.
- Biographical Dataset: After approximately 20 hours of documentation, the team clarified a perceived-gender definition using masculine, feminine, and neutral labels for biographies describing collections of individuals.
5 DISCUSSION
Data Cards support multiple transparency, review, comparison, reuse, and accountability activities by connecting dataset documentation with model and system artifacts. Their benefits depend on timely creation, organizational rigor, adaptable infrastructure, and mechanisms for collaboration and search.
- Benefits and Adoption: Creating Data Cards during dataset development can incorporate multiple perspectives, consolidate lifecycle documentation, and enable earlier responsible-AI feedback.
- Implementation: Creating Data Cards as a final step increased the perceived work required, while pipeline-connected knowledge infrastructure was identified as important for incorporating new knowledge.
- Interpretive Flexibility: A single Data Card can support reviews, audits, use-case decisions, dataset comparison, research reproduction, and adoption tracking.
- Artifact Networks: Embedding Data Card sections into model and system artifacts creates a network for fairness and accountability interrogations, knowledge transfer, and dataset reuse.
- Adoption Conditions: Organizations adopted Data Cards organically, but quality and comprehensiveness depended on producer rigor, uncertainty expression, dataset knowledge, and organizational requirements.
- Infrastructure: Centralized repositories with search and filtering over hundreds of Data Cards can help stakeholders identify suitable datasets and distribute accountability for dataset use.
6 CONCLUSION
The paper presents Data Cards as a transparent, purposeful framework for documenting datasets through human-centered design, adaptable structures, and evaluation frameworks. Two case studies provide qualitative and anecdotal evidence of efficacy, while the conclusion identifies template fragmentation and limited automation as important constraints.
- The paper presents Data Cards as a framework for transparent and purposeful dataset documentation, supported by human-centered design and adaptation frameworks.
- Two case studies provide qualitative and anecdotal evidence concerning Data Cards’ contribution to responsible AI systems.
- Using Google Docs enabled collaboration and preserved development history but limited answer formats, fragmented templates, and prevented streamlined collection and automation.
- Future work requires a principled way to extend and adapt templates without compromising comparability.
A TYPOLOGY OF STAKEHOLDERS
The figure presents a typology of dataset-lifecycle stakeholders, organized by type, identifiers, and tasks with example roles.
- Stakeholders are classified by type, identifiers, and tasks.
- The typology includes example roles across the dataset lifecycle.
- Including non-technical and indirect stakeholders early supports foresight for Data Card adoption.
B OFTEN FRAMEWORK AS A GENERATIVE TOOL
The table demonstrates how OFTEn generates questions about data consent across a dataset’s lifecycle and supports documentation standardization and forensic investigation over time.
- OFTEn generates questions about data consent across a dataset’s lifecycle.
- The framework was used during template creation to anticipate standardization requirements.
- OFTEn enables forensic investigation of dataset documentation over time.
C DATA CARD FOR COMPUTER VISION DATASET
The supplied figures show a five-page Data Card for a computer vision dataset.
- The computer vision Data Card begins on page 1 of 5.
- The computer vision Data Card continues through pages 2, 3, and 4 of 5.
- The computer vision Data Card concludes on page 5 of 5.
D DATA CARD FOR LANGUAGE TRANSLATION DATASET
The supplied figures show a three-page Data Card for a language translation dataset.
- The language translation Data Card begins on page 1 of 3.
- The language translation Data Card continues on page 2 of 3.
- The language translation Data Card concludes on page 3 of 3.
E DATA CARD TEMPLATE
The Data Card template organizes documentation across dataset overview, provenance, use, transformations, annotations, validation, sampling, applications, benchmarks, terminology, and reflections. Its sections capture both descriptive information and guidance for responsible dataset use, including sensitive attributes, access, retention, deletion, and potential risks.
- Dataset Overview: The Dataset Overview section provides a top-level summary, while Sensitivity and Version and Maintenance document risks, handling, updates, and maintenance.The overview can also be linked from other transparency artifacts to a more complete Data Card.
- Motivations & Use: Motivations and Intended Use describe the dataset’s research or application goals and known suitable or unsuitable use cases.Access, Retention and Wipeout document storage, prerequisites, policies, retention periods, exceptions, and deletion guidance.
- Dataset Provenance: Dataset Provenance records collection methods, upstream sources, selection criteria, source relationships, and recent or planned updates.The template is designed to accommodate multiple collection methods and sources.
- Extended Use: Extended Use documents joining, forking, sampling, machine-learning applications, benchmark performance, training and testing splits, and associated descriptive statistics.The template encourages automating split statistics to improve accuracy and rigor at scale.
- Data Processing: Transformations, Annotations, Validation Methods, and Sampling Methods capture processing steps, annotation characteristics, validation details, and sampling metrics or summaries.Transformation details can flag residual or system-level risks requiring oversight, while sampling documentation is conditional.
- Applications and Interpretation: Known Applications & Benchmarks, Terms of Art, and Reflections on Data provide model and evaluation information, interpretive terminology, caveats, and space for uncaptured context.Known application documentation includes expected performance and caveats for users.