Source-linked AI summary
Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure
Ben Hutchinson, Andrew Smart, Alex Hanna, Emily Denton, Christina Greer, Oddur Kjartansson, Parker Barnes, Margaret Mitchell
TL;DR
ML datasets often have opaque creation processes despite their central role in systems whose societal effects raise accountability concerns. This paper frames dataset development as cyclical engineering and adapts software lifecycle practices into documentation, oversight, and maintenance frameworks. The resulting framework makes dataset work and decision-making more visible while supporting accountability, though its benefits require further empirical evaluation and maintenance costs must be weighed against benefits.
Problem
ML datasets are often poorly documented, maintained, and answerable, leaving their creation processes and the decisions shaping them insufficiently visible for accountability.
Method
The paper frames datasets as cyclical technical infrastructure and engineering practice, adapting software lifecycle documentation, oversight, and maintenance practices to dataset development.
Results
The proposed framework makes dataset development processes, ownership, decisions, and data labor more visible through stage-specific accounts that support communication, reviews, audits, and accountability.
Takeaways & Limitations
Accountable ML development requires deliberate dataset processes in which responsible owners consult experts and produce detailed accounts of what happens and why.
Takeaways & Limitations
The benefits of data documentation need more empirical evaluation, and maintenance costs must be continually weighed against benefits.
Abstract
from arXiv · showhide
Rising concern for the societal implications of artificial intelligence systems has inspired demands for greater transparency and accountability. However the datasets which empower machine learning are often used, shared and re-used with little visibility into the processes of deliberation which led to their creation. Which stakeholder groups had their perspectives included when the dataset was conceived? Which domain experts were consulted regarding how to model subgroups and other phenomena? How were questions of representational biases measured and addressed? Who labeled the data? In this paper, we introduce a rigorous framework for dataset development transparency which supports decision-making and accountability. The framework uses the cyclical, infrastructural and engineering nature of dataset development to draw on best practices from the software development lifecycle. Each stage of the data development lifecycle yields a set of documents that facilitate improved communication and decision-making, as well as drawing attention the value and necessity of careful data work. The proposed framework is intended to contribute to closing the accountability gap in artificial intelligence systems, by making visible the often overlooked work that goes into dataset creation.
1 INTRODUCTION
The paper argues that poorly documented dataset creation undermines ML accountability and proposes an intentional, cyclical development process grounded in engineering and software practices. Documentation, oversight, and maintenance make dataset work more visible and support reviews, audits, and responsibility.
- Poorly documented and maintained datasets have opaque creation processes and frequently contribute to ML errors and harmful impacts.The paper presents engineering best practices around visibility and ownership as necessary, though not sufficient, for accountability.
- The paper frames datasets as technical infrastructure whose creation conditions are often invisible and undervalued.This framing motivates adapting software engineering and infrastructure practices to dataset accountability.
- Dataset development should be treated as a rigorous, visible engineering process rather than a hastily assembled or post hoc-justified activity.The proposed approach emphasizes deliberative goal-setting, documentation, reviews, and audits throughout development.
- The lifecycle is non-linear, with analysis, design, and evaluation central to iterative dataset development.This lifecycle frames dataset creation as a cycle rather than a one-time collection step.
- Key practices include lifecycle documentation, diverse oversight through audits and reviews, and maintenance for errors, technical debt, postmortems, and changing contexts.These practices are presented as complementary mechanisms for improving accountability across dataset creation and use.
2 DATASETS AS INFRASTRUCTURE
The paper presents datasets as both technical and knowledge infrastructures whose development and upkeep are routinely undervalued. It connects this devaluation to underspecified design work and argues for recognizing data labor through documentation and institutional practices.
- Datasets are shared, architected information-delivery bases with standardized interfaces that enable ML development and new processes.Their infrastructure role makes dataset design and maintenance foundational to the systems that depend on them.
- Dataset work is devalued relative to algorithmic achievements, while collection, cleaning, annotation, and maintenance receive little recognition.Benchmark use can increase a dataset’s value as a measuring instrument without increasing recognition for the labor that sustains it.
- Critical dataset design decisions are frequently underspecified, and ML data often lacks version control and institutional repository practices.The paper contrasts data’s foundational role with its limited treatment in venues, textbooks, and specialization structures.
- The field’s focus on narrow benchmark differences contributes to documenting algorithmic changes more than the processes of dataset creation.The paper describes this imbalance as part of a broader cycle that keeps dataset labor in the background.
- The authors propose robust documentation and greater academic recognition to make dataset labor more visible and valued as infrastructure work.They advocate treating data as first-class ML infrastructure and rewarding papers and expertise focused on data.
3 DATASET DEVELOPMENT AS ENGINEERING
The paper characterizes dataset development as engineering: datasets are purpose-built, fallible models of reality shaped by sociotechnical choices. Because goals, stakeholders, and reality change, development requires iteration, feedback, measurement, and shared responsibility.
- Datasets are designed for specific purposes, so their usefulness must be judged in relation to those purposes rather than as universally valid representations.The engineering framing emphasizes knowledge-how and practical, context-dependent goals.
- Datasets are inseparable from politics, so accountable engineering requires acknowledging the roles, stakes, and expertise of different participants.The paper rejects the view that dataset engineering is purely technical.
- Datasets are engineering models: abstracted, incomplete, and potentially fallible representations used for design, testing, prediction, evaluation, and experimentation.Their goals determine which dimensions of reality are prioritized and which are deemphasized or overlooked.
- Dataset development requires iteration and feedback loops because stakeholder needs, design assumptions, goals, and external reality can change.The paper treats engineering problems as ill-structured and emphasizes revising both problem definitions and solutions.
- 3.5 Cultures of dataset work mirror those of software: Software and ML development cultures share coordination challenges, while hacker culture can devalue the incremental and cooperative care needed for high-quality datasets.The paper identifies shared documentation and auditable processes as a managerial response to these cultural pressures.
- 3.5 Cultures of dataset work mirror those of software: Correspondences between dataset and software engineering cultures, goals, and methods motivate adapting software development practices to improve dataset visibility and quality.The paper uses this comparison as a bridge to its accountability framework.
4 TOWARDS ACCOUNTABILITY FOR DATASETS
The paper frames accountability as answerability supported by shared information, deliberation, and consequences. It proposes using the software development lifecycle to organize dataset accounts before, during, and after data collection and labeling.
- Accountability framework: Accountability requires actors to provide information that enables meaningful discussion and consequences.The paper focuses on the information-sharing phase and asks who should provide dataset information and what they should provide.
- Accountability framework: Dataset accounts are records of information needed as a necessary but insufficient precondition for accountability.The paper treats recording accounts fundamentally as bookkeeping while emphasizing that the details determine responsibility attribution and answerability.
- Lifecycle model: The dataset development lifecycle organizes accounts ex ante, in medias res, and ex post.This temporal organization covers documentation before, during, and after data collection and labeling.
- Lifecycle model: The proposed lifecycle model expands data management into requirements analysis, design, implementation, evaluation, and maintenance.It adapts the software development lifecycle to dataset development based on parallels between data infrastructure and engineering processes.
- Engineering practices: Engineering recommendations include written assumptions, independent reviews, sensitivity studies, visualization, and explicit attention to dataset limitations.The recommendations also advise treating datasets as guilty until proven innocent and evaluating them against experience and judgment.
4.2 Documentation of Dataset Development
Dataset development documentation turns lifecycle decisions into owned accounts covering requirements, design, implementation, testing, and maintenance. The framework emphasizes deliberate stakeholder consideration, explicit tradeoffs, testing for harms, and adaptation over time.
- Lifecycle documentation: Lifecycle documents assign owners and duties while recording the stages of dataset development.The paper identifies five lifecycle stages and treats their documentation as accounts.
- Requirements: Requirements analysis records why data is needed, intended and unintended uses, affected parties, and the six Ws.It also gathers and resolves conflicting stakeholder needs through accountable mechanisms.
- Requirements: Requirements work can be undermined by uncommunicated needs, vocabulary differences, premature data collection, and cognitive biases.Stakeholders may also force requirements to fit an existing dataset rather than articulate their specific needs.
- Design: Design distinguishes dataset goals from the means used to achieve them and evaluates whether existing datasets satisfy current requirements.The Dataset Design Document records the plan, design justifications, alternatives, tradeoffs, sourcing, annotation, privacy, and quality decisions.
- Implementation: Dataset implementation documents fine-grained decisions about data structures, control flows, and context-dependent choices.The paper compares implementation accounts to code comments.
- Testing: Dataset testing records evaluations and results, including wellformedness, fidelity, requirements testing, and adversarial testing.Adversarial or test-to-fail testing seeks unforeseen harms arising from dataset use.
- Maintenance: Maintenance addresses corrective, adaptive, and preventive goals because errors, poisoning, misconstrued labels, and changing phenomena may emerge.The framework supports manual edits and batch updates as part of ongoing human-data interaction.
- Maintenance: Maintenance costs must be weighed against benefits, and deletion may become preferable even though shared binary datasets cannot easily be fully removed.The paper therefore links contestability to tightly controlled dataset sharing absent complex cryptographic solutions.
4.3 Dataset Audits and Reviews
Dataset audits and reviews deliberate over documented accounts throughout the lifecycle. Audits examine artefacts against organizational commitments, while reviews also occur at requirements, design, and changing-context checkpoints.
- Audits: Audits review datasets and models through scoping, mapping, artefact collection, evaluation, and reflection.Audit teams analyze collected artefacts against organizational policies and commitments.
- Reviews: Dataset development documentation forms part of the audit trail and supports auditability by design.The paper presents audits as one review form among requirements reviews, design reviews, and other lifecycle checks.
- Reviews: The entire dataset lifecycle requires review during initial development and when contextual circumstances change.This extends review beyond a single post hoc audit.
4.4 Lessons from Infrastructure Governance
Infrastructure governance offers dataset accountability mechanisms for anticipating risks, managing contested decisions, and recognizing long-term and intergenerational effects. The paper emphasizes that governance choices involve tradeoffs between hierarchical and negotiated coordination.
- Risk governance: Infrastructure governance suggests prioritizing risk anticipation and management for dataset projects.The paper links this need to poor initial understanding of risks, challenges, costs, and benefits.
- Decision-making: Stakeholder consultation in requirements analysis must account realistically for how contested decisions are resolved.Possible mechanisms include hierarchical appeals to institutional authority and negotiated coordination requiring consensus.
- Long-term impacts: Dataset governance should consider that problems may take years to surface and that benefits and effects accrue across groups and generations.The paper specifically raises temporal discounting and effects on people too young to participate in present decisions.
5 DISCUSSION
The discussion connects dataset documentation to accountability, contestability, maintenance, reproducibility, recognition of data work, and clearer methodological practice. It argues that treating datasets as engineered infrastructure supports broader organizational and scientific improvements.
- Benefits of Data Documentation: Dataset documentation can clarify contracts among deployed-system components and help developers, maintainers, and users determine how much trust to place in a system.This is especially relevant to productionized and continuously trained systems.
- Benefits of Data Documentation: A lifecycle approach treats dataset maintenance as ongoing engineering work, including technical-debt mitigation, postmortems, premortems, and adaptation to changing contexts.Postmortems build institutional memory, while premortems identify gaps and vagueness during requirements analysis.
- Benefits of Data Documentation: Documentation makes dataset development more legible for contestation, audits, reviews, and transparent reporting.Visibility helps stakeholders challenge datasets and supports informed decisions, replication, and auditing.
- Benefits of Data Documentation: Recording provenance and epistemic uncertainty in data challenges the idea of a view from nowhere and makes datasets relational artifacts of their creation.The paper parallels explicit uncertainty with engineering cultures of exception-handling.
- Ecologies of Data Work: The paper calls for recognizing dataset expertise, engaging archival, corpus-design, and human-computation disciplines, and investing in maintenance, rewards, and organizational processes.These recommendations address the underappreciation and specialized demands of dataset work.
- Ecologies of Data Work: Clearer methodological framing is needed because ML goals determine dataset requirements and justify which properties are measured.The paper distinguishes methods from methodologies and treats datasets as engineered artifacts supporting ML goals.
6 CONCLUSIONS
The conclusion presents a framework for operationalizing dataset transparency and accountability through discrete, interconnected lifecycle stages. It emphasizes stage ownership, expert consultation, detailed documentation, and greater recognition of dataset work.
- Conclusions: The framework assigns responsibility for discrete, interconnected development stages and requires owners to document what happens and why.The proposed accounts include requirements specifications, design documents, and testing reports.
- Conclusions: The approach adapts software development lifecycle practices by treating datasets as technical infrastructure and dataset development as engineering.It uses related fields to document all stages of dataset development.
- Conclusions: The paper argues that accountability requires valuing dataset work comparably to infrastructure and model-building work.It advocates recognition mechanisms that allow data experts to claim responsibility for datasets.
APPENDIX A: TEMPLATE FOR DATASET REQUIREMENTS SPECIFICATION
The dataset requirements specification template organizes decisions about a dataset’s purpose, sources, instances, distributions, processing, performance, maintenance, sharing, risks, and ethics. It also records ownership and versioning information.
- Dataset context: The template begins with the dataset name, ownership, dates, envisioned scope, motivation, intended uses, non-intended uses, terminology, related documents, and example instances.These fields establish the dataset’s context and boundaries.
- Requirements: It documents stakeholder consultation, creation requirements, sources, collection methods, instance requirements, acceptable tradeoffs, and distributional requirements.Distributional requirements include who should be represented and in what numbers.
- Data processing: Processing requirements specify annotation, filtering, annotator responsibilities, validation, and acceptable tradeoffs.The template asks how data quality should be validated.
- Operations and sharing: The specification asks what users can expect, how data will be maintained and retained, and how licensing, access, usage, and distribution will be constrained.It includes maintenance frequency and sharing requirements.
- Risks and ethics: It records caveats, risks, consequences of meeting the requirements, and ethical considerations surrounding the data and its collection.The template explicitly includes data ethics documentation.
APPENDIX B: TEMPLATE FOR DATASET DESIGN DOCUMENT
The dataset design document template captures design context, data characteristics, discovery, annotations, quality measurement, privacy, maintenance, sharing, caveats, ethics, and work estimates. It prompts reviewers to surface difficult design questions.
- Design review: The template asks reviewers to consider what questions they would ask about a dataset design and which questions they would dread being asked.This encourages explicit critical review of design choices.
- Dataset overview: It records the dataset name, owner, dates, high-level overview, primary data types, content, funding, objectives, version, and background.These fields establish provenance, purpose, and scope.
- Construction and quality: The design document describes sources, selection and sampling criteria, annotators, training, rating tasks, procedures, and quality measurement and validation.It links data construction choices with annotation and evaluation practices.
- Characteristics and related datasets: It compares related datasets and explains why they are unsuitable, while documenting the dataset’s expected characteristics, correlations, conjunctions, and represented population.These fields clarify both comparative context and intended data properties.
- Operations and risks: The template includes privacy handling, maintenance and recovery plans, sharing controls and licensing, caveats, ethics and mitigation, and estimated time and costs.Operational and ethical constraints are recorded alongside resource requirements.
APPENDIX C: TEMPLATE FOR DATASET TESTING REPORT
The Dataset Testing Report template records dataset ownership and timing, defines what is being tested, links requirements and design documents, and checks whether ongoing data needs remain relevant.
- The report identifies the dataset and records its owner, creation date, and last-updated date.
- It prompts authors to specify what is being tested.
- It links the testing report to the requirements specification and design document.
- It asks whether the data are still needed and whether data requirements remain relevant and up-to-date.