Source-linked AI summary
Data and its (dis)contents: A survey of dataset development and use in machine learning research
Amandalynne Paullada, Inioluwa Deborah Raji, Emily M. Bender, Emily Denton, Alex Hanna
TL;DR
Benchmark datasets can make machine learning progress appear stronger than the underlying data and tasks warrant, while collection and reuse raise practical and ethical concerns. The paper surveys these issues and approaches to inspecting or modifying datasets, concluding that such interventions remain insufficient when datasets encode invalid tasks or spurious cues. It advocates more careful, contextualized, and ethically governed dataset practices.
Problem
Benchmark datasets can provide misleading tests of human-like reasoning, while data collection, management, reuse, and distribution raise practical and ethical concerns.
Method
The paper surveys recent literature on dataset design critiques, dataset-improvement methods, and broader practices concerning benchmarking, management, distribution, and reuse.
Results
The surveyed critiques identify representation biases, exploitable artifacts, spurious tasks, benchmarking overemphasis, and ethical, legal, and data-management concerns.
Takeaways & Limitations
The paper advocates carefully collected, contextually grounded datasets distributed with respect for intellectual-property and privacy rights and developed with relevant scholarly fields.
Takeaways & Limitations
Dataset modifications and bias-mitigation techniques are useful only when the dataset itself represents a valid task.
Abstract
from arXiv · showhide
Datasets have played a foundational role in the advancement of machine learning research. They form the basis for the models we design and deploy, as well as our primary medium for benchmarking and evaluation. Furthermore, the ways in which we collect, construct and share these datasets inform the kinds of problems the field pursues and the methods explored in algorithm development. However, recent work from a breadth of perspectives has revealed the limitations of predominant practices in dataset collection and use. In this paper, we survey the many concerns raised about the way we collect and use data in machine learning and advocate that a more cautious and thorough understanding of data is necessary to address several of the practical and ethical issues of the field.
1 Introduction
Datasets shape machine learning models, evaluation, and research agendas, but benchmark performance can rest on flawed dataset foundations. The survey argues that understanding dataset contents and development practices is necessary for addressing practical and ethical concerns.
- Datasets have been treated as limiting factors for algorithmic development and scientific progress, while benchmark datasets shape research goals and agendas.
- Reported ‘super-human’ benchmark performance may reflect shortcomings in datasets as tests of human-like reasoning.
- Large-scale web collection and non-expert annotation displaced slower expert practices while obscuring labor, judgments, biases, and context.
- Because deployed systems affect people’s lives and livelihoods, researchers and the public need to understand dataset contents and effects on performance.
- The survey reviews dataset design critiques, improvement methods, and broader concerns about benchmarking, data management, reuse, and dataset practices.
2 Definitions
The paper distinguishes benchmarks, tasks, capabilities, and datasets, emphasizing that datasets instantiate tasks through input-output pairs. Tasks may target practical use cases, human-like capabilities, or both.
- A task comprises input and output spaces plus an expected mapping, with intensional and extensional definitions.
- Datasets exemplify tasks as input-output pairs that conform, when valid, to the task’s intended mapping.
- Tasks can matter because they map to use cases or illustrate human cognitive capabilities targeted for machine programming.
3 Dataset design and development
Dataset design and development practices create problems of representation, artifacts, task validity, ethics, and reproducibility. These concerns span collection, annotation, documentation, and the assumptions that guide what datasets measure.
- Representation: Prominent datasets under-represent darker-skinned subjects and draw object-recognition images disproportionately from Western countries.
- Spurious correlations: Models can exploit shortcuts and incidental associations that perform poorly out of distribution rather than exhibiting human-like reasoning.
- Task framing: Collection decisions can formulate unjust or pseudoscientific tasks, including attempts to predict IQ from short-answer texts.
- Collection: Web scraping has included non-consensual pornography and personal photographs used without subjects’ knowledge.
- Annotation: Annotator judgments and retrieval procedures can bias labels, as illustrated by ImageNet’s mismatch with intended object-recognition tasks.
- Documentation: Different inter-annotator agreement thresholds produced substantially different reconstructed ImageNet datasets, raising reproducibility concerns.
4 Filtering, augmenting, and other twists on datasets
The surveyed methods use audits, statistical tools, perturbations, counterfactual data, and bias mitigation to inspect or improve datasets. However, these interventions remain limited when datasets themselves encode spurious cues or invalid tasks.
- Dataset inspection: Statistical tools can help locate spurious cues in massive datasets, while manual audits and domain expertise remain important for discovering unexpected contents.
- Perturbation and augmentation: Inoculation by fine-tuning interprets failures on perturbed inputs, while manually constructed counterfactuals aim to improve generalization.
- Limitations: VQA-CP shows that redistributing examples can create a perturbed dataset that remains solvable through randomly generated answers.
- Bias mitigation: Bias mitigation methods include counterfactually guided objectives, dataset-bias unlearning, and evaluation that reflects variable human judgment.
- Limitations: Procedural modifications and bias mitigation help only when the dataset represents a valid task.
5 Dataset culture
The paper examines dataset culture beyond technical construction, focusing on how benchmarking, reuse, storage, distribution, and legal practices shape datasets’ meanings and consequences. It argues that datasets must be understood through their relationships with research communities, data subjects, and legal institutions.
- Benchmarking practices: Benchmarking culture can prioritize leaderboard performance while obscuring what high scores actually demonstrate about model capabilities.Critiques call for stronger empirical rigor, negative results, disaggregated metrics, and error analysis alongside standard benchmark scores.
- Data management and distribution: Sensitive human-derived data require secure storage, appropriate dissemination, and stronger governance, but machine learning lacks established practices for managing such data.Researchers have distributed biometric datasets without consent, privacy policies, or effective controls, and derivative copies can persist after removal.
- Use and reuse: Reusing data outside its original context can create privacy and ethical problems, as illustrated by the Pima Indians Diabetes Dataset and surveillance-derived datasets.The PIDD became a widely reused toy classification task, while DukeMTMC involved openly distributed surveillance footage collected without consent.
- Legal issues: Benchmark datasets also raise legal questions across collection, annotation, training, evaluation, and downstream reuse because sources carry different permissions and copyright conditions.Open licensing does not resolve privacy, informed-consent, or opt-out concerns, and proposed licensing alternatives do not address scraped-web copyright or privacy implications.
- Dataset use and reuse: Machine learning datasets often abstract away the human labor, judgments, biases, and contexts involved in their creation and use.The paper frames datasets as culturally enacted objects rather than singular technical artifacts.
6 Conclusion
The survey finds that dataset-centered machine learning culture intertwines scientific and ethical concerns across representation, artifacts, task validity, documentation, benchmarking, reuse, and law. It advocates careful, contextualized dataset practices rather than defining progress primarily through benchmark performance.
- Dataset design and development: Dataset critiques identify representation bias, exploitable artifacts, spurious tasks, and inadequate annotation or documentation as foundational pitfalls.These problems can undermine scientific inquiry and allow models to succeed without the intended real-world capability.
- Dataset design and development: Adversarial or challenge datasets may remove some spurious cues but leave most problems in the original datasets and research paradigm unresolved.
- Dataset culture: Benchmark overemphasis excludes other evaluation practices, domain-connected modeling, and approaches not optimized for the available benchmark set.The survey also links dataset management, reuse, and distribution to care for data subjects, contextual risks, and legal vulnerabilities.
- Paths forward: Carefully designed datasets can make patterns of injustice visible, while educational data-collection interventions can build technical literacy for self-advocacy.
- Paths forward: The authors advocate smaller, more carefully collected and contextualized datasets that respect intellectual-property and privacy rights and model realistic capabilities.They predict this shift would encourage approaches beyond scale and improve the field’s ability to design equitable systems for deployment contexts.