Source-linked AI summary
Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research
Bernard Koch, Emily Denton, Alex Hanna, Jacob G. Foster
TL;DR
Benchmark research has received limited study of how datasets are used and reused within and across machine learning task communities. Using Papers With Code data from 2015–2020, the paper analyzes these patterns and finds increasing concentration, cross-task dataset adoption, and institutional concentration in dataset origins.
Problem
Research has paid relatively little attention to benchmark dataset use and reuse across machine learning subcommunities, despite benchmarking’s central role in organizing research.
Method
The paper analyzes dataset creation, adoption, and usage across machine learning task communities and time using the Papers With Code corpus from 2015–2020.
Results
Across the field, dataset use increasingly concentrates on fewer benchmarks, many datasets cross task boundaries, and dominant datasets originate from a handful of elite institutions.
Takeaways & Limitations
The findings inform community-wide efforts concerning scientific rigor, ethical evaluation, and equity and access in machine learning research.
Takeaways & Limitations
The results depend on Papers With Code’s structure and coverage, including imperfect publication coverage and potentially subjective task annotations.
Abstract
from arXiv · showhide
Benchmark datasets play a central role in the organization of machine learning research. They coordinate researchers around shared research problems and serve as a measure of progress towards shared goals. Despite the foundational role of benchmarking practices in this field, relatively little attention has been paid to the dynamics of benchmark dataset use and reuse, within or across machine learning subcommunities. In this paper, we dig into these dynamics. We study how dataset usage patterns differ across machine learning subcommunities and across time from 2015-2020. We find increasing concentration on fewer and fewer datasets within task communities, significant adoption of datasets from other tasks, and concentration across the field on datasets that have been introduced by researchers situated within a small number of elite institutions. Our results have implications for scientific evaluation, AI ethics, and equity/access within the field.
1 Introduction
Benchmark datasets organize machine learning research by providing shared comparison points and signals of progress. This paper examines how their use varies across task communities and over time, finding increasing concentration, cross-task adoption, and institutional concentration.
- Benchmark datasets provide stable comparison points, coordinate researchers around shared problems, and signal collective progress.
- The paper studies dataset usage across machine learning task communities and from 2015–2020 using the Papers With Code corpus.
- Dataset use within most task communities is increasingly concentrated on fewer datasets.
- Most papers in most tasks use datasets originally created for other tasks, despite task communities creating more datasets than they import.
- The dominant datasets were introduced by researchers at a small number of elite institutions.
2 Related Work: Scientific, Social, and Ethical Importance of Datasets
Benchmarking coordinates machine learning research and lowers barriers to comparing contributions, but established benchmark cultures can privilege top-line performance over broader evaluation. Prior work therefore questions benchmark validity, real-world relevance, and representational coverage while motivating field-level analysis of dataset concentration and reuse.
- Scientific importance: Benchmarks formalize tasks through datasets and quantitative metrics, allowing researchers to evaluate models against shared resources.
- Social importance: Benchmarking reduces participation barriers and simplifies comparison by aligning the community around agreed measures of progress.
- Scientific and social concerns: SOTA requirements on established benchmarks can valorize narrow performance and create barriers to developing new ideas.
- Ethical and ecological concerns: Critical work identifies benchmark concerns including shortcuts, narrow demographic coverage, and uncertain suitability for real-world evaluation.
- Research questions: The paper asks how concentrated task communities are on specific datasets and whether researchers borrow datasets from other tasks.
- Contribution: The paper contributes the first field-level quantitative analysis of machine learning benchmarking practice.
3 Data
The study uses Papers With Code metadata to analyze dataset usage within and across task communities, distinguishing each dataset’s origin task from its destination task. It restricts analyses to usable, labeled, sufficiently large task communities and links dataset-introducing papers to institutional affiliations for the institutional analysis.
- Data source: Papers With Code is an open-source repository containing machine learning papers, datasets, and evaluation tables, with community and algorithmic curation.
- Data source: The dataset archive contains metadata including modality, introduction date, and the paper that introduced each dataset.
- Task relations: The study distinguishes a dataset’s origin task from the destination task of the paper using it, enabling analysis of cross-task transfer.
- Task relations: Transfer is recorded between origin and destination tasks and between their parent tasks to capture broad and fine-grained dynamics.
- Preprocessing and analysis samples: The analyses exclude unused datasets, noisy usages without shared task labels, and smaller parent tasks below the median size threshold.
- Institutional analysis: The institutional analysis links dataset-introducing papers to Microsoft Academic Graph and retains usages with annotated last-author affiliations.
4 Methods and Findings
The paper measures dataset concentration and cross-task reuse in the PWC corpus using Gini coefficients, descriptive proportions, and regression models. It finds increasing concentration over time, substantial adoption of datasets from other tasks, and strong concentration of dataset usage among elite institutions.
- Methods: Gini coefficients measure dataset-use concentration within tasks, ranging from 0 when usage is equal to 1 when one dataset is used exclusively.The analysis uses a sample-corrected Gini and excludes tasks or task-years with fewer than 10 papers.
- Methods and Findings: Model 1 predicts a 0.113 marginal increase in full-dataset Gini from 2015–2020 after controlling for task size, task age, and task-specific effects.By 2020, the median task Gini was 0.60; increases were attenuated for Natural Language Processing communities.
- Methods and Findings: The analysis used mixed-effects logistic regression for adoption and creation proportions, but lacked enough task-year events to identify temporal trends in either outcome.Disaggregated task-years with no events made these metrics undefined in many cases.
- Methods and Findings: More than half of all task communities use datasets from other tasks at least 57.8% of the time, rising to 71.9% in Computer Vision and 74.1% in Methodology.Natural Language Processing differs: half of its communities adopt datasets less than 27.4% of the time.
- Methods and Findings: Over 50% of dataset usages in PWC as of June 2021 came from twelve institutions, while institutional Gini concentration increased to above 0.80 in recent years.The same increasing concentration pattern appears for datasets across PWC more generally.
5 Discussion
Benchmark use is increasingly concentrated within task communities, while datasets also travel across tasks and originate disproportionately from elite institutions. These patterns raise concerns about scientific validity, ethical dataset use, and inequality, while motivating more diverse evaluation and equity-oriented interventions.
- Social stratification in MLR: More than 50% of 43,140 dataset usages corresponded to datasets introduced by twelve elite, primarily Western institutions.Globally, dataset-usage inequality is increasing when the corpus is examined without task disaggregation.
- Scientific rigor and ecological validity of MLR: Dataset concentration within task communities has increased over time, even though communities generally create more datasets than they import.Newly created datasets are used at lower rates than imported datasets.
- Scientific rigor and ecological validity of MLR: NLP communities moderate the broader concentration trend, create new datasets at higher rates, and use outside datasets at lower rates.The authors note that the reasons for this distinct pattern remain unresolved and suggest larger or more accessible NLP communities as possibilities.
- Scientific rigor and ecological validity of MLR: Concentration on established benchmarks is encouraged by the value placed on state-of-the-art performance, but over-concentration can distort measures of progress and intensify ethical risks in high-stakes applications.The discussion connects benchmark incentives to concerns about overfitting and biased facial-recognition evaluation.
- Scientific rigor and ecological validity of MLR: Only 7.49% of Image Generation papers evaluate on datasets developed for Image Generation, and only one heavily used dataset was designed specifically for that task.Cross-task adoption can support methodological advancement, but may leave datasets insufficiently aligned with new problem domains.
- Social stratification in MLR: Diversifying evaluation beyond top-line benchmark metrics and funding dataset creation at less-resourced institutions are proposed responses to concentrated usage and benchmark influence.The authors caution that rewarding dataset development alone may not diversify dataset usage or the perspectives shaping research agendas.
6 Conclusion
The paper examines benchmark creation, adoption, and usage within and across machine-learning task communities. It finds concentration in a small number of datasets and institutions.
- Benchmarking practices are heavily concentrated on a small number of datasets within each task community.
- Benchmarking practices are also heavily concentrated on datasets originating from a small number of well-resourced institutions across the field.
- The work empirically examines patterns of dataset creation, adoption, and usage within and across machine-learning task communities.
A.1 Sensitivity Analyses
The study uses three sensitivity analyses to assess potential biases from its corpus, curatorial decisions, and model selection or robustness.
- Three sensitivity analyses assess potential biases in the Papers With Code corpus.
- The analyses examine biases stemming from the authors’ curatorial decisions.
- The analyses also examine model selection and robustness, while identifying potential limitations that remain unaddressed.
Coverage biases in PWC
The findings depend on how well Papers With Code represents machine-learning research more broadly. Its coverage is incomplete but captures a large share of citations from papers at ten major conferences.
- 58.9% of 46,774 papers from ten top machine-learning conferences appeared in Papers With Code.
- Papers With Code covered 89.3% of the collective citations received by those 46,774 papers.
- Coverage ranged from 38.9% of 2015 papers to 68.8% of 2020 papers, while annual citation coverage never fell below 86% from 2015 onward.
- The analysis does not address possible under-annotation of dataset creations or usages within included Papers With Code papers.
Robustness to Cleaning Decisions
Robustness checks examine alternative task selections and find that concentration over time generally persists, though estimates vary with task granularity and small-sample concerns.
- Focus on larger, higher-level tasks: The analysis focuses on higher-level parent tasks to reduce double counting of dataset transfers and avoid potentially idiosyncratic smaller-task labels.
- Robustness to Cleaning Decisions: Including all 269 parent tasks preserves a significant increase in Gini over time for all but the smallest tasks.
- Robustness to Cleaning Decisions: Including parent and child tasks preserves the increase in concentration over time, but the authors discourage interpreting these findings because Gini is biased in small samples.
Cleaning of spurious annotations
The analysis addressed incomplete task annotations by manually labeling dataset-introducing papers and tested whether relaxing annotation requirements altered the findings. The main results remained stable under these checks.
- Two authors manually extracted task-label evidence for widely used datasets lacking annotations, including MNIST, CIFAR-10, CelebA, and ImageNet.
- 92,874 dataset usages from 46,697 dataset-using papers were scraped, with task labels available for 49,589 usages.
- Relaxing the overlapping-task requirement expanded Analysis 3 to 78,289 usages of 2,174 datasets in 46,842 papers, primarily using algorithmically labeled data.
- Including potentially noisy usages preserved the concentration findings: Gini increased from 0.31 to 0.86 across datasets and from 0.38 to 0.80 across institutions.
- The number of institutions accounting for more than half of usages shrank to nine under the relaxed annotation analysis.The listed institutions include Princeton, Stanford, Microsoft, AT&T, Max Planck, CUHK, Google, NYU, and Toyota Technical Institute at Chicago.
Robustness to Model Design Choices
The robustness analyses examined alternative concentration metrics, model specifications, corpus-growth confounding, and supplemental distributions. These checks generally reproduced the reported temporal concentration pattern.
- Using Entropy-based metric instead of Gini: Pielou evenness normalizes observed dataset-use entropy by maximum possible entropy, with higher values indicating greater evenness and lower values indicating concentration.
- Using Entropy-based metric instead of Gini: At 0.69, median Pielou evenness across parent tasks was fairly high, yet its regression results matched Gini's model specification, significant parameters, and trends.
- Model Selection for Regression 1: The model-selection analysis compared simpler specifications against a fully restricted model using Akaike and Bayesian information criteria.
- Robustness to corpus growth: Controlling for corpus growth with residuals from an exponential time regression produced an insignificant coefficient and did not affect the results.
- Supplemental figures: Figure A1 reports rising Gini concentration over time across task communities, while Figure A4 presents corresponding predicted Pielou evenness with 95% confidence intervals.
- Supplemental figures: Figure A2 truncates the usage distribution by dropping 3,760 datasets with fewer than five papers and eight datasets with more than 500 uses.
- Supplemental figures: Figure A3 documents the Papers with Code corpus used for the analyses, containing 137,510 papers as of June 2021.