Source-linked AI summary
DataPerf: Benchmarks for Data-Centric AI Development
Mark Mazumder, Colby Banbury, Xiaozhe Yao, Bojan Karlaš, William Gaviria Rojas, Sudnya Diamos, Greg Diamos, Lynn He, Alicia Parrish, Hannah Rose Kirk, Jessica Quaye, Charvi Rastogi, Douwe Kiela, David Jurado, David Kanter, Rafael Mosquera, Juan Ciro, Lora Aroyo, Bilge Acun, Lingjiao Chen, Mehul Smriti Raje, Max Bartolo, Sabri Eyuboglu, Amirata Ghorbani, Emmett Goodman, Oana Inel, Tariq Kane, Christine R. Kirkpatrick, Tzu-Sheng Kuo, Jonas Mueller, Tristan Thrush, Joaquin Vanschoren, Margaret Warren, Adina Williams, Serena Yeung, Newsha Ardalani, Praveen Paritosh, Lilith Bat-Leah, Ce Zhang, James Zou, Carole-Jean Wu, Cody Coleman, Andrew Ng, Peter Mattson, Vijay Janapa Reddi
TL;DR
DataPerf addresses the limited attention given to dataset quality and the saturation of fixed, model-centric benchmarks. It introduces a suite and platform for evaluating data-centric algorithms across diverse tasks, with standardized comparisons focused on dataset changes. The initial release comprises five benchmarks and open-source infrastructure, while early challenge results favored automated data-selection and augmentation strategies.
Problem
ML research has focused more on models than datasets, while fixed benchmarks can saturate and leave dataset breadth, difficulty, and fidelity underexamined.
Method
DataPerf freezes model architectures, training hyperparameters, and task metrics while benchmarking dataset changes across diverse data-centric tasks.
Results
The initial DataPerf release comprises five benchmarks with distinct rules, evaluation methods, baseline implementations, and an open-source evaluation platform.
Takeaways & Limitations
DataPerf supports systematic dataset development by adding rigor to training- and test-set quality assessment and enabling continued community expansion.
Takeaways & Limitations
The data-centric challenge was computationally expensive, with power, memory, and bandwidth identified as major limitations.
Abstract
from arXiv · showhide
Machine learning research has long focused on models rather than datasets, and prominent datasets are used for common ML tasks without regard to the breadth, difficulty, and faithfulness of the underlying problems. Neglecting the fundamental importance of data has given rise to inaccuracy, bias, and fragility in real-world applications, and research is hindered by saturation across existing dataset benchmarks. In response, we present DataPerf, a community-led benchmark suite for evaluating ML datasets and data-centric algorithms. We aim to foster innovation in data-centric AI through competition, comparability, and reproducibility. We enable the ML community to iterate on datasets, instead of just architectures, and we provide an open, online platform with multiple rounds of challenges to support this iterative development. The first iteration of DataPerf contains five benchmarks covering a wide spectrum of data-centric techniques, tasks, and modalities in vision, speech, acquisition, debugging, and diffusion prompting, and we support hosting new contributed benchmarks from the community. The benchmarks, online evaluation platform, and baseline implementations are open source, and the MLCommons Association will maintain DataPerf to ensure long-term benefits to academia and industry.
1 Introduction
DataPerf responds to saturated, model-centric benchmarking by making dataset improvement itself the object of competition and evaluation. It offers a broad, extensible benchmark and platform infrastructure for data-centric research.
- DataPerf motivation and approach: Existing benchmarks often rely on fixed datasets whose breadth, difficulty, and fidelity are insufficiently examined, while benchmark saturation raises concerns about genuine capability progress.The paper links data quality to generalization, bias, safety, and persistent fairness issues in deployed systems.
- DataPerf motivation and approach: DataPerf benchmarks dataset improvement rather than only model design, freezing architectures, hyperparameters, and metrics to isolate changes to the data.Its benchmarks cover major stages of data-centric development while comparing solutions through relative dataset-driven improvements.
- DataPerf scope: The initial suite spans training-set selection, data cleaning and debugging, data acquisition, and diffusion-model prompting across vision and speech tasks.Each benchmark is tied to a real-world use case and includes submission rules, evaluation scripts, and a baseline.
- DataPerf infrastructure: DataPerf provides an extensible open-source platform that lets researchers and organizations host additional data-centric benchmarks and challenges.A working group under the MLCommons Association maintains the benchmarks and platform beyond a single competition.
2 DataPerf Benchmarking Suite
DataPerf reframes benchmarking around data-centric development, using competitive, reproducible challenges to evaluate dataset improvement across multiple ML tasks. Its initial suite spans selection, cleaning, debugging, acquisition, and prompting, while lessons from an earlier competition motivate automated methods, novel optimizations, and supporting infrastructure.
- The Data-Centric AI Challenge: Iterating on datasets with a fixed model increased DCAI classification accuracy from 64.4% to 85.8%, approaching the 90.2% human-level performance.The competition used Roman-numeral digit classification and attracted 2,500 submissions.
- The Data-Centric AI Challenge: Automated data-selection and augmentation strategies outperformed the expected focus on discovering and correcting labeling problems in the earlier competition.Successful tactics also included identifying noisy or mislabeled images, correcting class imbalance, and enhancing long-tail examples.
- The Data-Centric AI Challenge: The initial competition exposed infrastructure constraints because data-centric evaluation can require substantial computation, memory, storage, and bandwidth.The authors identify more efficient training on user-submitted data as necessary for this competition format.
- Challenges, Benchmarks, and Leaderboards: DataPerf combines public challenges and leaderboards with an extensible platform so researchers can compare data-centric solutions and propose new benchmarks.Each challenge supplies submission rules, evaluation scripts, and a baseline, while Dynabench hosts the online evaluation infrastructure.
- Challenges, Benchmarks, and Leaderboards: DataPerf covers training-set selection, data cleaning and debugging, data acquisition, and generative-model prompting across speech and vision tasks.The suite was selected to cover underserved data-centric pipeline components and exercise the online platform infrastructure.
3 Related Work
DataPerf builds on data-centric benchmarking and competition efforts while extending them through an open platform for contributed benchmarks and persistent comparison.
- DataPerf draws inspiration from data-centric benchmarks and competitions spanning AutoML, semi-supervision, data selection, data cleaning, multimodal filtering, and adversarial example discovery.
- Unlike fixed-purpose prior efforts, DataPerf solicits user-contributed benchmarks through an extensible hosting platform.
- Persistent online leaderboards let new data-centric solutions be compared with prior submissions.
- The baseline implementations are intentionally non-exhaustive, encouraging both newcomers to apply existing methods and researchers to propose novel methods.
- The working group plans to add challenges and integrate benchmarks in areas including tabular active learning, label uncertainty, and noisy annotations.
4 Statement of Ethics
The ethics discussion identifies privacy and participant-well-being risks associated with benchmark registration, artifact sharing, and sensitive imagery.
- Dynabench registration collects usernames and email addresses that may correspond to personally identifiable information.
- Uploaded submission artifacts may optionally be viewed by other users as open benchmark results.
- Adversarial Nibbler collects potentially sensitive harmful or disturbing depictions that may negatively affect participants and raters.
- Adversarial Nibbler therefore requires participant guidelines covering preparation, organizer communication, and psychological-support resources.
5 Conclusion and Future Work
DataPerf expands ML research from model improvement to dataset improvement through five benchmarks and an open, extensible evaluation platform, with planned community growth and hidden-set evaluation.
- DataPerf aims to improve machine learning by extending research from models to models and datasets, with systematic benchmarking of dataset quality.
- Its initial release contains five benchmarks with distinct rules, evaluation methods, baseline implementations, and an open-source evaluation platform.
- DataPerf plans to expand through community-contributed benchmarks and a Closed Division that evaluates algorithms on hidden training sets.
A.1 Terminology for Training Sample Selection
The terminology distinguishes individual training samples and the full allowed pool from the submitted subset selected to form a coreset.
- A training sample is an individual data point in a dataset, while training set selection chooses a small training set from a larger potentially noisy pool.
- Training IDs are integer enumerations or unique strings corresponding to files for individual samples.
- Allowed training IDs comprise the full list of potential samples from which participants may form subsets.
- Selected training IDs are the submitted subset drawn from the allowed training IDs and represent the concretized coreset evaluated online.
A.2 Reproducibility
DataPerf provides open-source challenge code, documentation, baselines, and fixed references to support reproducible benchmark participation. Challenge repositories preserve static competition versions while allowing owners to maintain their projects.
- A.2 Reproducibility: Challenge repositories provide code, documentation, evaluation resources, and baseline implementations for the inaugural DataPerf tasks.The repositories cover speech selection, vision selection, vision debugging, data acquisition, and Adversarial Nibbler.
- A.2 Reproducibility: DataPerf fixes each challenge to a parent-repository commit hash, preserving static competition conditions while retaining repository-owner control.The approach also supports visibility within challenge owners’ GitHub organizations.
- A.2 Reproducibility: Extended challenge documentation is reproduced from source repositories as of August 2023 to provide a fixed reference for requirements and evaluation structure.The paper notes that future selection and debugging challenges may diverge from these technical specifications.
A.3 Selection for Speech
The speech-selection benchmark chooses effective samples from a noisy multilingual corpus for low-resource keyword spotting. It evaluates selected training sets across languages using repeated model training and macro F1.
- A.3 Selection for Speech: Figure 4 reports training and evaluation sample counts for target keywords and nontarget data across the benchmark’s three languages.The target evaluation samples were manually verified for correctness.
- A.3 Selection for Speech: Evaluation trains ten models with different random seeds and reports macro F1 across target and nontarget words for each language.Participants train a six-category model using a maximum of 25 or 60 samples from the training pool.
A.4.1 User Interface and Prompt Types
Adversarial Nibbler collects prompts through an interface that generates images, allowing participants to submit either benign or intentionally subversive prompts. The interface supports exploration of unsafe text-to-image model behavior.
- A.4.1 User Interface and Prompt Types: Participants type prompts into the interface and generate images for their submissions.The interface is used to collect prompts for the Adversarial Nibbler challenge.
- A.4.1 User Interface and Prompt Types: Participants may submit ostensibly benign prompts or intentionally subversive prompts designed to produce unsafe images.Benign prompts may contain biased language without being prima facie intended to produce unsafe images.
A.4.2 Ethics and Instructions for Participants
Adversarial Nibbler combines participant guidance, protected validation, and two evaluation metrics for crowdsourced unsafe-prompt discovery. Its rules permit broad experimentation while limiting exposure, system load, and misuse.
- A.4.2 Ethics and Instructions for Participants: The challenge supplies participation guidelines, FAQs, communication channels, preparation advice, and psychological-support resources for potentially disturbing image generations.Participants are advised to split work into shorter sessions, communicate with teammates, and take breaks.
- A.4.3 Validation of Submissions: Participants do not validate one another’s images; trained raters perform all validation with access to additional resources.This arrangement is intended to reduce potential harms and stress from viewing others’ images and prompts.
- A.4.3 Validation of Submissions: Submissions receive Model Fooling Score and Prompt Creativity Score evaluations.The two metrics assess adversarial effectiveness and the breadth of participants’ exploratory strategies.
- A.4.3 Validation of Submissions: Model Fooling Score measures the quantity and severity of successful safety-related attacks after checking prompt safety and prompt-image unsafety.The validation verifies both that the prompt appears safe and that the resulting image together with the prompt is unsafe.
- A.4.3 Validation of Submissions: Prompt Creativity Score uses the participant’s submission set relative to the full dataset to reward different attack strategies and unsafe-image types.The supplied passage identifies the score as a composite calculated at the end of the competition.
- A.4.4 Rules for the Competition: Each participant or team needs a free DynaBench account, must submit its DynaBench name, and must license generated images permissively.The rules require a permissive license such as CC-BY for images created in the challenge.
- A.4.4 Rules for the Competition: Participants are limited to 50 image-generation sets daily, and suspected noncompetition use can trigger manually reviewed removal and account suspension.The daily limit is intended to prevent system overload and encourage creativity in attack strategies.
- A.4.4 Rules for the Competition: Competition rules permit participants to test their own models, discuss submissions, use language models, and seek others’ safety judgments.Participants may use external resources to explore text-to-image model failures.