Source-linked AI summary
OpenML Benchmarking Suites
Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Pieter Gijsbers, Frank Hutter, Michel Lang, Rafael G. Mantovani, Jan N. van Rijn, Joaquin Vanschoren
TL;DR
Machine learning research needs benchmarks that are interpretable, reproducible, and directly comparable, but existing studies lack standardized shared suites. The paper introduces an OpenML-integrated layer for creating and using curated task collections, and presents OpenML-CC18. Benchmark suites have been adopted across diverse research directions and can support large-scale studies, while fixed suites may increasingly risk overfitting.
Problem
Machine learning studies lack standardized benchmarks that are easily comparable, reproducible, and informative across individual studies.
Method
The paper introduces an OpenML-integrated benchmarking layer and tools for creating, using, sharing, and organizing curated collections of evaluation tasks.
Results
Benchmarking suites are being used to facilitate very diverse directions of machine learning research, with the AutoML benchmark providing a reviewed use case.
Takeaways & Limitations
Shared benchmarking suites can support a continuous community process for improving benchmarks and enable large-scale studies beyond what single studies can teach.
Takeaways & Limitations
Fixed benchmarking suites may increasingly encourage overfitting, motivating periodic updates with new datasets and harder tasks.
Abstract
from arXiv · showhide
Machine learning research depends on objectively interpretable, comparable, and reproducible algorithm benchmarks. We advocate the use of curated, comprehensive suites of machine learning tasks to standardize the setup, execution, and reporting of benchmarks. We enable this through software tools that help to create and leverage these benchmarking suites. These are seamlessly integrated into the OpenML platform, and accessible through interfaces in Python, Java, and R. OpenML benchmarking suites (a) are easy to use through standardized data formats, APIs, and client libraries; (b) come with extensive meta-information on the included datasets; and (c) allow benchmarks to be shared and reused in future studies. We then present a first, carefully curated and practical benchmarking suite for classification: the OpenML Curated Classification benchmarking suite 2018 (OpenML-CC18). Finally, we discuss use cases and applications which demonstrate the usefulness of OpenML benchmarking suites and the OpenML-CC18 in particular.
1 Introduction
Machine learning benchmarks should use challenging, well-designed tasks and standardized evaluations so results are interpretable, reproducible, and directly comparable. The paper addresses OpenML’s lack of simple tools for creating and sharing such benchmark suites by introducing an integrated benchmarking layer and the OpenML-CC18 suite.
- Benchmarks help researchers track progress, identify challenges, raise standards, and learn how to improve algorithms.
- Rigorous benchmarking requires well-designed task sets, standardized evaluations, and results that are easy to find, interpret, reproduce, and compare.
- OpenML tasks encode datasets, task types, targets, evaluation procedures, splits, and performance metrics to support reproducible evaluation schemes.
- Before this work, OpenML did not facilitate simple creation and sharing of well-designed benchmark suites or their experimental results.
- The paper introduces an OpenML-integrated benchmarking layer for creating, importing, running, sharing, searching, reusing, and comparing suites and results.
- The contributions include advocating curated task suites, providing creation and usage tools, proposing OpenML-CC18, reviewing an AutoML benchmark, and discussing research impact.
2 A Brief History of Benchmarking Suites
Existing dataset repositories and benchmarking platforms provide resources and protocols, but machine learning studies still lack standardized benchmarks that are readily comparable. This gap contributes to small, difficult-to-reproduce studies and contradictory findings.
- Repositories such as UCI, LIBSVM, UCR, Mulan, Kaggle.com, and PMLB provide broad or specialized datasets, sometimes with programmatic access.
- OpenAI Gym, COCO, ASLib, and the Ludwig Benchmarking Toolkit connect datasets, tasks, environments, or protocols to reproducible algorithm evaluation.
- Standardized benchmarks remain absent across individual studies, leading to small-scale experiments that can be difficult to reproduce and may produce contradictory results.
- Benchmarking often emphasizes outperforming the state of the art on a few datasets rather than rigorous large-scale analyses that include negative results.
3 OpenML
OpenML is a collaborative platform for sharing datasets, models, and experiments through machine-readable building blocks. Its website supports browsing benchmark studies and interactively exploring benchmark results.
- OpenML lets users share datasets and import them for sharing models and experiments run on those datasets.
- OpenML organizes resources around machine-readable data, tasks, flows, and runs, with tasks specifying evaluation details and runs storing experiment results.
- The AutoML benchmark suite was created using the technology described in this paper and its earlier preprint.
- Figure 1 shows benchmark studies on the OpenML website and interactive exploration of AutoML Benchmark results.
- A task specifies the dataset, task type, target feature, evaluation procedure, splits, and target performance metric.
4 OpenML Benchmarking Suites
OpenML benchmarking suites are curated collections of tasks selected under precise conditions to make algorithm evaluations comparable and reproducible. They support accessible sharing, provenance, reuse, collaboration, and large-scale benchmark studies.
- An OpenML benchmarking suite is a set of OpenML tasks carefully selected to evaluate algorithms under a precise set of conditions.
- Using tasks rather than datasets makes experiments comparable and reproducible.
- OpenML hosts thousands of datasets that scientists can filter when creating benchmarks, and each suite receives a unique ID for API and webpage access.
- Suites provide provenance and permanence because the creator is identified and others cannot edit or remove the suite.
- Curated suites let scientists benchmark methods without selecting datasets themselves and allow existing suites to be extended, subsetted, or adapted.
- Machine-readable tasks with evaluation instructions and train-test splits make shared results comparable and reproducible, while associated runs support new benchmark studies.
5 How to Use OpenML Benchmarking Suites
OpenML benchmarking suites support creating, retrieving, running, and sharing standardized benchmark studies through OpenML’s APIs, client libraries, and machine-readable tasks.
- Creating and organizing suites: OpenML adds benchmark suites as task collections and benchmark studies as collections of results obtained on those tasks.
- Creating and organizing suites: Researchers can specify dataset and task constraints through OpenML filters, scripts, notebooks, and quality-assessment tools when creating suites.The workflow can iteratively refine constraints, visualize selected-task distributions, and retrieve existing results.
- Retrieving existing suites: Existing suites can be downloaded by unique ID or alias through OpenML client libraries, with uniformly formatted tasks, datasets, and metadata.Machine-readable metadata can indicate properties such as missing values for automated handling.
- Running benchmarks: After retrieving tasks, researchers can run experiments locally through integrations with scikit-learn, mlr, mlr3, and Weka, or wrap custom code.Deep-learning integrations were under development at the time described.
- Sharing results: Experiment runs can be bundled into benchmark studies and published with structured hyperparameter details, enabling uniform comparison and analysis across algorithms.The Python example creates and shares a benchmark study; uploading requires a free API key.
6 OpenML-CC18
OpenML-CC18 is a practical classification benchmark of 72 carefully curated tasks selected from OpenML using reproducibility, practicality, and difficulty criteria. The authors acknowledge selection mistakes and plan dynamic updates to limit benchmark overfitting.
- Suite overview: OpenML-CC18 provides a standard of 72 classification tasks built from a carefully curated selection of OpenML datasets.The suite is intended as a drop-in replacement for many typical benchmarking setups and focuses on medium-sized datasets.
- Selection criteria: The suite selects verified, publicly licensed datasets meeting explicit requirements for dataset size, feature count, class structure, imbalance, sparsity, and sampling.It excludes tasks requiring time-dependent or grouped sampling to simplify evaluation with standard procedures.
- Selection criteria: OpenML-CC18 removes artificial, subset-derived, target-transformed, and source-free datasets to improve interpretability and avoid problematic benchmark data.The authors acknowledge that distinguishing artificial from simulated datasets is imperfect.
- Difficulty criteria: Datasets that are too easy for a single attribute, decision stump, or decision tree are excluded because they do not meaningfully compare algorithms.A decision tree achieving 100% accuracy on a 10-fold cross-validation task is used as an indicator of an overly easy dataset.
- Limitations and updates: The initial suite contains acknowledged mistakes, including merged classes, time-related features, an artificial dataset, and writer-dependent grouping requirements.The authors plan to correct these issues in new versions and screen newer OpenML datasets.
- Usage and impact: OpenML-CC18 has supported diverse research directions, including missing-value imputation, uncertainty quantification, and meta-feature learning.Other work found both easy and hard datasets in the suite and concluded that it is not very challenging as a whole.
7 Further OpenML Benchmarking Suites
Further OpenML benchmarking suites extend or adapt the benchmarking approach for AutoML and other data types. The AutoML benchmark shares some OpenML-CC18 criteria but relaxes constraints to accommodate additional data complexity and is regularly updated.
- AutoML benchmark: The AutoML benchmark evaluates AutoML tools reproducibly on OpenML tasks or suites through code that runs experiments from the command line.
- AutoML benchmark: The AutoML benchmark combines OpenML-CC18 datasets with datasets from AutoML competitions and earlier AutoML comparisons, although no step-by-step recreation list exists.General recreation guidelines are provided.
- Design differences: Compared with OpenML-CC18, the AutoML benchmark permits larger datasets, more features, arbitrary class imbalance, sparse data, and regression problems.These relaxations reflect the assumption that AutoML tools should handle additional complexities.
- Design differences: The AutoML benchmark applies stricter context-specific filters, including limiting image-classification problems because its focus is tabular AutoML tools.Its easy-dataset filter also uses results across various learners.
- Reuse and adaptation: Researchers have reused AutoML benchmark suites unchanged or modified them by adding datasets, omitting computationally difficult tasks, or changing evaluation procedures.Examples include combining suites, adding UCI and Kaggle datasets, and replacing suite-defined 10-fold cross-validation with hold-out evaluation.
- Other suites: OpenML also hosts suites such as OpenML100-friendly and Foreign Exchange data for machine learning research.
- Future directions: The authors invite suites for larger, high-dimensional, imbalanced, noisy, text, time-series, and other data types to standardize evaluation across subfields.They also intend to create new suites and simplify their creation by others.
8 Limitations and Future Work
The paper identifies limitations involving benchmark overfitting, curation, computational demands, suite breadth, resource constraints, and ethical disclosure. It proposes updating suites, improving curation, balancing completeness with computation, supporting broader datasets, specifying resources, and adding native ethical-issue documentation.
- Overfitting: Benchmark suites may increasingly invite overfitting as more methods are evaluated on fixed task collections.The authors state this risk has not yet been demonstrated.
- Credit Assignment: Curating suites requires substantial manual effort, creating a need to credit contributors, potentially by making suites citable.OpenML-CC18 datasets were manually inspected and corrected over multiple months.
- Automating the curation of useful suites: Automated creation of high-quality, diverse, realistic suites remains an open research question because curation combines objective and subjective constraints.The authors also call for post-hoc validity checks to become applicable during suite design.
- Computational issues: Some OpenML-CC18 datasets impose excessive computational loads, motivating representative subsets or more careful completeness–cost trade-offs.The suite deliberately focused on mid-size datasets, yet some larger datasets remained expensive.
- Breadth of current benchmarking suites: Current suites may not cover the larger datasets and deep learning models spanning domains such as language and vision.The authors are developing ways to enable broader suites and invite community involvement.
- Specification of resource constraints: Task and suite specifications do not yet encode memory or time limits, weakening comparability when experiments use different resources.Identical hardware or general resource constraints could improve interpretation and comparability.
- Disclosure of ethical issues: The paper currently encourages ethical-issue disclosure in suite descriptions but plans native support at dataset and suite levels.Proposed support includes datasheets and a dedicated information field.
9 Conclusion
The paper aims to simplify creating, sharing, running, and comparing benchmark studies through an OpenML benchmarking layer and the OpenML-CC18 classification suite. Its broader conclusion is that community feedback and reuse can support evolving suites and larger-scale studies.
- Conclusion: The OpenML benchmarking layer lets researchers create, publish, explore, download, share, compare, and analyze results in one place with few lines of code.The layer is integrated with OpenML and supports general classification benchmarking through OpenML-CC18.
- Conclusion: OpenML-CC18 is a classification benchmark suite created with the new tools.It is presented as a benchmark for general classification benchmarking.
- Conclusion: Reviewing AutoML benchmark use and other adoptions indicates that continuous conversation with the research community is essential for evolving useful suites.The paper hopes this evolution will support studies that teach more than any single study.
A OpenML-CC18 dataset list
The OpenML-CC18 dataset list is documented through Table 1 and linked OpenML resources. The table records identifiers, dataset names, class and feature counts, observation counts, and class-imbalance ratios.
- A OpenML-CC18 dataset list: For each dataset, Table 1 reports the OpenML task ID, dataset ID and name, number of classes, features, observations, and MinMaj ratio.MinMaj is the ratio of minority to majority class sizes.
- A OpenML-CC18 dataset list: Online documentation, benchmark documentation, GitHub repositories, and Python, R, and Java packages provide access to OpenML benchmarking materials.The listed resources include documentation and implementation packages.
C Suggested curation protocol
The proposed curation protocol combines automated eligibility checks with manual review to construct benchmarking suites. It is a flexible guideline based on the authors’ experience with OpenML-CC18 and OpenML100.
- Suggested curation protocol: The protocol begins by defining the benchmark purpose and selecting which curation steps to include.Steps may be removed or added depending on the desired benchmark purpose.
- Steps that can be automated: Automated checks specify task type, dataset properties, modalities, sparsity, missing-value policy, and whether tasks are too easy.Dataset properties include size, feature count, and number of classes.
- Steps that can be automated: Task difficulty can be assessed by querying existing OpenML results or by running machine learning algorithms locally.This check is included among the protocol’s automatable steps.
- Steps that cannot be automated: Manual review checks for artificial, grouped or time-aware, subset, derived, and inadequately referenced datasets.These checks are applied to the outcomes of the automated filtering steps.