Source-linked AI summary
OpenML-Python: an extensible Python API for OpenML
Matthias Feurer, Jan N. van Rijn, Arlind Kadra, Pieter Gijsbers, Neeratyoy Mallik, Sahithya Ravi, Andreas Müller, Joaquin Vanschoren, Frank Hutter
TL;DR
OpenML provides a platform for sharing machine-learning datasets and experiments, while programmatic access and reuse span the platform’s data and experiment records. OpenML-Python connects it to Python tools and supports conducting and sharing experiments; the paper presents it as a way to improve reproducibility, benchmarking, and collaboration.
Problem
OpenML’s datasets and experiment results need an accessible route into Python workflows so researchers can share and reuse them.
Source: §1 · pp. 1–2Method
Results
OpenML-Python provides access to OpenML data and experiments and supports sharing and reusing the data, metadata, and empirical results of ML studies.
Source: §6 · p. 4Takeaways & Limitations
The authors conclude that OpenML-Python allows better reproducibility, simpler benchmarking, and easier collaboration on ML projects.
Source: §6 · p. 4
Abstract
from arXiv · showhide
OpenML is an online platform for open science collaboration in machine learning, used to share datasets and results of machine learning experiments. In this paper we introduce OpenML-Python, a client API for Python, opening up the OpenML platform for a wide range of Python-based tools. It provides easy access to all datasets, tasks and experiments on OpenML from within Python. It also provides functionality to conduct machine learning experiments, upload the results to OpenML, and reproduce results which are stored on OpenML. Furthermore, it comes with a scikit-learn plugin and a plugin mechanism to easily integrate other machine learning libraries written in Python into the OpenML ecosystem. Source code and documentation is available at https://github.com/openml/openml-python/.
1. Introduction
OpenML supports sharing and building on empirical machine-learning research. OpenML-Python connects that platform to Python tools, simplifying access to its data and experiments and the sharing of new experiments.
1. Introduction
- OpenML is a collaborative online ML platform intended to support sharing and building on prior empirical research.Source: §1 · p. 1
- OpenML collects millions of shared experiments linked to exact pipelines and hyperparameter settings, with logging and uploading available through a REST API.Source: §1 · p. 2
- OpenML-Python integrates OpenML into the Python ML ecosystem, providing programmatic access to OpenML data and automating the sharing of new experiments.The paper presents its core design, extensibility to new ML libraries, and examples of common research tasks.Source: §1 · p. 2
2. Use cases for the OpenML-Python API
OpenML-Python supports common tasks for sharing and reusing machine-learning data and results. It provides dataset access and contribution tools, experiment publication and retrieval, and interfaces for integrating ML libraries.
Working with datasets
- Users can retrieve and filter OpenML datasets and metadata, convert datasets to numpy, scipy, or pandas structures, and upload datasets or define tasks.A task combines a dataset, train/test split, and target attribute.Source: §2 · p. 2
Publishing and retrieving results
- OpenML-Python can run experiments for a task and flow, then publish reproducible results including hyperparameter settings and random states.The interface supports integrating new ML libraries; scikit-learn is already integrated.Source: §2 · p. 2
Use cases in published works
- Published studies have used OpenML-Python for large dataset studies, meta-learning metadata, hyperparameter-importance analysis, and algorithm comparisons.The cited examples include studies by Feurer et al. (2015), Fusi et al. (2018), Perrone et al. (2018), van Rijn and Hutter (2018), and Strang et al. (2018).Source: §2 · p. 2
3. High-level Design of OpenML-Python
OpenML-Python maps OpenML’s machine-learning study entities to Python classes and supports offline use. Its extension interface standardizes how library code interacts with the package, with an implementation for scikit-learn.
High-level Design of OpenML-Python
- OpenML-Python represents OpenML entities as classes in separate submodules, mapping platform concepts to Python objects and facilitating offline usage.The described entities include datasets, tasks, flows, and runs; runs record evaluations of flows on tasks.Source: §3 · pp. 2–3
- The extension interface standardizes interaction between ML library code and OpenML-Python, and the paper provides an extension for scikit-learn.The scikit-learn extension can be used with libraries that follow the scikit-learn API.Source: §3 · p. 3
- An extension converts library models to OpenML flows, trains models, formats predictions, and creates runs for upload; it can also handle search procedures and upload their traces.The traces include hyperparameters and scores for models evaluated during search.Source: §3 · p. 3
4. Examples
The examples demonstrate retrieving and plotting SVM results and running a scikit-learn classifier across the OpenML-CC18 benchmark suite. The code also shows how a run can be uploaded.
4. Examples
- The Figure 1 example retrieves results and evaluations for an SVM on the ‘letter’ dataset and generates a contour plot of predictive accuracy.The accompanying caption identifies the plot as an SVM hyperparameter contour plot.Source: §4 · p. 3
- The example code includes an optional run upload, which requires an API key.Source: §4 · p. 4
- The Figure 2 example trains and evaluates a scikit-learn classification pipeline on each task in the OpenML-CC18 benchmark suite.The shown code builds a decision-tree pipeline and iterates over the suite’s tasks to run the classifier on their splits.Source: §4 · p. 4, Figure 2
5. Project development
OpenML-Python is developed publicly with community contributions, while continuous integration and automated checks support a coherent, robust code base.
5. Project development
- The project is developed publicly on GitHub, where an issue tracker supports bug reports, feature requests, and usage questions.Source: §5 · p. 4
- Continuous integration for Windows and Linux, plus automated type and style checking, helps maintain a coherent and robust code base.Source: §5 · p. 4
- Documentation combines tutorials, examples, and API documentation, and is rendered on continuous integration servers.Source: §5 · p. 4
- The package uses established third-party tools and scientific-computing libraries, is written in Python 3, and uses a 3-Clause BSD License.Documentation uses Sphinx, while scientific-computing packages include NumPy, SciPy, and pandas.Source: §5 · p. 4
6. Conclusion
OpenML-Python brings OpenML interaction into Python to support sharing and reuse of machine-learning data and results. It includes a scikit-learn extension, a mechanism for integrating other Python ML libraries, and project documentation and code examples.
6. Conclusion
- Sharing and reusing data, meta-data, and empirical results from ML studies supports better reproducibility, simpler benchmarking, and easier collaboration.Source: §6 · p. 4
- OpenML-Python includes a scikit-learn extension and a mechanism for integrating other machine-learning libraries written in Python.Source: §6 · p. 4
- The project hosts its source code on GitHub and provides documentation, a list of extensions, and code examples.Source: §6 · p. 4