Source-linked AI summary
OpenML: networked science in machine learning
Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, Luis Torgo
TL;DR
Machine learning research produces detailed data, code, experiments, and models that papers do not adequately capture or organize for reuse. The paper introduces OpenML, an online service for sharing, structuring, and reusing these materials, with the goal of improving research collaboration and productivity. OpenML is presented as enabling broader reuse, new discoveries, greater visibility, and collaboration at scale, while dynamic data can limit repeatability in some tasks.
Problem
Machine learning results are usually published in summarized papers, leaving detailed data, code, evaluations, and models unorganized for reuse, reinterpretation, or mining.
Method
OpenML shares and organizes machine learning data, code, tasks, runs, models, predictions, and evaluations online for access, reuse, comparison, and discussion.
Results
OpenML supports more productive research by enabling reuse of shared results, rapid analysis of combined data, direct comparison with comparable work, and broader collaboration and visibility.
Takeaways & Limitations
OpenML brings networked-science practices to machine learning so researchers can collaborate globally, build on shared work, and tackle harder problems.
Takeaways & Limitations
Some dynamic-data tasks, such as Twitter feeds, do not produce repeatable results, and repeatability is not expected for those tasks.
Abstract
from arXiv · showhide
Many sciences have made significant breakthroughs by adopting online tools that help organize, structure and mine information that is too detailed to be printed in journals. In this paper, we introduce OpenML, a place for machine learning researchers to share and organize data in fine detail, so that they can work more effectively, be more visible, and collaborate with others to tackle harder problems. We discuss how OpenML relates to other examples of networked science and what benefits it brings for machine learning research, individual scientists, as well as students and practitioners.
1. INTRODUCTION
The internet enables networked science by allowing researchers to share detailed data and code, interact globally, and build directly on one another’s work. Online tools can organize and mine information beyond journals, although they must preserve the journal system’s functions for shared data.
- The internet enables researchers to share detailed observations, methods, data, and code beyond what journals can print.
- Online tools support global, real-time interaction among many researchers.
- Networked science lets researchers build on observations and techniques, reuse them, mine collective data, and scale collaborations to harder problems.
- The internet increasingly acts as a collective short-term working memory, while journals remain the collective long-term memory.
- Online scientific tools must ensure that shared data preserves functions associated with the journal system.
2. NETWORKED SCIENCE
Networked science organizes shared ideas, data, code, and experiments so many people can contribute, reuse information, and discover questions or solutions that individual researchers could not easily address. Its effectiveness depends on scalable collaboration and incentives that recognize contributions.
- Networked science structures information from many scientists into a coherent whole and enables broad reuse of collected data.
- Designed serendipity: Designed serendipity arises when shared ideas, questions, observations, or tools reach people with complementary expertise.
- Dynamic division of labor: Dynamic division of labor lets researchers with different skills or resources handle tasks suited to them, speeding progress.
- Scalable online collaboration encourages small contributions, divides complex problems into subtasks, and builds a structured information commons.
- Networked tools need reputation systems that show who contributed what and connect valuable reuse with authorship, citations, or esteem.
- Open data: Open data can support new discoveries by allowing combined experiments and large datasets to be queried for questions that were previously difficult to answer.
3. MACHINE LEARNING
Machine learning produces datasets, code, evaluations, and models too detailed for papers, yet its results are rarely organized for reuse. Existing repositories and experiment platforms help with sharing or benchmarking but do not satisfy all requirements for scalable collaboration.
- Machine learning studies involve large datasets, complex code, evaluations, and models that papers cannot adequately represent.
- Most machine learning work is published only in highly summarized forms, limiting reuse, reinterpretation, and mining of results.
- Without prior experiments to build on, researchers duplicate effort and rerun experiments, limiting study depth, interpretability, and generalizability.
- Prior work: Public dataset, software, and experiment repositories improve reproducibility or comparability but often require specialized formats or server-specific code.
- Prior work: Existing tools may lack queryable reuse, online discussion, easy contribution, broad experiment coverage, or sharing of models beyond benchmarks.
4. OPENML
OpenML is an online infrastructure for sharing, organizing, comparing, and discussing machine-learning data, tasks, implementations, and results. Its structured tasks, versioned resources, integrations, and programmatic interfaces support reproducible comparison and collaboration.
- OpenML services: OpenML lets researchers share data sets, tasks, flows, and runs, while organizing associated information and results online.Its services are available through the website and REST API.
- Access and comparison: OpenML compares results across flows, parameter settings, data sets, and tasks, and provides pages, visualizations, discussions, APIs, and software integrations.The platform is integrated with tools including Weka, R, MOA, RapidMiner, and KNIME.
- Data sets: Data sets can be uploaded or referenced by URL, automatically versioned, and annotated with attribution and licensing information.OpenML supports references to landing pages and APIs for large scientific repositories.
- Tasks: Tasks specify inputs, expected outputs, and scientific protocols so that researchers solve the same problem under comparable conditions.A classification task can include a target attribute, 10-fold cross-validation, fixed data splits, required predictions, and server-side evaluation measures.
- Flows and runs: Flows represent algorithms, workflows, or scripts, while runs record their application to specific tasks together with outputs and parameter settings.OpenML versions uploaded flows and stores run details, including evaluation distributions and parameter-setting metadata.
5. NETWORKED MACHINE LEARNING
OpenML enables a fully networked approach to machine learning, supporting new discoveries, scalable collaboration, and benefits for researchers, students, and broader audiences.
- OpenML enables a fully networked approach to machine learning.
5.1 OpenML and networked science
OpenML organizes and shares datasets, code, and experiments so researchers can reuse information, discover patterns, and divide large studies across contributors.
- Sharing datasets, code, and experimental results at scale can stimulate designed serendipity and a dynamic division of labor.
- Combined experiments can answer previously difficult questions, support routine analyses, and enable metalearning about algorithm performance.OpenML can reduce some analyses from days of setup and experimentation to minutes by making data readily available.
- OpenML enables large-scale studies and collaborative problem solving by allowing scientists to contribute according to their skills, time, or resources.
- Scientists from other domains can contribute datasets and tasks, suggest techniques, design workflows, run experiments, and interpret practical results.
5.2 Scaling up collaboration
OpenML scales collaboration by making contributions easy, dividing complex work into manageable tasks, and organizing results for efficient discovery, comparison, and verification.
- Scientists can contribute anything from a single run, dataset, or comment to new tasks, evaluation measures, and tool integrations.OpenML plugins simplify importing tasks, running workflows, and exporting results.
- OpenML aggregates results by task and flow, while searches, filters, SQL, and downloads support targeted analysis of shared information.
- Shared code, server-side evaluations, and task-defined reporting standards make experiments more verifiable and objectively comparable.Contributions are linked to specific people, timestamps, and licensing information.
5.3 Benefits for scientists
OpenML offers individual scientists more time, knowledge, and reputation by organizing experiments, connecting results, enabling reuse, and increasing visibility and credit.
- More time: OpenML saves time by assisting with dataset discovery, implementation selection, experiment setup, organization, benchmarking, and routine analysis.Stored experiments remain accessible through browsers across locations and devices.
- More knowledge: Linking results to others’ work supports new discoveries, global discussion, collaborations, and larger, more generalizable studies.
- More reputation: OpenML increases reputation by making work more visible, connecting scientists with collaborators, and clarifying how contributors should be credited.
- More reputation: OpenML provides citation, reuse, productivity, visibility, and funding-related benefits for shared research resources.Shared resources can receive credit even when results are not positive, and reuse is automatically tracked through runs and studies.
5.4 Benefits for students
OpenML gives students an accessible, organized way to understand machine learning and participate in ongoing research. It also helps them identify techniques, open problems, and opportunities to contribute.
- Browsing organized results online is more accessible than browsing hundreds of papers, giving students an overview of machine learning’s state of the art.The overview also includes interesting new techniques and open problems.
- Students can contribute small or large amounts to ongoing research while learning how to become machine learning researchers.
- Online discussions can reveal new ideas and mistakes, helping students improve their work over time.
- OpenML provides students and young scientists with a large playground to learn machine learning more quickly and discover where they can contribute.
6. FUTURE WORK
Future work focuses on making OpenML studies more complete, supporting controlled sharing and collaboration, improving contribution credit, and expanding data-type support. These changes aim to make experiments easier to document, reuse, and extend.
- 6. FUTURE WORK: The project discusses proposed improvements and welcomes contributions because OpenML is open source.
- 6.1 OpenML studies: OpenML studies will combine experiments, runs, datasets, and flows on one page, with descriptions and supplementary materials attached.Published studies can be linked from papers so readers can find the associated research materials.
- Studies, datasets, and flows may be temporarily private or restricted to friends while researchers collaborate before publication.The proposed setting supports trusted-colleague feedback and editing while treating restricted access as temporary.
- Future leaderboards should credit contributors whose solutions influenced a collaboratively constructed flow, not only the current best solution.
- OpenML initially focuses on ARFF because interpreting dataset structure is needed for data characteristics and train-test splits.Support for additional data types, including graphs, is planned according to researchers’ needs.
7. CONCLUSIONS
The paper introduces OpenML as a networked approach for sharing, organizing, and reusing machine learning data, code, and experiments. It argues that this approach can improve productivity, visibility, collaboration, and discovery while enabling new kinds of studies.
- OpenML is an online service for sharing, organizing, and reusing machine learning data, code, and experiments.
- OpenML is designed to scale collaborations and reward scientists for sharing data more openly.
- The authors believe networked machine learning can help scientists work more productively, make discoveries faster, and become more visible.
- The approach may forge new collaborations and support types of studies that were previously practically impossible.