Source-linked AI summary
Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Science
Randal S. Olson, Nathan Bartley, Ryan J. Urbanowicz, Jason H. Moore
TL;DR
Machine-learning pipeline design involves many choices and can require substantial expertise, motivating automation for non-experts. The paper introduces TPOT, which evolves tree-based pipelines and evaluates them on simulated and benchmark data. TPOT often improves on basic analyses, while Pareto optimization produces compact pipelines; large-data-set runtime remains a limitation.
Problem
Effective machine-learning pipeline use requires expert and domain knowledge or exhaustive search, creating demand for tools accessible to non-experts.
Method
TPOT uses genetic programming to evolve tree-based sequences of data transformations, models, and parameters; TPOT-Pareto also minimizes pipeline complexity.
Results
Across simulated and UCI benchmarks, TPOT often improves over basic analyses, and TPOT-Pareto produces compact pipelines without sacrificing classification accuracy.
Takeaways & Limitations
TPOT can act as a data-science assistant that explores data, discovers features, and recommends pipelines while leaving users free to apply domain knowledge.
Takeaways & Limitations
TPOT remains slow on large data sets, often requiring several hours or days for analysis.
Abstract
from arXiv · showhide
As the field of data science continues to grow, there will be an ever-increasing demand for tools that make machine learning accessible to non-experts. In this paper, we introduce the concept of tree-based pipeline optimization for automating one of the most tedious parts of machine learning---pipeline design. We implement an open source Tree-based Pipeline Optimization Tool (TPOT) in Python and demonstrate its effectiveness on a series of simulated and real-world benchmark data sets. In particular, we show that TPOT can design machine learning pipelines that provide a significant improvement over a basic machine learning analysis while requiring little to no input nor prior knowledge from the user. We also address the tendency for TPOT to design overly complex pipelines by integrating Pareto optimization, which produces compact pipelines without sacrificing classification accuracy. As such, this work represents an important step toward fully automating machine learning pipeline design.
1. INTRODUCTION
Machine learning pipeline design requires many preprocessing, model, and parameter choices that can burden inexperienced practitioners. The paper asks whether evolutionary algorithms can automate this design and introduces TPOT to optimize pipelines with little user input.
- Machine learning tools increasingly need to serve non-experts, but effective use often requires expertise, domain knowledge, and exhaustive search.
- Each supervised-learning pipeline involves numerous choices about preprocessing, feature selection, model selection, and parameter optimization.
- TPOT uses genetic programming to automatically design and optimize data transformations and supervised-learning models for classification accuracy.
- The paper evaluates TPOT on simulated genetic-analysis data and nine UCI benchmark data sets, comparing standard TPOT with TPOT-Pareto.
- Pareto optimization jointly considers classification accuracy and pipeline complexity to produce effective, compact pipelines.
2. RELATED WORK
Earlier automation work often optimized only pipeline subsets, such as hyperparameters or feature construction. The paper situates TPOT as a broader approach that searches combinations of preprocessing and modeling operators.
- Machine-learning automation research has historically focused on optimizing subsets of the pipeline, including model hyperparameters.
- Randomly evaluating parameter settings can find strong hyperparameter choices more efficiently than exhaustive grid search.
- Automated feature construction systems can generate features from relational databases and achieved expert-level performance in three competitions.
- Auto-sklearn uses Bayesian optimization over fixed pipelines containing one data preprocessor, one feature preprocessor, and one model.
- TPOT extends this direction by targeting automated design of complete, potentially arbitrarily large machine-learning pipelines.
3. METHODS
TPOT represents supervised-learning pipelines as trees of reusable operators and evolves their structures and parameters with genetic programming. The evaluation uses simulated genetic data and nine UCI supervised-learning benchmarks.
- 3.1 Pipeline Operators: TPOT implements preprocessing, decomposition, feature-selection, and modeling operators using existing scikit-learn implementations.
- 3.2 Assembling Tree-based Pipelines: Tree-based pipelines pass one or more copies of the input data through operators that modify features and can combine branches before classification.
- 3.2 Assembling Tree-based Pipelines: TPOT evaluates final classification predictions after splitting each data set into stratified 75% training and 25% testing sets.
- 3.3 Evolving Tree-based Pipelines: Genetic programming evolves both operator sequences and operator parameters, modifying, removing, or inserting pipeline sequences to maximize classification accuracy.
- 3.3 Evolving Tree-based Pipelines: TPOT-Pareto optimizes classification accuracy and total operator count simultaneously by maintaining a Pareto front for reproduction selection.
- 3.4 GAMETES Data Sets: The evaluation includes 360 GAMETES data sets generated from 12 genetic models and nine hand-picked UCI supervised-learning benchmarks.
4. RESULTS
Across simulated and UCI benchmarks, TPOT generally matched or exceeded the random-forest baseline, while Pareto optimization produced more consistent and compact pipelines.
- GAMETES benchmarks: TPOT versions achieved 80%+ accuracy on the easiest GAMETES data sets, compared with 63% for RF.The results indicate that TPOT discovered epistatic interactions through preprocessing and modeling.
- Pipeline consistency: TPOT-Pareto tended to discover more consistent classification pipelines, especially on larger and higher-heritability GAMETES data sets.Its accuracy distributions had lower variance than those of the other TPOT versions.
- UCI benchmarks: TPOT generally matched RF across UCI data sets and significantly outperformed it on Hill-Valley and car-evaluation.On Hill-Valley-without-noise, TPOT-Pareto achieved 100% accuracy in all 30 replicates.
- Search efficiency: TPOT-Random typically matched guided TPOT in accuracy but was slower because randomly generated pipelines could be needlessly complex.Several large-data-set runs failed to finish within 120 hours, whereas guided versions completed equivalent evaluations in less than 48 hours.
- Pipeline size: TPOT-Pareto achieved comparable performance with 2 operators on average, versus 4 for TPOT and 6 for TPOT-Random.Figure 5 compares final pipeline sizes across random search, guided search, and Pareto optimization.
5. DISCUSSION
The discussion frames TPOT as a data-science assistant rather than a replacement for practitioners, while identifying search efficiency and scalability as important concerns.
- Role of TPOT: TPOT is intended to explore data, discover novel features, and recommend pipelines that users can export and combine with domain knowledge.The authors released it as an open source Python package.
- Guided search: Random search often matched guided search in accuracy, but it was slower, became computationally infeasible beyond 2,000 records, and produced less interpretable pipelines.Guided and Pareto search generated more compact pipelines that were easier to interpret and apply in production.
- Limitations and future work: TPOT remained fairly slow on large data sets, often requiring several hours or days for analysis.Planned improvements included seeding populations with promising pipelines and biasing genetic operators toward fitness-improving changes.
6. CONCLUSIONS
The conclusions present tree-based pipeline optimization as a promising approach for accessible, time-saving machine learning automation, with Pareto optimization improving compactness without sacrificing accuracy.
- Contribution: Tree-based pipeline optimization aims to make machine learning more accessible to non-experts and automate its most tedious parts.The authors describe it as a step toward fully automating pipeline design.
- Performance: TPOT matched a basic machine learning analysis across diverse data sets without user input or prior knowledge and sometimes significantly outperformed it.The improvement came from discovering combinations of preprocessing and modeling operators.
- Pipeline compactness: Pareto optimization produced compact, easy-to-interpret pipelines without sacrificing classification accuracy.This addresses TPOT’s tendency to generate overly complex pipelines.