Source-linked AI summary
Automating biomedical data science through tree-based pipeline optimization
Randal S. Olson, Ryan J. Urbanowicz, Peter C. Andrews, Nicole A. Lavender, La Creis Kidd, Jason H. Moore
TL;DR
Machine learning pipeline design requires numerous expert-guided and time-consuming choices, motivating automation of the full pipeline-design process. The paper introduces TPOT, which evolves tree-based pipelines with genetic programming and demonstrates competitive accuracy plus novel synthetic feature construction on simulated and real genetic data. The authors identify overfitting as a major limitation and present the work as an early step toward fully automated pipeline design.
Problem
Machine learning pipeline design involves many preprocessing, model, and parameter choices that can require expert knowledge or time-consuming search.
Method
TPOT uses genetic programming to automatically construct and optimize tree-based pipelines containing data transformations, feature selection, feature construction, and classifiers.
Results
TPOT built pipelines with competitive classification accuracy and discovered synthetic feature constructors that significantly improved accuracy across simulated and real-world genetic data sets.
Takeaways & Limitations
TPOT can serve as a data science assistant that explores data, discovers novel features, and recommends pipelines for practitioners to develop with domain knowledge.
Takeaways & Limitations
TPOT produced a pipeline that significantly overfit the testing data on the CGEMS prostate cancer data set despite a 75%/25% training/testing evaluation.
Abstract
from arXiv · showhide
Over the past decade, data science and machine learning has grown from a mysterious art form to a staple tool across a variety of fields in academia, business, and government. In this paper, we introduce the concept of tree-based pipeline optimization for automating one of the most tedious parts of machine learning---pipeline design. We implement a Tree-based Pipeline Optimization Tool (TPOT) and demonstrate its effectiveness on a series of simulated and real-world genetic data sets. In particular, we show that TPOT can build machine learning pipelines that achieve competitive classification accuracy and discover novel pipeline operators---such as synthetic feature constructors---that significantly improve classification accuracy on these data sets. We also highlight the current challenges to pipeline optimization, such as the tendency to produce pipelines that overfit the data, and suggest future research paths to overcome these challenges. As such, this work represents an early step toward fully automating machine learning pipeline design.
1 Introduction
Machine learning pipeline design requires many expert-guided, time-consuming choices across preprocessing, modeling, parameter tuning, and validation. This paper introduces TPOT, which uses genetic programming to automatically construct and optimize pipelines for competitive classification and feature discovery.
- Machine learning applications often require expert knowledge, awareness of analytical assumptions, or exhaustive search, making them time-consuming and computationally demanding.
- Typical pipelines involve dozens of choices for preprocessing, model selection, parameter settings, and other operations.
- Practitioners may spend several weeks tuning model parameters and data transformations before achieving acceptable performance.
- Evolutionary algorithms motivate the question of whether intelligent systems can automatically design machine learning pipelines.
- TPOT uses genetic programming to construct transformation-and-model pipelines, achieving competitive accuracy and discovering synthetic feature constructors that significantly improve accuracy.
2 Related Work
Earlier automated machine learning work optimized individual pipeline components, including hyperparameters and feature construction. The paper positions TPOT as an early attempt to optimize entire pipelines automatically.
- Earlier automated machine learning pipeline optimization generally focused on specific pipeline elements rather than complete pipelines.
- Grid search exhaustively explores model-parameter combinations, while random exploration can find high-performing settings faster than exhaustive search.
- Bayesian optimization has effectively optimized model hyperparameters and outperformed manual tuning by expert practitioners in prior research.
- Automated feature construction has produced expert-level performance in three machine learning competitions using relational data.
- The authors state that no published work had yet optimized entire machine learning pipelines, establishing a blueprint for future research.
3 Methods
The methods represent pipelines as trees of feature operations and classifiers, then evolve those trees and their parameters with genetic programming. TPOT is evaluated on diverse simulated genetic data, using stratified training and testing accuracy as pipeline fitness.
- 3.1 Decision Trees and Random Forests: Decision trees and random forests perform binary classification, with random forests aggregating votes from trees trained on bootstrap samples.
- 3.2 Synthetic Feature Construction: Synthetic features can be generated from classifier outputs, allowing decision trees and random forests to add classifications as new features.
- 3.3 Decision Tree-based Feature Selection: Feature selection exhaustively evaluates every two-feature combination and ranks pairs by decision-tree training classification accuracy.
- 3.4 Tree-based Pipelines: TPOT implements tree-based pipelines from feature construction, feature selection, decision-tree classification, and random-forest classification operators.
- 3.4 Tree-based Pipelines: Pipeline fitness is based on classification accuracy after stratified division into 75% training and 25% testing data.
- 3.5 Genetic Programming: Genetic programming evolves operator sequences and their parameters in tree pipelines to maximize final classification accuracy.
- 3.6 GAMETES Simulated Data Sets: The evaluation uses 12 models and 360 associated GAMETES data sets representing pure, strict epistatic genetic models.
4 Results
TPOT performed best on larger, less noisy GAMETES data sets and benefited from feature construction and selection, while random pipeline generation matched genetic-programming selection. On CGEMS prostate cancer data, TPOT discovered synthetic-feature pipelines with competitive testing accuracy but substantial cross-validation degradation.
- GAMETES Simulated Data Sets: TPOT exceeded 80% testing accuracy with 200 records at 0.4 heritability and exceeded 65% with 800 records at 0.1 heritability.At 200 records or fewer and 0.1 heritability, performance was near the 50% chance level.
- GAMETES Simulated Data Sets: TPOT with feature selection and construction performed significantly better than random forest and models-only TPOT on all but the most difficult data sets.The comparison indicates that the additional feature operators improved classification accuracy beyond optimizing model choice and parameters alone.
- GAMETES Simulated Data Sets: TPOT with genetic-programming selection did not perform significantly differently from TPOT random search across the GAMETES tests.Randomly generating 10,000 pipelines eventually discovered a top-performing pipeline in these data sets.
- CGEMS Prostate Cancer Data Set: On CGEMS prostate cancer data, TPOT achieved 60.8% balanced testing accuracy versus 59.8% previously reported with CES, but fell to 51.7% under 10-fold cross-validation.The discrepancy suggests overfitting despite a 75%/25% training/testing split during optimization.
- CGEMS Prostate Cancer Data Set: Adding three synthetic features increased CGEMS balanced testing accuracy from 54.7% on raw data to 58.7% after the second step and 60.8% at the final classifier.The pipeline used random forest, decision tree, and random forest feature constructors before final random forest classification.
- CGEMS Prostate Cancer Data Set: Feature-importance analysis showed synthetic features were built from and then reused alongside SNP information, including NAT2, BCL2, RAF1, and BID.Synthetic Feature 1 became important in constructing Synthetic Feature 2, while later features integrated prior synthetic features with new SNP information.
5 Discussion
TPOT is presented as a data science assistant that can recommend pipelines while leaving practitioners responsible for integrating domain knowledge. The discussion reports useful discoveries alongside limitations involving guided search, overfitting, and interaction detection.
- TPOT is intended to assist practitioners by exploring data, discovering features, and recommending pipelines rather than replacing domain expertise.The authors released it as an open-source Python package.
- Guided search did not perform significantly differently from randomly generated pipelines, likely because TPOT lacks clear building blocks for genetic programming.The authors propose dividing pipeline operators into better building blocks.
- TPOT overfit the testing data on the CGEMS prostate cancer data set despite using a 75%/25% training/testing split.The authors suggest multi-objective or Pareto optimization incorporating generalization or pipeline complexity.
- Random forest and decision tree classifiers failed to detect epistatic feature pairs in GAMETES data sets, motivating interaction-aware methods such as SURF.The discussion attributes this failure to algorithms focusing on single feature-class correlations.
- TPOT achieved competitive classification accuracy and discovered synthetic features that significantly contributed to classification on the CGEMS prostate cancer data set.NAT2 and BCL2 were among the SNPs used by TPOT and previously associated with prostate cancer aggressiveness.
- Future development targets more feature-selection and construction operators, additional machine-learning models, and improved hyperparameter optimization.Examples include SURF, support vector machines, logistic regression, k-nearest neighbors, and Bayesian optimization.
6 Conclusions
The paper presents tree-based pipeline optimization as an approach to automating a laborious part of machine learning design. Its demonstrations show competitive pipelines and novel operators, while identifying overfitting as a challenge on the path toward fuller automation.
- Tree-based pipeline optimization automatically creates and optimizes machine-learning pipelines on simulated and real-world genetic data sets.
- TPOT built pipelines with competitive classification accuracy and discovered novel operators, including synthetic feature constructors that significantly improved accuracy.
- Overfitting remains a current challenge, making this work an early step toward fully automating machine-learning pipeline design.