Source-linked AI summary
Data-driven Advice for Applying Machine Learning to Bioinformatics Problems
Randal S. Olson, William La Cava, Zairah Mustahsan, Akshay Varik, Jason H. Moore
TL;DR
Bioinformatics researchers face difficulty choosing among many machine-learning algorithms, while prior comparisons often covered relatively few problems. This paper evaluates 13 algorithms on 165 supervised classification datasets, quantifies model-selection and hyperparameter-tuning effects, and recommends five algorithms with parameters based on benchmark coverage.
Problem
Researchers face difficulty selecting appropriate machine-learning algorithms, and prior comparisons often evaluated relatively few example problems.
Method
The paper empirically compares 13 supervised classification algorithms on 165 standardized PMLB datasets, including full hyperparameter optimization.
Results
The analysis shows problem-dependent algorithm performance, significant differences across algorithms, and approximately a 20% accuracy increase from model selection and tuning, with larger gains on some datasets.
Takeaways & Limitations
Five recommended algorithms and parameter settings cover 106 of 165 datasets within 1% balanced accuracy, while tuning those algorithms covers 163 of 165 datasets.
Takeaways & Limitations
The recommendations are starting points rather than universally best choices, and the experiments do not account for feature preprocessing, construction, or selection.
Abstract
from arXiv · showhide
As the bioinformatics field grows, it must keep pace not only with new data but with new algorithms. Here we contribute a thorough analysis of 13 state-of-the-art, commonly used machine learning algorithms on a set of 165 publicly available classification problems in order to provide data-driven algorithm recommendations to current researchers. We present a number of statistical and visual comparisons of algorithm performance and quantify the effect of model selection and algorithm tuning for each algorithm and dataset. The analysis culminates in the recommendation of five algorithms with hyperparameters that maximize classifier performance across the tested problems, as well as general guidelines for applying machine learning to supervised classification problems.
1. Introduction
Bioinformatics increasingly uses machine learning for complex biological prediction, but the abundance of available algorithms makes selection difficult. This study addresses that challenge through a broad, reproducible comparison intended to provide data-driven guidance.
- Motivation: Machine learning is increasingly applied to predictive analytics and biological pattern detection in bioinformatics.Applications include GWAS, epistasis detection, and cancer-metastasis identification in pathology images.
- Motivation: Open-source implementations give researchers many algorithm choices, but this can create “choice overload” when selecting a model.Some projects could improve through use of a better machine learning algorithm.
- Prior evidence: Earlier empirical algorithm comparisons often evaluated many algorithms on relatively few problems, limiting the breadth of evidence.Most studies used between 4 and 12 example problems, while only 7 used upwards of 112.
- Study aim: The study compares 13 popular open-source algorithms across 165 supervised classification problems with full hyperparameter optimization.Its goals include quantifying tuning effects, examining algorithm performance patterns, and developing recommendations.
2. Methods
The study evaluates 13 machine learning algorithms from scikit-learn on 165 standardized supervised classification datasets, including biomedical problems. It uses grid-search hyperparameter tuning with 10-fold cross-validation and balanced accuracy, alongside extensive computational analysis and reproducible code.
- Algorithms: The study compares 13 popular machine learning algorithms from scikit-learn, spanning common algorithm classes and Gradient Tree Boosting.The classes include Naive Bayes, linear, tree-based, distance-based, ensemble, and kernel-based methods.
- Evaluation: Each algorithm’s hyperparameters are tuned with fixed-grid search and 10-fold cross-validation, comparing average balanced accuracy across folds.Balanced accuracy accounts for class imbalance, while expert knowledge defines the tuning ranges.
- Datasets: The benchmark contains 165 supervised classification datasets from the Penn Machine Learning Benchmark (PMLB), standardized and centrally accessible.PMLB includes biomedical tasks such as disease diagnosis, post-operative decision making, and exon-boundary identification.
- Preprocessing: Features are standardized by subtracting their means and scaling them to unit variance before algorithm evaluation.This preprocessing is required by some methods, including distance-based classifiers.
- Reproducibility: The experiments comprise over 5.5 million algorithm and parameter evaluations, with complete optimization code and analysis results made available.The release supports reproducible comparisons across datasets.
3. Results
Across 165 supervised classification problems, algorithm performance differed substantially, with ensemble tree methods generally strongest and Naïve Bayes methods weakest. Model selection and hyperparameter tuning materially improved accuracy, motivating five recommended algorithms that cover most benchmark datasets near the best observed performance.
- Algorithm Performance: Ensemble-based tree algorithms occupied three of the four highest average rankings, while the three lowest-ranked algorithms were Naïve Bayes methods.Rankings were based on 10-fold cross-validation balanced accuracy, with lower rank indicating higher accuracy.
- Algorithm Performance: The Friedman test found statistically significant performance differences across algorithms (p < 2.2e−16), and Gradient Tree Boosting significantly outperformed every algorithm except Random Forest at p < 0.01.Multinomial NB was significantly outperformed by every algorithm except Gaussian NB.
- Algorithm Coverage: No single algorithm performed best across all 165 datasets; even Multinomial NB matched or exceeded Gradient Tree Boosting on 9 datasets.The comparison treated algorithms as tied when their accuracies were within 1% in the pairwise outperformance analysis.
- Effect of Tuning and Model Selection: Hyperparameter tuning often improved an algorithm’s accuracy by 3-5%, with improvements reaching 50% in some cases.Figure 3 compares tuned classifiers with scikit-learn default settings using 10-fold cross-validation accuracy.
- Effect of Tuning and Model Selection: Selecting and tuning the best model increased accuracy by approximately 20% on average, with improvements exceeding 60% for certain datasets.The comparison was against average performance on each dataset.
- Algorithm Coverage: Five algorithms and associated parameter settings covered 106 of 165 datasets within 1% of the best balanced accuracy, while tuning those five expanded coverage to 163 datasets.The recommendations maximize the number of benchmark datasets covered within 1% of the best 10-fold cross-validation balanced accuracy.
4. Discussion and Conclusions
The study supports data-driven machine-learning recommendations while emphasizing that algorithm performance depends on the dataset and that model selection and tuning substantially improve predictive accuracy. Its five recommended algorithm configurations are useful starting points, but broader applicability is limited by interpretability concerns and the focus on classification without feature-engineering analysis.
- 13 algorithms across 165 datasets showed that state-of-the-art tree-based ensembles were strong, while performance remained problem-dependent.The authors made the full experiments and results available online for bioinformaticians.
- The five recommendations should be treated as starting points because they will not work best on all supervised classification problems.The authors suggest dataset similarity analysis or automated model-selection and tuning tools for more nuanced applications.
- Predictive accuracy may trade off against interpretability, making linear models and shallow decision trees useful when black-box models are difficult to interpret.The discussion identifies LIME as a possible aid for explaining individual predictions from complex models.
- The analysis does not cover regression, feature preprocessing, feature construction, or feature selection, which the authors identify as future extensions.Future work also aims to relate dataset properties to algorithm performance for more tailored recommendations.