Source-linked AI summary
Optimizing Ensemble Weights and Hyperparameters of Machine Learning Models for Regression Problems
Mohsen Shahhosseini, Guiping Hu, Hieu Pham
TL;DR
The paper addresses the limitation of tuning base learners and ensemble weights independently. It introduces GEM-ITH, a nested optimization method that tunes both concurrently, and reports improved prediction accuracy and generalizability across ten public data sets.
Problem
Existing ensemble methods commonly tune base-model hyperparameters independently from ensemble weighting, leaving their joint optimization insufficiently studied.
Method
GEM-ITH uses nested optimization to tune each base learner’s hyperparameters and ensemble weights concurrently, with Bayesian search and heuristic learner generation.
Results
GEM-ITH improved each base learner’s prediction accuracy, outperformed state-of-the-art ensemble methods, and generalized across ten publicly available data sets.
Takeaways & Limitations
Tuning hyperparameters inside the optimal ensemble-creation procedure was reported to produce better prediction accuracy than treating tuning and weighting independently.
Takeaways & Limitations
Because GEM-ITH has high computational complexity, the paper considers it more appropriate for small to medium size data sets.
Abstract
from arXiv · showhide
Aggregating multiple learners through an ensemble of models aim to make better predictions by capturing the underlying distribution of the data more accurately. Different ensembling methods, such as bagging, boosting, and stacking/blending, have been studied and adopted extensively in research and practice. While bagging and boosting focus more on reducing variance and bias, respectively, stacking approaches target both by finding the optimal way to combine base learners. In stacking with the weighted average, ensembles are created from weighted averages of multiple base learners. It is known that tuning hyperparameters of each base learner inside the ensemble weight optimization process can produce better performing ensembles. To this end, an optimization-based nested algorithm that considers tuning hyperparameters as well as finding the optimal weights to combine ensembles (Generalized Weighted Ensemble with Internally Tuned Hyperparameters (GEM-ITH)) is designed. Besides, Bayesian search was used to speed-up the optimizing process, and a heuristic was implemented to generate diverse and well-performing base learners. The algorithm is shown to be generalizable to real data sets through analyses with ten publicly available data sets.
1. Introduction
The paper motivates ensemble regression by noting that combining diverse learners can capture more of the data structure than a single model. It proposes jointly tuning base-model hyperparameters and ensemble weights, evaluated across multiple data sets.
- Ensembles combine multiple learners to reduce bias, variance, or both, because a single model may not capture the full data structure.
- Bagging, boosting, and stacking/blending differ in how they combine learners and address bias, variance, noise, and overfitting.
- Weighted ensemble design must address both base-model hyperparameter tuning and the weights used to form final predictions.
- GEM-ITH jointly tunes hyperparameters and model weights, using Bayesian search instead of exhaustive search to speed optimization.
- Experiments on data sets from different areas were conducted to assess the algorithm’s generalizability.
2. Background
The background frames regression as prediction of continuous responses and identifies a gap in treating model tuning and ensemble weighting as separate steps. The paper addresses this gap through a nested optimization framework that considers their interaction.
- The paper focuses on regression, where continuous responses are predicted from data with n instances and p predictor variables.
- Prior work includes constant and dynamic weighting approaches and applications of ensembles across neural networks, drought prediction, proteins, and data streams.
- Existing ensemble studies commonly treat base-model construction and weighted averaging as independent processes.
- The paper integrates parameter tuning and ensemble-weight design in one framework to consider the bias-variance tradeoff together.
- The proposed nested optimization uses cross-validation to optimize hyperparameters and ensemble weights at different levels while minimizing prediction mean squared error.
3. Materials and methods
The methods formulate weighted ensemble construction as an optimization problem for regression and use cross-validation to obtain predictions for weight estimation. GEM-ITH extends this setup by tuning hyperparameters within ensemble optimization.
- Prediction error is discussed through bias and variance, with irreducible noise variance identified as unreducible by model choice.
- The ensemble objective minimizes mean squared error for weighted predictions of continuous targets.
- The weights are constrained to be nonnegative and sum to one, producing a convex optimization problem with a globally optimal solution.
- Cross-validation supplies out-of-bag predictions used as inputs to estimate ensemble weights and helps provide diverse training sets for base learners.
- GEM assumes each base learner is tuned before weighting, whereas GEM-ITH searches hyperparameter combinations within the ensemble-design process.
- The optimization-based algorithm searches for the hyperparameters that produce the best-performing ensemble.
4. Generalized Ensemble Model with Internally Tuned hyperparameters (GEM-ITH)
GEM-ITH jointly searches ensemble weights and base-learner hyperparameters, using Bayesian search to reduce the cost of evaluating the combined search space.
- Joint optimization: GEM-ITH searches for hyperparameters and ensemble weights together rather than tuning base models independently before weighting them.The nested optimization seeks the hyperparameter combination producing the least prediction error.
- Search strategy: Bayesian search selects candidate hyperparameter combinations for each base learner, replacing computationally expensive exhaustive search.The selected candidates generate the combinations evaluated by GEM-ITH.
- Search strategy: With b Bayesian iterations and k base learners, GEM-ITH evaluates b^k total hyperparameter combinations.Each combination supplies cross-validation predictions used by the optimization model.
- Optimization procedure: For each candidate combination, cross-validation predictions are used to compute ensemble weights and an objective value, with the minimum objective selected.The algorithm outputs the optimal objective value MSE* and corresponding solution.
5. Results and discussion
Experiments evaluated GEM-ITH with diverse base learners on ten public data sets against several ensemble benchmarks. GEM-ITH improved over base learners and won on 9 of 10 data sets, while its computational complexity limits practical use to small and medium data sets.
- Experimental setup: GEM-ITH was evaluated on ten public data sets using four heuristically selected base learners, five-fold cross-validation, and repeated experiments.Twenty percent of each data set was reserved for testing.
- Benchmarks: The benchmark comparison included GEM, averaging-based ensembles, and stacked ensembles using linear regression, random forest, or k-nearest neighbors.These methods formed the state-of-art comparison set.
- Results: GEM-ITH improved prediction accuracy over the individual base learners in the reported Table 3 results.Table 3 reports mean squared prediction errors alongside the ensemble and benchmark results.
- Results: GEM-ITH found different optimal hyperparameters from independently tuned GEM models, and internal tuning produced higher-quality ensembles.This pattern was observed for Energy Efficiency and reported as consistent with the other data sets.
- Results: GEM-ITH won on 9 out of 10 public data sets against the state-of-art benchmarks.Figure 2 compares average normalized test errors across the data sets.
- Computational considerations: Because GEM-ITH has high computational complexity, the authors consider it more appropriate for small to medium size data sets.Computation time depends heavily on base-learner complexity and data-set dimensions.
6. Conclusion
The study concludes that GEM-ITH improves ensemble prediction accuracy across ten public data sets and differs from independently tuned GEM in its selected hyperparameters. The authors identify classification, other ensemble methods, and further speed-up as future directions.
- Major findings: GEM-ITH improved the prediction accuracy of each base learner and outperformed the compared state-of-art ensemble methods.The conclusion also reports improvement over the GEM scheme.
- Major findings: Internal hyperparameter tuning within ensemble optimization produced different selections from independent GEM tuning and was associated with higher-quality predictions.The conclusion presents this as confirmation of the study’s hypothesis.
- Generalizability: Results from ten publicly available data sets with diverse properties were presented as evidence of GEM-ITH’s generalizability to real data sets.The conclusion frames this as addressing the paper’s generalization question.
- Future directions: Future work includes extending the nested algorithm to classification and applying internal hyperparameter tuning to other ensemble methods.The authors also propose further acceleration of ensemble creation with hyperparameter tuning.