Source-linked AI summary
Issues in Stacked Generalization
K. M. Ting, I. H. Witten
TL;DR
The paper studies which higher-level model and inputs work best for stacked classification. It evaluates confidence-based stacking with MLR and finds that stacking generally outperforms model selection, majority vote, arcing, and bagging.
Problem
The paper addresses which higher-level generalizer and input attributes are most suitable for stacked classification.
Method
It evaluates MLR as a higher-level model that combines the confidence generated by level-0 classification models.
Results
Stacking achieves higher predictive accuracy than model selection and majority vote and outperforms both arcing and bagging in the reported comparisons.
Takeaways & Limitations
The findings support confidence-based MLR stacking as an effective approach for combining classification algorithms.
Takeaways & Limitations
The reported evaluation includes two small real-world datasets, constraining the stated comparison scope.
Abstract
from arXiv · showhide
Stacked generalization is a general method of using a high-level model to combine lower-level models to achieve greater predictive accuracy. In this paper we address two crucial issues which have been considered to be a `black art' in classification tasks ever since the introduction of stacked generalization in 1992 by Wolpert: the type of generalizer that is suitable to derive the higher-level model, and the kind of attributes that should be used as its input. We find that best results are obtained when the higher-level model combines the confidence (and not just the predictions) of the lower-level ones. We demonstrate the effectiveness of stacked generalization for combining three different types of learning algorithms for classification tasks. We also compare the performance of stacked generalization with majority vote and published results of arcing and bagging.
Summar y
The summary identifies MLR as the strongest level-1 generalizer, especially for combining lower-level confidence estimates, while highlighting its interpretability and competitive performance against majority vote, arcing, and bagging. Stacking outperforms both alternatives on three datasets but performs poorly on two small datasets.
- MLR is the best of the four learning algorithms for use as the level-1 generalizer.
- Non-negativity constraints make little difference to predictive accuracy but improve MLR’s interpretability of each model’s contribution to prediction classes.
- Comparison With Arcing And Bagging: Stacking performs better than both arcing and bagging on three datasets and is very competitive overall.
- Comparison With Arcing And Bagging: Stacking beats arcing but loses to bagging on Diabetes, performs very poorly on Glass and Ionosphere, and cross-validation produces poor estimates for small datasets.
. Discussion
The discussion emphasizes that stacking accommodates heterogeneous level-0 models, including both stable and unstable learners, while outperforming simpler voting in several settings. MLR is the strongest level-1 generalizer identified, though its limitations and computational trade-offs remain relevant.
- Model diversity: Stacking handles substantial performance differences among heterogeneous level-0 models, including cases where NB performs very poorly on the Vowel and Euthyroid datasets.The paper states that stacking copes well when level-0 model performance varies substantially.
- Comparison with bagging: Bag-stacking almost always achieves higher predictive accuracy than bagging models derived from C4.5 or NB.The reported difference is using an adaptive level-1 model rather than simple majority voting.
- Model diversity: Stacking works with both unstable learners, such as decision trees and neural networks, and stable learners, such as NB and IB1.Arcing and bagging are described as benefiting unstable algorithms, whereas stacking is reported to work with both categories.
- Level-1 generalization: Non-negativity constraints are irrelevant to MLR’s predictive accuracy in the classification setting.The paper contrasts this result with earlier work reporting benefits from such constraints on an artificial dataset.
- Level-1 generalization: MLR is the most successful level-1 candidate identified and performs better than IB1, its nearest competitor.The discussion notes that MLR’s known limitations include linear decision boundaries and suitability primarily for unimodal probability densities.
. Conclusions
The paper concludes that successful stacked generalization uses class probabilities as higher-level inputs and multi-response least squares linear regression as the generalizer. This confidence-based implementation improves predictive accuracy over model selection and majority vote, while remaining competitive with arcing and bagging.
- Conclusions: Successful stacked generalization uses class probabilities as higher-level inputs and multi-response least squares linear regression as the high-level generalizer.Class probabilities provide confidence measures that the linear regression combines; alternative algorithms have limitations or are unsuitable for aggregating confidences.
- Conclusions: The implementation achieves better predictive accuracy than cross-validation-based model selection and majority vote, while remaining competitive with arcing and bagging.These findings concern combining three different types of learning algorithms.
- Conclusions: Non-negativity constraints in least-squares regression are unnecessary to guarantee improved classification accuracy, but they remain preferred for interpretability.The conclusion contrasts this result with stacked regression.
- Conclusions: The successful implementation enables earlier model-combination methods based on weighted majority vote, averaging, or other non-level-1 computations to use level-1 learning to improve predictive accuracy.The implication is that these earlier methods can incorporate the learning step used by stacked generalization.