PaperlayerView exact revision on arXiv

Source-linked AI summary

Ensemble deep learning: A review

M. A. Ganaie, Minghui Hu, A. K. Malik, M. Tanveer, P. N. Suganthan

arXiv:2104.02395v3cs.LGcs.AIcs.CV

TL;DR

Deep learning models are difficult to train because their many weights complicate optimization, motivating ensemble approaches that combine deep and randomized models. The paper reviews recent deep ensemble developments, finding that implicit and explicit ensembles are faster than training multiple deep models and that consensus clustering can improve performance in most cases.

  • Problem

    Deep learning models are difficult to train because their large number of weights complicates the problem.

  • Method

    The paper comprehensively reviews recent deep ensemble models, including randomized learning modules and implicit or explicit ensemble approaches.

  • Results

    Implicit/explicit ensembles are faster than training multiple deep models, and consensus clustering can improve performance in most cases.

  • Takeaways & Limitations

    The review covers deep ensemble applications across different domains and identifies open problems, including big data, for future research.

  • Takeaways & Limitations

    Bagging-based deep ensembles may incur heavy training time when optimizing the ensemble.

Abstract

from arXiv · show

Ensemble learning combines several individual models to obtain better generalization performance. Currently, deep learning architectures are showing better performance compared to the shallow or traditional models. Deep ensemble learning models combine the advantages of both the deep learning models as well as the ensemble learning such that the final model has better generalization performance. This paper reviews the state-of-art deep ensemble models and hence serves as an extensive summary for the researchers. The ensemble models are broadly categorised into bagging, boosting, stacking, negative correlation based deep ensemble models, explicit/implicit ensembles, homogeneous/heterogeneous ensemble, decision fusion strategies based deep ensemble models. Applications of deep ensemble models in different domains are also briefly discussed. Finally, we conclude this paper with some potential future research directions.

1. Introduction

Deep ensemble learning combines deep architectures with ensemble methods to improve generalization. This review surveys the approaches, applications, challenges, and future directions of deep ensemble models.

  • Ensemble learning combines predictions from multiple models through methods such as averaging or voting to improve performance over individual models.
  • Deep learning extracts high-level features through hierarchical representations and has achieved state-of-the-art results across domains including detection and segmentation.
  • Deep ensemble models aim to combine the advantages of deep learning and ensemble learning for improved model performance.
  • Practical deep ensemble design must address model diversity, training time, model complexity, and prediction fusion.
  • The review organizes deep ensemble research across ensemble strategies, fusion approaches, model types, and applications, while identifying open research directions.

2. Research Methodology

The review searched Google Scholar and Scopus for studies on ensemble and deep ensemble learning. Articles were screened by title, abstract, and full text before being synthesized by methodological approach.

  • Studies were retrieved from Google Scholar and Scopus using ensemble learning and deep ensemble learning keywords.
  • Articles were screened first by title and abstract and then by their full-text versions.
  • The selected articles were elaborated according to ensemble learning and deep learning approaches.

3. Theory

The theory section explains ensemble success through error decompositions, diversity, and statistical, computational, and representational arguments. It also contrasts major ensemble strategies and their effects on bias, variance, and classifier diversity.

  • Bias-Variance Decomposition: A proper ensemble classifier can achieve smaller squared error than its individual predictors through ambiguity decomposition.
  • Bias-Variance Decomposition: Bias-variance-covariance decomposition represents ensemble error using target values, model outputs, ensemble size, variance, and covariance terms.
  • Theoretical foundations: Ensemble theory includes bias-variance, strength-correlation, stochastic discrimination, and margin-based explanations.
  • Ensemble strategies: Bagging reduces variance, whereas boosting-based ensembles reduce both bias and variance.
  • Statistical, Computational and Representational Aspects: Ensembling can expand the representable function space when individual hypotheses cannot represent the true unknown function.
  • Ensemble diversity: Diversity is generated through methods including bagging, AdaBoost, random subspace, random forest, and output smearing.

4. Ensemble Strategies:

The review presents bagging as a classical ensemble strategy that trains models on sampled datasets and combines their predictions. Deep-learning applications report improved generalization, but computational cost and scalability remain important constraints.

  • Bagging: Bagging generates multiple training bags, fits independent base models, and combines their predictions through strategies such as voting or averaging.For classification, majority voting is commonly used; regression uses averaging.
  • Bagging: Random Forest decorrelates decision trees by randomly selecting a feature subset at each split, helping prevent over-fitting.Its bagged predictor was heuristically shown to have lower variance than the original predictor.
  • Bagging: Half subbagging is computationally efficient while remaining as accurate as bagging.Subbagging and half subbagging were introduced because bagging is computationally expensive.
  • Bagging: Bagging-based ensembles can outperform individual multilayer perceptrons and regularize neural networks, improving generalization.These findings were reported for bagging-based neural-network ensembles and comparisons with other regularization techniques.
  • Bagging: Roughly Balanced Bagging and Neighbourhood Balanced Bagging address class imbalance, but neither had been explored in deep-learning architectures.The review identifies these methods as potential approaches for deep-learning imbalance problems.
  • Bagging: Deep-model bagging can be parallelized to reduce training time when suitable hardware is available, yet optimizing multiple deep models may remain infeasible.Deep learning’s high training time makes optimization across multiple training bags difficult.

4.2. Boosting

Boosting builds stronger deep ensembles by combining weak learners or deep models stagewise, often reweighting difficult examples. The reviewed approaches improve performance or efficiency in several settings, while training complexity and overfitting remain concerns.

  • Boosting: Boosting converts weak learning models into stronger classifiers through forward stagewise additive modeling.AdaBoost minimizes exponential loss, whereas gradient boosting generalizes the framework to arbitrary differentiable loss functions.
  • Boosting: Deep boosting selects classifiers and their weights stage by stage according to data-dependent classifier complexity.Its interpretation is given through structural risk minimization at each learning stage.
  • Boosting: Incremental Boosting CNN accumulates information across training batches to reduce overfitting caused by limited data in each mini-batch.It uses decision stumps over single neurons as weak learners and learns weights with AdaBoost within each mini-batch.
  • Boosting: CNN features followed by a boosted forest were more efficient than end-to-end CNN training in both performance and time.The CNN generated high-level features, while the boosted forest performed final classification.
  • Boosting: Snapshot boosting combines snapshot ensembling and boosting to improve generalization without increasing training cost.The approach trains each base network and combines its outputs through a meta-learner.
  • Boosting: BoostResNet was reported as more computationally efficient than DeepResNet while using multi-channel representation boosting.BoostResNet updates channel scalar values during boosting rounds to minimize multiclass error.

4.3. Stacking

Stacking integrates outputs from multiple base models through a meta-learning model, extending ensemble learning into deep architectures. The review covers stacked networks for large-scale, sparse, temporal, robotic, and vision applications, alongside scalability limitations.

  • Stacking: Stacking uses predictions from different learning models as meta-data for training a meta-model that makes the final prediction.Cross-validation partitions provide training and testing subsets for generating base-model predictions.
  • Stacking: Deep Convex Networks stack convex modules whose outputs feed adjacent higher modules alongside the original input.Each module contains linear input units, nonlinear hidden units, and a second linear layer matching the target classes.
  • Stacking: Kernel Deep Convex Networks improve over DCN, but kernelization increases memory requirements and may limit scalability to large datasets.Optimal performance also requires tuning stacked-network levels and kernel parameters.
  • Stacking: Random Fourier feature-based kernel deep convex networks approximate the Gaussian kernel, reducing training time and supporting evaluation on large-scale datasets.This approximation targets the memory requirements associated with the kernelized model.
  • Applications: Stacked architectures extend to sparse coding, reinforcement learning, neural architecture search, image deblurring, and abnormal detection.Examples include S-DSN, robotic grasp-and-place systems, repeatedly stacked blocks for neural architecture search, and hierarchical multi-patch deblurring networks.
  • Limitations and extensions: Deep Stacking Networks lack temporal representations and are less effective for problems with temporal dependencies.Recurrent Deep Stacking Networks combine DSNs with RNN-related mechanisms, using Echo State Network initialization and batch gradient descent fine-tuning.
  • Applications: A stacked conditionally restricted Boltzmann machine and deep neural network achieved superior performance with fewer parameters and training samples.The cited result concerns the specific combined architecture described in the review.

4.4. Negative Correlation Based Deep Ensemble Methods

Negative correlation learning trains ensemble members to capture diverse aspects of the data while minimizing ensemble and individual-model errors. Deep extensions span classification, regression, crowd counting, and incremental learning, with winner-takes-all outperforming simple averaging in one evaluation.

  • Deep extensions: Deep negative correlation learning extends the NCL concept to deep architectures, including decorrelated convolutional networks and deep regression ensembles.The reviewed regression architecture uses convolutional feature-mapped weak regressors.
  • Negative correlation learning: Negative correlation learning encourages diversity among ensemble members while minimizing the ensemble’s empirical risk through individual-network error functions.The approach was evaluated for both regression and classification.
  • Decision fusion: Classification ensembles using NCL were evaluated with simple averaging and winner-takes-all, while regression used simple average combination.The reported comparison found winner-takes-all better than simple averaging for NCL classification ensembles.
  • Generalized NCL: Generalized Negative Correlation Learning controls ensemble diversity through a generalized bias-variance decomposition and smoothly interpolates among existing approaches.The review reports superior performance for GNCL.
  • Incremental learning: NCL-based methods also support incremental learning through dynamically modified weighted-majority voting and ensemble incremental learning.These approaches combine sub-classifiers for incremental-learning tasks.

4.5. Explicit / Implicit Ensembles

Explicit and implicit ensembles both derive multiple predictive networks from deep models, but differ in whether model parameters are shared. These approaches reduce the cost of ensembling, while raising trade-offs involving training time and ensemble diversity.

  • Implicit ensembles: Implicit ensembles share weights and approximate model averaging with a single unthinned network at test time.
  • Implicit ensembles: Dropout randomly removes hidden nodes during training, creating an implicit ensemble of networks with shared weights and reducing overfitting.It trains an exponential number of models through shared parameters.
  • Implicit ensembles: Stochastic depth randomly drops residual blocks during training while retaining the network depth at testing, and Swapout generalizes this idea by dropping units or blocks.
  • Explicit ensembles: Explicit ensembles avoid parameter sharing; snapshot ensembling averages models saved at multiple local minima while keeping training time equal to a single model.
  • Trade-offs: Explicit and implicit ensembles can reduce base-model training costs, but shared lower-level features may limit prediction diversity.

4.6. Homogeneous & Heterogeneous ensembles

Homogeneous ensembles reuse one learner while heterogeneous ensembles combine different model families or perspectives. Deep ensembles can improve performance through diversity, but independently training many deep networks is computationally expensive.

  • Heterogeneous ensembles: Heterogeneous ensembles combine models from different families or algorithms, using shared data but distinct modeling perspectives.
  • Homogeneous ensembles: Homogeneous ensembles reuse the same base learner, so training must induce diversity and avoid identical errors across models.
  • Homogeneous ensembles: Homogeneous models can differ through bootstrapped samples, feature subsets, initialization, augmentation, regularization, or training epochs.
  • Limitations: Training multiple deep learning models is often infeasible because deep models have high training costs and millions or billions of parameters require optimization.
  • Performance: Ensembles of multiple networks with averaged outputs showed better performance in image classification and disease prediction.
  • Performance: Heterogeneous deep models can offer lower computation and higher diversity, and complex heterogeneous fusion architectures show better generalization performance.

4.7. Decision Fusion Strategies

Decision fusion combines base-learner outputs through rules such as averaging, voting, Bayesian combination, or stacking. The review emphasizes that fusion choice should reflect learner performance, bias, dependence, and diversity.

  • Overview: The output-combination rule determines an ensemble’s effective performance, while naive averaging is not data adaptive and can be suboptimal.
  • Unweighted averaging: Unweighted averaging is reasonable for comparably performing learners but can be suboptimal with heterogeneous, weak, or overconfident learners.
  • Unweighted averaging: Simple averaging can improve deep-ensemble generalization by reducing variance among high-variance, low-bias models.
  • Adaptive fusion: An adaptive metalearner can combine learners’ strengths when individual models perform particularly well on different subclasses.
  • Majority voting: Majority voting selects labels by vote counts and is less biased toward one base learner, but dependent learners can cause particular events to dominate.
  • Majority voting: For image classification, shallow-network predictions were reported as more diverse, supporting a hypothesis that shallow majority-vote ensembles may outperform deep ones.
  • Bayesian fusion: Bayesian optimal classifiers combine hypotheses using posterior probabilities, but overfitting motivates separating training and evaluation data.
  • Stacking: Stacked generalization learns a weighted linear combination of base predictions through a meta-learner, using cross-validation and constrained least squares.

5. Applications

Deep ensemble models have been applied across healthcare, speech, image classification, forecasting, and other domains. The reviewed literature most frequently uses boosting, stacking, heterogeneous, and implicit strategies.

  • Application domains: The review groups applications into healthcare, speech, image classification, forecasting, and other domains.
  • Healthcare: Healthcare studies frequently use heterogeneous ensembles, and combining different model families in one framework performs better in that domain.
  • Application domains: Speech applications account for 5.6% of models, while forecasting accounts for 9%.
  • Ensemble strategies: Across ensemble deep frameworks, boosting accounts for 18.2%, stacking 12.5%, and bagging 4.5% of implemented techniques.
  • Ensemble strategies: Heterogeneous and implicit ensembles account for 11.4% and 10.2%, respectively, while several other strategies each contribute smaller shares.

6. Conclusions and future works

The review summarizes recent deep ensemble learning developments, applications, and future research directions. It highlights performance and efficiency benefits alongside unresolved challenges in training, diversity, model selection, fusion, and emerging settings.

  • The paper reviews deep ensemble learning across traditional and recent approaches, including bagging, boosting, stacking, implicit/explicit, homogeneous/heterogeneous, and fusion-based ensembles.
  • Deep ensemble models have been applied across health care, speech, image classification, forecasting, and other domains.
  • Future work includes reducing bagging training time, increasing within-model diversity, and developing negative-correlation and heterogeneous ensemble methods.
  • Implicit/explicit ensembles are faster than training multiple deep models, while randomized modules can create deep ensembles without additional training cost.
  • Model selection and prediction fusion remain unresolved because the number and type of base learners are problem-dependent and combination strategies remain insufficiently studied.
  • Open directions include semi-supervised ensemble learning, integration with planning- and learning-based reinforcement learning, and applications to big-data learning.
Loading 2104.02395v3…