PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Ensemble Learning under the Era of Deep Learning

Yongquan Yang, Haijun Lv, Ning Chen

arXiv:2101.08387v6cs.LGcs.AI

TL;DR

Ensemble deep learning can improve generalization but incurs far greater training and testing costs than traditional ensemble learning. This survey analyzes published-work trends and methodologies to identify the development gap, intrinsic problems, and technical challenges affecting broader deployment.

  • Problem

    Ensemble deep learning offers generalization benefits, but its substantially higher training and testing overhead than traditional ensemble learning limits deployment across applications.

  • Method

    The article synthesizes published-work data analyses and fundamental discussions of traditional and deep ensemble learning methodologies, advances, and unattainability.

  • Results

    Published-work analyses show ensemble learning remains prosperous, while ensemble deep learning research severely lags behind traditional ensemble learning because of much larger training and testing overheads.

  • Takeaways & Limitations

    The survey identifies intrinsic problems and technical challenges that must be addressed for ensemble learning to reach a wider range of specific fields in the deep-learning era.

  • Takeaways & Limitations

    Federated ensemble deep learning remains imperfectly defined, and it is unclear whether additional learners related to a pretrained learner produce an optimal ensemble.

Abstract

from arXiv · show

Due to the dominant position of deep learning (mostly deep neural networks) in various artificial intelligence applications, recently, ensemble learning based on deep neural networks (ensemble deep learning) has shown significant performances in improving the generalization of learning system. However, since modern deep neural networks usually have millions to billions of parameters, the time and space overheads for training multiple base deep learners and testing with the ensemble deep learner are far greater than that of traditional ensemble learning. Though several algorithms of fast ensemble deep learning have been proposed to promote the deployment of ensemble deep learning in some applications, further advances still need to be made for many applications in specific fields, where the developing time and computing resources are usually restricted or the data to be processed is of large dimensionality. An urgent problem needs to be solved is how to take the significant advantages of ensemble deep learning while reduce the required expenses so that many more applications in specific fields can benefit from it. For the alleviation of this problem, it is essential to know about how ensemble learning has developed under the era of deep learning. Thus, in this article, we present fundamental discussions focusing on data analyses of published works, methodologies, recent advances and unattainability of traditional ensemble learning and ensemble deep learning. We hope this article will be helpful to realize the intrinsic problems and technical challenges faced by future developments of ensemble learning under the era of deep learning.

1 Introduction

Ensemble learning improves generalization by combining multiple base learners, but ensemble deep learning incurs much higher training and testing costs because modern neural networks contain millions to billions of parameters. This survey analyzes published works, methodologies, advances, and limitations to identify intrinsic problems and technical challenges for deploying ensemble learning more broadly under deep learning.

  • Motivation: Ensemble learning combines multiple base learners to achieve better generalization and has succeeded across diverse artificial intelligence applications.Increasing the number of base learners also increases the costs of training and testing the ensemble learner.
  • Motivation: Millions to billions of neural-network parameters make ensemble deep learning’s training and testing overheads far greater than those of traditional ensemble learning.Snapshot, fast geometric ensembling, and stochastic weight averaging are identified as fast ensemble deep learning algorithms.
  • Survey scope: The survey examines existing ensemble learning algorithms and related applications to understand how ensemble learning developed during the deep-learning era.Its stated aim is to reveal intrinsic problems and technical challenges for deploying ensemble learning in a wider range of specific fields.
  • Survey contributions: Published-work analysis shows ensemble learning’s prosperity and the gap between ensemble deep learning and traditional ensemble learning.The survey also discusses development routes, methodologies, recent advances, and unattainability in both traditional ensemble learning and ensemble deep learning.

2 Data Analysis of Published Works

This section analyzes publication data to illustrate ensemble learning’s prosperity and the gap between traditional ensemble learning and ensemble deep learning. It traces EDL’s slow progress before 2014 and rapid growth afterward alongside deep neural networks, big data, and high-performance computing.

  • 2 Data Analysis of Published Works: Publication analysis examines ensemble learning’s prosperity and the gap between traditional ensemble learning and ensemble deep learning.The section uses published-work data to compare these research areas.
  • 2 Data Analysis of Published Works: Ensemble learning research history traces back to 1990 and has remained a research hotspot because of its advantages in improving learning-system generalization.The field’s theoretical algorithms and applications have been actively studied for about 30 years.
  • 2 Data Analysis of Published Works: Research on ensembles of neural networks also traces back to 1990, but ensemble deep learning progressed very slowly before 2014.EDL refers to ensembles of deep neural networks.
  • 2 Data Analysis of Published Works: After 2014, ensemble deep learning research began growing rapidly, supported by deep neural networks’ success, big-data popularization, and high-performance computing resources.Figure 2 compares publication counts for traditional ensemble learning and ensemble deep learning from 1990 to 2019.

3 Traditional Ensemble Learning

This section presents traditional ensemble learning (TEL) within the usual machine learning paradigm, covering its methodology, well-known implementations, recent advances, and unattainability.

  • TEL has played a major role in the research history of ensemble learning.
  • The section introduces TEL from the paradigm of usual machine learning.
  • It discusses TEL methodology, well-known implementations, recent advances, and unattainability.

3.1 Preliminary

Usual machine learning (UML) operates on raw data and corresponding targets through two essential components: feature extraction and model development. Feature extraction creates learnable representations, while model development evolves a learner mapping those representations to targets.

  • Usual machine learning: UML starts with a raw data set D and corresponding target set T, where each raw data point has a matching target.The paradigm defines D = {d_1, ⋯, d_n} and T = {t_1, ⋯, t_n}.
  • Feature extraction: Feature extraction uses extracting methods and a converting procedure to transform raw data points into learnable representations.The extracting methods obtain representation types, and the converting procedure incorporates their results across the raw data set.
  • Model development: Model development comprises a learning algorithm, learning strategy, and evolving procedure that construct and update the learner.The optimization procedure updates learner parameters under an objective function for classification or regression.
  • Testing: The evolved learner forms a mapping between representations F and corresponding targets T and predicts a target for each test raw data point.At testing, the learner produces the predicted target corresponding to d_test.

3.2 Methodology of TEL

Traditional ensemble learning (TEL) extends UML by generating diverse base learners and integrating them into an ensemble learner after feature extraction. Its methodology distinguishes heterogeneous and homogeneous base learners and uses ensembling criteria plus an integrating procedure to configure the final ensemble.

  • Methodology of TEL: TEL replaces UML’s model development with generating multiple diverse base learners and forming an ensemble learner.The methodology includes feature extraction, base-learner generation, and ensemble formation.
  • Generating base learners: Base learners are heterogeneous when generated with different learning algorithms and homogeneous when generated with the same learning algorithm.Learning strategies evolve the resulting base learners under the selected learning algorithms.
  • Forming ensemble learner: Forming the ensemble learner comprises an ensembling criterion and an integrating procedure.The ensembling criterion constructs and configures the ensemble learner, while the integrating procedure applies that configuration.
  • Forming ensemble learner: Building appropriate ensembling criteria is the key point in forming the ensemble learner, while heterogeneous-base-learner strategies are usually simpler than homogeneous-base-learner strategies.The criteria govern ensemble construction and configuration from base-learner predictions.

3.3 Well-known implementations for methodology of TEL

This section surveys well-known TEL methodologies for generating diverse, accurate base learners and forming ensemble learners. It organizes base-learner strategies by data- and feature-level manipulation and ensembling criteria into weighting, meta-learning, and ensemble selection methods.

  • Scope: TEL methodology primarily addresses generating base learners and forming ensemble learners, while weakening feature-extraction discussion as beyond its scope.The section emphasizes learning strategies rather than machine-learning algorithms, which are treated as a topic more appropriate in UML.
  • Generating base learners: Base learners should be sufficiently diverse while maintaining high individual prediction accuracy.Both the learning algorithm and learning strategy must be considered when generating multiple base learners.
  • Generating base learners: Data-level strategies create diversity by repeatedly sampling or reweighting training sets, including Bagging, Dagging, and Boosting.Bagging uses random sampling, Dagging samples disjoint datasets without replacement, and Boosting reweights training samples according to performance.
  • Generating base learners: Feature-level manipulation samples training features as inputs for different base learners, while Random Forests combine data- and feature-level sampling.Other diversity strategies encourage negative correlation through complementary learning or use different random initializations for ANNs.
  • Forming ensemble learners: Ensembling criteria comprise weighting, meta-learning, and ensemble selection methods for combining base-learner outputs.Weighting assigns different output weights, meta-learning maps base-learner predictions and targets to reduce generalization error, and ensemble selection limits the number of base learners to reduce prediction-time and storage costs.

3.4 Recent advances of TEL

This section reviews highly cited recent traditional ensemble learning advances across diverse AI applications, emphasizing feature representations, base-learner generation, learning strategies, and ensemble integration. Across the reviewed works, hand-crafted features, classic learning algorithms, and data- or feature-level resampling were predominant.

  • Selection of recent TEL advances: The review selects highly cited TEL advances because hundreds or thousands may be proposed annually, using Web of Science searches with defined filtering rules.The search was accessed on February 3, 2021, and focused on works published during the preceding ten years.
  • Review framework: The TEL methodology is expanded beyond learner generation and ensemble criteria to describe feature-representation extraction and base-learner algorithms needed for specific applications.The review organizes these aspects through Tables 3–7, covering reviewed works, feature extraction, learning algorithms, learning strategies, and ensembling criteria.
  • Application advances: Reviewed TEL advances span applications including 3D human action recognition, protein-protein interaction prediction, DNase I site identification, RNA-site detection, landslide mapping, and load forecasting.The approaches combine application-specific features with methods such as Fourier temporal pyramids, ELMs, random forests, SVMs, heterogeneous learners, and wavelet transforms.
  • Application advances: Additional reviewed advances address multi-class imbalance, soil-moisture forecasting, flood susceptibility mapping, and biomedical lung-cancer detection using decomposition, decomposition-based coding, Bagging, and voting strategies.These works employ DECOC, EEMD/CEEMDAN, KNN ensembles, generalized neural networks, and majority voting.
  • Observed development trends: Most reviewed TEL approaches used hand-crafted features, frequently applied SVM, KNN, and neural networks, and commonly manipulated data or feature levels through resampling.The frequently listed neural-network algorithms included ELM, ANN, MLP, and GNN.

3.5 Unattainability

Section 3.5 discusses TEL’s unattainability, noting that recent advances largely target specific applications through combinatorial innovations rather than fundamentally new learning strategies.

  • 3.5 Unattainability: Most recent TEL advances propose solutions for specific applications by combining existing learning strategies for base-learner generation with existing ensembling criteria.These advances primarily rely on combinatorial innovations.
  • 3.5 Unattainability: Few recent TEL advances introduce new learning strategies for generating base learners or new approaches for forming ensemble learners.The supplied passage identifies this as a limitation of recent TEL research.

4 Usual Ensemble Deep Learning

Usual ensemble deep learning (UEDL) evolves traditional ensemble learning by introducing deep learning through three basic patterns: deep feature extraction, deep base-learner generation, and deep ensemble-learner formation. Recent UEDL work primarily follows these patterns, but retains traditional ensemble-learning structure and therefore increases training time and space demands.

  • Methodology: UEDL introduces deep learning into traditional ensemble learning through feature extraction, base-learner generation, and ensemble-learner formation.The three patterns can also be combined, and the final learner used for testing is generally regarded as an ensemble deep learner.
  • Methodology: Pattern A uses pretrained or unsupervised-trained deep models to extract more expressive features from raw data.The extracted deep representations can then support subsequent ensemble-learning procedures.
  • Recent advances: Five reviewed works employed pattern A, while eight employed pattern B to replace traditional learning algorithms with deep learning algorithms for generating base deep learners.Pattern B can generate heterogeneous or homogeneous deep base learners, using raw data or extracted features.
  • Unattainability of UEDL: UEDL retains the traditional ensemble-learning paradigm while introducing deep learning, considerably increasing the time and space demands of training multiple base learners.Although some learning strategies and ensemble criteria are tailored to UEDL, this retained structure is identified as its primary methodological issue.

5. Fast Ensemble Deep Learning

Fast ensemble deep learning (FEDL) incorporates deep-learning characteristics to reduce ensemble deep learning’s time and space overheads. Existing approaches improve deployment but remain costly, while their definitions and ensemble optimality require further development.

  • 5. Fast Ensemble Deep Learning: FEDL reduces ensemble deep learning overheads by exploiting the intrinsic characteristics of deep learning rather than retaining traditional ensemble learning’s paradigm.It emerges specifically to address the time and space costs of ensemble deep learning.
  • 5. Fast Ensemble Deep Learning: Snapshot and FGE reduce training time by generating multiple base deep learners along stochastic-gradient-descent optimization paths, but testing still requires storing and averaging their predictions.SWA addresses testing expenses by fusing learners through weight averaging after local minima accumulate near low-value loss regions.
  • 5. Fast Ensemble Deep Learning: FEDL comprises pre-training a base deep learner, finding local minima to generate additional learners, and forming an ensemble in model or parameter space.Parameter-space fusion combines multiple base deep learners into a single deep learner.
  • 5. Fast Ensemble Deep Learning: LoMiFoSS-FEDL reduces FEDL’s burden by finding and integrating local minima in a sub-parameter space.Its methodology retains pre-training, base-learner generation, and ensemble formation, with testing equivalent to parameter-space fusion.
  • 5. Fast Ensemble Deep Learning: FEDL remains limited by high expenses in resource-constrained or large-dimensional applications and by uncertainty that related base learners yield the optimal ensemble solution.The survey calls for clearer definitions and methodologies that may use less tightly related learners.

6 Discussion

The discussion finds that ensemble learning continues to develop, but ensemble deep learning (EDL) lags because its training and testing costs are much larger than traditional ensemble learning (TEL). It reviews methodological progress, identifies limitations in usual and fast EDL, and highlights future directions combining TEL ideas with deep learning.

  • 6 Discussion: Published-work analyses show ensemble learning is prosperously developing, whereas EDL research severely lags behind TEL research because EDL has much larger training and testing overheads.The stated overhead difference concerns both time and space, and applies to training and testing stages.
  • 6 Discussion: Recent TEL advances have improved solutions across applications, but mostly combine existing base-learner generation strategies with existing ensemble-formation criteria rather than introducing new strategies or criteria.The discussion characterizes these advances as application-specific and based on combinatorial innovations.
  • 6 Discussion: Usual EDL advances introduce deep learning into feature extraction, base-learner generation, or ensemble-learner formation, but retain TEL’s paradigm and substantially increase training costs.These three patterns have promoted solutions for various applications, while retaining the multiple-base-deep-learner paradigm.
  • 6 Discussion: FEDL aims to reduce EDL costs more intrinsically by exploiting inherent characteristics of deep learning, but its current alternative methodologies provide an incomplete definition.The discussion identifies more appropriate and comprehensive definitions of FEDL as a promising future direction.
  • 6 Discussion: A reverse development direction is to introduce TEL ideas into deep learning for FEDL, exemplified by deep negative correlation learning using a divide-and-conquer strategy.This direction contrasts with UEDL’s introduction of deep learning into TEL.
Loading 2101.08387v6…