Source-linked AI summary
The Relative Performance of Ensemble Methods with Deep Convolutional Neural Networks for Image Classification
Cheng Ju, Aurélien Bibaut, Mark J. van der Laan
TL;DR
The paper addresses limited comparative evidence about ensembles of deep neural networks for image recognition. It evaluates several ensemble methods across CNN candidate configurations and examines over-confidence, finding that the Super Learner performed best among the studied methods.
Problem
Few studies had fully investigated or compared ensemble methods for deep neural networks in image recognition.
Method
The study empirically compares unweighted averaging, majority voting, the Bayes Optimal Classifier, and the discrete Super Learner across CNN libraries with varied checkpoints, stochastic training runs, and structures.
Results
Across the experiments, the Super Learner achieved the best performance among the ensemble methods studied.
Takeaways & Limitations
Data-adaptive weighting lets the Super Learner combine candidate models while addressing weaknesses from weak or over-confident learners.
Takeaways & Limitations
Because CNNs are computationally intensive, the study used a single validation split to compute Super Learner weights instead of conventional multiple-fold cross-validation.
Abstract
from arXiv · showhide
Artificial neural networks have been successfully applied to a variety of machine learning tasks, including image recognition, semantic segmentation, and machine translation. However, few studies fully investigated ensembles of artificial neural networks. In this work, we investigated multiple widely used ensemble methods, including unweighted averaging, majority voting, the Bayes Optimal Classifier, and the (discrete) Super Learner, for image recognition tasks, with deep neural networks as candidate algorithms. We designed several experiments, with the candidate algorithms being the same network structure with different model checkpoints within a single training process, networks with same structure but trained multiple times stochastically, and networks with different structure. In addition, we further studied the over-confidence phenomenon of the neural networks, as well as its impact on the ensemble methods. Across all of our experiments, the Super Learner achieved best performance among all the ensemble methods in this study.
1 Introduction
Ensemble methods combine multiple baseline models, but the best method depends on the learners, data, and prediction problem. The paper highlights limited understanding of ensembles for deep neural networks and the weaknesses of naive averaging with heterogeneous or over-confident candidates.
- Ensemble methods combine several baseline models using rules to produce predictions.
- A learner’s relative performance depends on model assumptions, the data-generating distribution, sample size, dimensionality, and the bias-variance trade-off.
- Cross-validation can assess learners honestly, but selecting one best algorithm does not exploit combinations of predictors.
- Deep neural networks have achieved strong results across computer vision, machine translation, and social network analysis, while generally exhibiting high variance and low bias.
- Ensemble behavior with deep networks remained insufficiently studied, and naive unweighted averaging was not data-adaptive for bad or heterogeneous learner libraries.
- The study compares ensemble methods for CNN-based image recognition, treating each CNN as a black-box estimator that maps images to class-probability vectors.
2 Background
The background introduces common ensemble strategies and their assumptions, then describes Bayesian voting and stacking as data-combination approaches. It emphasizes that averaging and voting can be vulnerable to weak, dependent, or over-confident learners.
- An algorithm candidate, hypothesis, or base learner denotes one deep CNN, while a library is the set of base learners used by an ensemble.
- Unweighted Average: Unweighted averaging combines the output scores or probabilities of all base learners without learned weights.
- Unweighted Average: Averaging can reduce variance when deep networks are sufficiently uncorrelated, but averaging after softmax avoids differences in score scale.
- Unweighted Average: Unweighted averaging is most reasonable for similar, comparably performing learners and is vulnerable to weak or over-confident candidates in heterogeneous libraries.
- Majority Voting: Majority voting selects the label receiving the most base-learner votes, but discards probability information and can be dominated by similar dependent learners.
- Bayes Optimal Classifier: The Bayes Optimal Classifier treats each learner as a hypothesis about P(y|x), weighting it using a prior and its validation-set likelihood.
- Stacking and Bayesian Voting: Stacking learns combination weights with a meta-learner, whereas Bayesian weights can become concentrated on one hypothesis when validation samples are large.
3 Super Learner: a Cross-validation based Stacking
The Super Learner extends stacking by optimizing ensemble weights using cross-validated loss, with a single-split variant suited to computationally intensive deep-learning settings. Its weights can be constrained and implemented as a 1×1 convolution over base-learner outputs.
- Super Learner: Super Learner extends stacking by combining base learners through cross-validation while minimizing cross-validated risk.
- Cross-validated Loss: The cross-validated loss evaluates each base learner on validation-fold predictions generated without training on that fold.
- Weight Optimization: The method learns an optimal weight vector for combining predictions from multiple base learners.
- Loss Function: For binary classification, the basic Super Learner uses negative Bernoulli log-likelihood as its loss function.
- Neural-Network Outputs: Stacking can operate before softmax, and the framework also supports score-level ensembling for K-class neural-network outputs.
- Single-Split Variant: A single-split Super Learner optimizes weights on a held-out validation set instead of using multiple-fold cross-validation when large deep-learning datasets and libraries make cross-validation costly.
- Neural-Network Perspective: The Super Learner can be viewed as a neural network with a 1×1 convolution whose kernel weights the base learners’ scores.
4.1 Data
The experiments use CIFAR-10, a 10-class image-recognition benchmark with 50,000 training images and 10,000 testing images.
- CIFAR-10 contains 10 classes of natural RGB images sized 32×32.
- The dataset provides 50,000 training images and 10,000 testing images.
- Each class contributes 5,000 training images and 1,000 testing images.
4.2 Network description
The study uses several deep convolutional architectures as candidate learners, including NIN, GoogLeNet, VGG, and Residual Network variants for CIFAR-10.
- Network architectures: NIN uses MLP convolutional layers, global average pooling, and no conventional fully connected layers.Each MLP layer combines a larger-kernel convolution, two 1×1 convolutions, and max pooling.
- Network architectures: GoogLeNet uses inception modules that reduce dimensions with 1×1 convolutions, split propagation into four flows, and concatenate them.
- Network architectures: VGG uses consecutive 3×3 convolutions within blocks, max pooling, increasing filter counts, and three fully connected layers before softmax.
- Network architectures: The experiments use VGG net D with 16 layers and refer to it simply as VGG net.
- Network architectures: Residual Network stacks bottleneck blocks with 1, 3, and 1 convolutional kernels plus parameter-free identity shortcuts.
- Network architectures: For CIFAR-10, the Residual Network uses 6n convolutional layers across feature-map sizes 32, 16, and 8, plus softmax.For example, n = 5 gives 32 total layers.
4.3 Training
Training separates CIFAR-10 into training, validation, and testing data, then trains each architecture with its own optimizer and regularization settings for 200 epochs.
- Data split: The data split uses 45,000 training images, 5,000 validation images, and 10,000 testing images.
- Optimization and regularization: NIN training uses Adam with learning rate 0.001, L-2 penalty weight 0.001, and dropout rate 0.5.
- Optimization and regularization: VGG training uses SGD with momentum 0.9, an initial learning rate of 0.01, scheduled decay, L-2 penalty weight 10−3, and dropout rate 0.5.
- Optimization and regularization: GoogLeNet training uses learning rate 0.05, weight decay 10−3, momentum 0.9, 4% learning-rate decay every 8 epochs, and dropout rate 0.4.
- Optimization and regularization: Residual Network training uses SGD with weight decay 0.0001, momentum 0.9, batch normalization, no dropout, and learning-rate division by 10 every 32k iterations.
- Training schedule: All networks use mini-batches of 128 and are trained for 200 epochs.
4.4 Results
The experiments compare ensemble methods across checkpoint, repeated-training, heterogeneous-network, over-confidence, and weak-learner settings. Super Learner was generally the strongest method, while naive averaging and majority voting were vulnerable to poorly optimized, weak, or over-confident candidates.
- The study compares unweighted averaging, majority voting, the Bayes Optimal Classifier, and Super Learner variants, using the best testing-set base learner as an empirical oracle.
- Ensemble of Same Network with Different Training Checkpoints: For ResNet 8 checkpoints, Super Learner substantially outperformed naive averaging and majority voting because early learners were under-optimized.For ResNet 110, meta-learners performed similarly, possibly because the deeper network was more stable during training.
- Ensemble of Same Network Trained Multiple Times: 2.54% improvement for shallow networks versus 1.43% for deeper networks was achieved after ensembling repeated stochastic trainings with Super Learner.Discrete Super Learner with negative log-likelihood selected the best single learner, whereas its error-loss version selected a slightly weaker one.
- Ensemble of Networks with Different Structures: Network-in-Network showed over-confidence: it had lower testing cross-entropy but worse prediction accuracy than the other heterogeneous networks.Its training cross-entropy was 0.13104 versus 0.02233 for VGG, so over-confidence was not reasonably attributable solely to over-fitting.
- Over-confident Model: 0.23% before softmax and 5% after softmax were the prediction-accuracy decreases caused by adding over-confident NIN to unweighted averaging.Super Learner instead increased accuracy from 0.9405 to 0.9414 when NIN was added.
- Prediction with All Candidates: Adding many weak candidates deteriorated unweighted averaging and could dominate majority voting, while Super Learner accuracy remained stable.Across libraries containing weak and over-confident learners, Super Learner computed weights data-adaptively without tedious human pre-selection.
4.5 Discussion
The study compares ensemble methods for deep CNNs on CIFAR-10 and finds that the Super Learner addresses weaknesses of unweighted averaging through data-adaptive validation-set weighting.
- The experiments compare widely used ensemble methods with deep convolutional neural networks on CIFAR-10.
- Unweighted averaging performs well when base learners have comparable performance but is sensitive to over-confident candidates.
- The Super Learner optimizes base-learner weights on a validation set in a data-adaptive manner.
- The Super Learner uses a simple structure that can be viewed as a 1 × 1 convolution layer stacked on the base learners’ outputs.
- Because it adaptively assigns weights, the Super Learner can improve predictions even when the library includes weak learners.
- The implementation uses a validation set instead of conventional multiple-fold cross-validation because CNNs are computationally intensive and image-recognition validation sets are typically large.