PaperlayerView exact revision on arXiv

Source-linked AI summary

Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks

Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, Dhruv Batra

arXiv:1511.06314v1cs.CVcs.LGcs.NE

TL;DR

CNN ensembling is usually a post-hoc average of independently trained models, leaving the best way to create diverse ensembles underexplored. The paper systematically compares strategies, introduces TreeNets and diversity-aware training, and reports improved ensemble performance and oracle accuracy. Its end-to-end coupled approaches also face training-cost and data-fragmentation constraints.

  • Problem

    The paper addresses the limited treatment of CNN ensembling as an explicit problem for determining effective strategies to create ensembles.

  • Method

    The authors compare ensembling strategies and introduce TreeNets, ensemble-aware losses, diversity-encouraging losses, and a distributed model-parallel training framework.

  • Results

    TreeNets improve ensemble performance, while diverse ensembles trained end-to-end with a unified loss achieve significantly higher oracle accuracies than classical ensembles.

  • Takeaways & Limitations

    The findings indicate that high-level diversity is important for ensemble performance, whereas low-level filters may be shared across members.

  • Takeaways & Limitations

    Coupled ensemble training creates computational difficulties, and data fragmentation can hinder diversity-encouraging training, especially for larger networks.

Abstract

from arXiv · show

Convolutional Neural Networks have achieved state-of-the-art performance on a wide range of tasks. Most benchmarks are led by ensembles of these powerful learners, but ensembling is typically treated as a post-hoc procedure implemented by averaging independently trained models with model variation induced by bagging or random initialization. In this paper, we rigorously treat ensembling as a first-class problem to explicitly address the question: what are the best strategies to create an ensemble? We first compare a large number of ensembling strategies, and then propose and evaluate novel strategies, such as parameter sharing (through a new family of models we call TreeNets) as well as training under ensemble-aware and diversity-encouraging losses. We demonstrate that TreeNets can improve ensemble performance and that diverse ensembles can be trained end-to-end under a unified loss, achieving significantly higher "oracle" accuracies than classical ensembles.

1. Introduction

The paper treats CNN ensembling as an explicit design problem, evaluating parameter sharing and ensemble-aware losses alongside conventional approaches. It introduces TreeNets and a distributed training framework to study how sharing and diversity affect ensemble performance.

  • CNN ensembles commonly lead vision benchmarks, and GoogLeNet improved accuracy by five percentage points over a single base model of the same architecture.
  • The paper compares ensembling strategies from standard bagging to parameter sharing and ensemble-aware losses across datasets and architectures.
  • TreeNets share a variable number of initial layers across ensemble members, exploiting generic lower-level features while preserving deeper member-specific representations.
  • Coupling ensemble members through shared parameters or unified losses creates computational difficulties because the networks can no longer be trained independently in parallel.
  • The authors introduce TreeNets, ensemble-aware and diversity-encouraging losses, and a distributed model-parallel framework for training coupled ensembles.

2. Related Work

Prior ensemble work induces diversity through initialization, bootstrapped data, or explicit decorrelation, while CNN ensembles have received less attention as an ensembling-process problem. This paper applies these ideas to deep CNNs and introduces end-to-end learned specialization without manually designed class subsets.

  • Ensemble-learning theory and experiments link improved performance to diversity in member error distributions.
  • Traditional methods create diversity indirectly through different parameter initializations or bootstrapped training sets, while other methods penalize correlated errors.
  • Prior decorrelation methods were effective on shallow networks but had not been applied to deeper architectures.
  • The paper generalizes Multiple Choice Learning to CNNs so members can specialize over class subsets or feature-space regions without human-designed specialization.
  • Most CNN ensembles use random initializations, data subsets, or both, with examples including VGG, AlexNet, GoogLeNet, and sequence-to-sequence RNN ensembles.

3. Experimental Design

The experiments evaluate ensembling methods on CIFAR10, CIFAR100, and ILSVRC using standard CNN architectures and training routines. Performance is assessed through both averaged ensemble predictions and an oracle selecting the best member per example.

  • Datasets and Architectures: The study evaluates three image-classification benchmarks: CIFAR10, CIFAR100, and the 2012 ILSVRC.
  • Datasets and Architectures: CIFAR10 uses the Caffe CIFAR10 Quick network, while CIFAR100 uses Network in Network with the authors’ reference training procedure.
  • Datasets and Architectures: ILSVRC experiments use Network in Network and CaffeNet, trained with stochastic gradient descent using dataset-specific batch sizes and learning-rate schedules.
  • Evaluation Metrics: Ensemble-Mean Accuracy measures predictions formed by averaging member beliefs and selecting the most confident class.
  • Evaluation Metrics: Oracle Accuracy measures performance when an oracle selects the prediction of the most accurate ensemble member for each example.

4. Random Initialization and Bagging

Random initialization and bagging both improve over single models, but bagging can reduce ensemble-mean accuracy because resampling removes unique training examples. For deep networks, random initialization may therefore be preferable to bagging.

  • Standard Ensembling Techniques: The comparison evaluates Random Initialization, Bagging, and Combined ensembles, varying parameter initialization, sampled training data, or both.
  • Results: All ensembles outperform their single base models, but bagging reduces Ensemble-Mean Accuracy relative to random initialization while Oracle Accuracy remains nearly constant.
  • Results: Bagging can produce poorly calibrated networks because confident incorrect predictions negatively affect averaged ensemble results.
  • Data Effects: Sampling M examples with replacement retains approximately 63% of unique examples for large M, so bagging discards over a third of the available unique data.
  • Data Effects: Experiments using 31,500 unique examples reproduced similar accuracy reductions, indicating that lost unique data drives bagging’s negative effect.
  • Conclusion: For deep networks, random initialization may be sufficient and preferred over bagging because their large parameter spaces can generate diversity without sacrificing training data.

5. Parameter Sharing with TreeNets

TreeNets share early CNN layers while branching into independent deeper layers, placing them between single models and fully independent ensembles. Experiments show they can preserve or improve ensemble accuracy with fewer parameters and transfer to object detection.

  • TreeNet design: TreeNets share initial layers before branching into independent layers, reducing duplicated computation while retaining separate ensemble members.Each root-to-leaf path acts as a network, and shared-layer computation need not be repeated at test time.
  • Results: 64.08 ±0.00 ensemble accuracy was reported for one listed configuration, compared with 58.90 ±0.13 for a single model.The listed configuration also includes split-point results such as conv1 at 65.50 ±0.24 and cccp1 at 65.69 ±0.08.
  • TreeNet design: Table 2 evaluates TreeNet ensembles with branch points at different parameterized depths for ILSVRC-Alex and ILSVRC-NiN.Splitting at conv2 means layers through conv2 are shared and later layers are independent.
  • Results: TreeNets can outperform full ensembles while reducing parameter count; the best ILSVRC-NiN TreeNet improved standard-ensemble accuracy with 7% fewer parameters.The authors suggest shared low-level weights receive supervision from multiple branches, potentially improving low-level representations.
  • Results: TreeNet branches achieved around 2 to 3 percentage points higher accuracy than independently trained ensemble members across split points.This contrasts with classical ensembles, whose members perform about as well as the base architecture.
  • Results: On PASCAL VOC 2007 object detection, TreeNet initialization increased mean average precision by about 0.7% versus starting from a standard ensemble.The increase was statistically significant across multiple runs.

6. Training Under Ensemble-Aware Losses

Optimizing ensemble-average outputs can reduce performance by eliminating gradient diversity, whereas an explicit diversity-encouraging MCL loss produces specialized ensembles with substantially higher oracle accuracy. The experiments also reveal a trade-off: stronger specialization can reduce Ensemble-Mean Accuracy.

  • Optimizing averaged scores or probabilities reduced performance compared with independently trained models, with probability averaging degrading more.The authors attribute this to reduced diversity and, for probability averaging, numerical instability.
  • Averaging outputs gives all ensemble members identical back-propagated gradients, sharing responsibility for mistakes and eliminating gradient diversity.This differs from independently trained networks, whose gradients vary with individual performance.
  • Adding Diversity via Multiple Choice Learning: The MCL loss assigns each example to its lowest-loss predictor, directly encouraging ensemble members to specialize and diversify.Its oracle set-loss is the minimum loss over predictors; during optimization, examples are assigned to their most accurate predictors before model training.
  • Adding Diversity via Multiple Choice Learning: At k=1, individual members achieved only 19-27% accuracy while ensemble oracle accuracy exceeded 93%, demonstrating strong specialization across examples.Increasing k exposes each member to more data and reduces oracle accuracy as the diversifying effect weakens; k=4 is a standard ensemble.
  • Adding Diversity via Multiple Choice Learning: MCL assignments produced an almost complete, emergent division of the label space at k=1, becoming more uniform as k increased.The divisions were learned from the loss rather than hand-designed or pre-initialized, and nonspecialized networks were agnostic to relevant image content.
  • Adding Diversity via Multiple Choice Learning: A linearly combined MCL and cross-entropy loss improved CIFAR10 Ensemble-Mean Accuracy by 1% over a standard ensemble while balancing diversity with general performance.Pure MCL induced enough specialization that Ensemble-Mean Accuracy suffered.

7. Distributed Ensemble Training

The paper introduces MPI-Caffe to support distributed ensemble training across GPUs and machines, addressing the communication required when learners are coupled. Its MPI layers implement cross-process data and gradient exchange, while measurements indicate low communication overhead for a large shared layer.

  • MPI-Caffe enables coupled ensemble learners to communicate across GPUs and machines using the MPI standard.The framework was developed to make large-scale ensemble experiments feasible when training on a single GPU is prohibitively expensive.
  • MPIBroadcast forwards inputs across processes and accumulates their gradients during back-propagation.This layer provides communication in both the forward and backward passes.
  • A single network specification can define MPI-Caffe models distributed across multiple GPUs.Figure 3 distinguishes cross-process communication from network input/output.
  • MPIGather collects inputs from multiple processes into one network and routes backward gradients to the corresponding inputs.Its forward and backward operations connect distributed network components while preserving gradient routing.
  • 0.49% of the forward-backward pass time was spent communicating when broadcasting nearly 36 million pool2 floats per batch.This measurement was obtained on a cluster using one Tesla K20 GPU per node and a maximum interconnect bandwidth of 5.8 GB/sec.

8. Discussion and Conclusion

The paper’s experiments identify diversity as the central factor governing ensemble construction. The authors conclude that diversity is most useful in high-level representations, while shared low-level filters can improve efficiency, and they leave structured prediction for future work.

  • Discussion and Conclusion: Diversity is the paper’s unifying theme across bagging, ensemble-aware losses, MCL, and TreeNets.
  • Discussion and Conclusion: Random parameter initialization provides more useful ensemble diversity than bags containing duplicated examples.
  • Discussion and Conclusion: Averaging members’ beliefs before computing losses reduces gradient diversity, whereas the diversity-inducing MCL loss can significantly improve performance.
  • Discussion and Conclusion: TreeNets indicate that diversity matters in high-level representations, while low-level filters may benefit from being shared.
  • Discussion and Conclusion: Future work would adapt MCL to structured prediction problems, where diverse models may provide greater benefit.

Appendix A. TreeNet Object Detection Results on PASCAL VOC 2007

The appendix evaluates TreeNets for PASCAL VOC 2007 object detection with Fast R-CNN. Across both bounding-box evaluation settings, TreeNets outperform standard ensembles and single models by significant margins.

  • Appendix A. TreeNet Object Detection Results on PASCAL VOC 2007: The PASCAL VOC 2007 evaluation uses Fast R-CNN with classification and bounding-box regression losses on 20 object classes.
  • Appendix A. TreeNet Object Detection Results on PASCAL VOC 2007: The study fine-tunes four instances of each model and reports classwise average precision statistics and mean average precision.
  • Appendix A. TreeNet Object Detection Results on PASCAL VOC 2007: Table 7 compares a standard ensemble, TreeNets split after conv1, conv2, and conv3, and a single model using averaged bounding boxes.
  • Appendix A. TreeNet Object Detection Results on PASCAL VOC 2007: In both tasks, TreeNets outperform standard ensembles and single models by significant margins.
  • Appendix A. TreeNet Object Detection Results on PASCAL VOC 2007: Table 8 reports average precision for TreeNet models when predicted bounding boxes are not used.

Appendix B. Instability of Averaged Softmax Outputs

The appendix explains why training with cross-entropy over averaged softmax outputs underperforms standard and score-averaged ensembles. Averaging probabilities introduces predictor-specific gradient weighting, producing less stable optimization behavior.

  • Appendix B. Instability of Averaged Softmax Outputs: Training under cross-entropy over averaged softmax outputs reduces performance relative to standard ensembles and score-averaged ensembles.
  • Appendix B. Instability of Averaged Softmax Outputs: The softmax derivative can involve multiplying very small probabilities, creating underflow risks that the cross-entropy derivative avoids.
  • Appendix B. Instability of Averaged Softmax Outputs: For averaged probabilities over M predictors, the gradient for each predictor receives an additional weighting factor ranging from 0 to M.
  • Appendix B. Instability of Averaged Softmax Outputs: The averaged-softmax derivative retains the standard term but multiplies it by predictor-specific weighting, which can make gradients less stable.
  • Appendix B. Instability of Averaged Softmax Outputs: When a predictor assigns little probability to the correct class relative to other predictors, its gradient weighting approaches zero.
  • Appendix B. Instability of Averaged Softmax Outputs: The appendix describes MCL training as alternating loss computation, example partitioning, and gradient updates using each predictor’s lowest-loss subset.
  • Appendix B. Instability of Averaged Softmax Outputs: Classical MCL initializes assignments with k-means and trains each predictor to completion on its corresponding subset before reassignment.

Appendix D. Visualizations for MCL Trained Ensembles

The appendix visualizes how MCL-trained ensemble members differ and documents MPI-Caffe mechanisms for constructing distributed TreeNet ensembles. It describes broadcast and gather layers, their network-parsing behavior, usage constraints, and supported distributed configurations.

  • Visualizations for MCL Trained Ensembles: MCL visualizations track how CIFAR10 class examples are distributed across predictors during training, with many classes stabilizing after additional iterations.Figure 4 contrasts the class distribution early in training with its evolution after 220 iterations.
  • Visualizations for MCL Trained Ensembles: Guided-backprop reconstructions compare standard-ensemble models with MCL-trained members across output, conv1, conv2, and conv3 layers.The visualizations represent gradients of neuron outputs with respect to input images, highlighting features influencing network outputs.
  • MPI-Caffe Implementation: MPI-Caffe exposes cross-GPU and cross-machine communication as model layers, enabling distributed networks, extremely large models, and ensemble-aware model parallelism.The framework includes MPIBroadcast and MPIGather layers and modifies Caffe’s initialization and supporting components.
  • MPI Layer Behavior: A three-process CIFAR10-Quick TreeNet example uses MPI-Caffe to implement parameter sharing and score-averaged training across GPUs.The example modifies a LeNet structure and uses broadcast and gather layers to coordinate the distributed ensemble.
  • MPI Layer Behavior: MPIBroadcast copies an input blob to processes during the forward pass and sums copy gradients during backpropagation, while MPI-Gather collects process-local blobs at a root and routes gradients back.Broadcast and gather layers alter the distributed network structure through root-specific parsing and blob handling.
  • MPI-Caffe Capabilities and Constraints: MPI-Caffe supports large single-path networks, sophisticated ensemble losses, arbitrary MPI-layer ordering, and faster independent-network training when communication is faster than disk access.Gather-layer use requires the bottom blob in every communication-group process and one top blob per process.

Appendix E.3. Communication Cost Analysis

The communication-cost analysis measures sharing overhead for layers in an ILSVRC-Alex×5 ensemble on a distributed K20 cluster. The reported overhead is small and scales approximately linearly with the size of the shared layer.

  • Communication Cost Analysis: The experiment measured communication time as a fraction of the forward-backward pass while sharing layers of the ILSVRC-Alex×5 architecture across nodes.Each network ran on a separate node, with one node also holding the shared layers; the cluster used Tesla K20 GPUs and 5.8 GB/sec maximum interconnect bandwidth.
  • Communication Cost Analysis: Communication overhead is quite small and scales approximately linearly with the size of the shared layer.Figure 13 plots the fraction of forward-backward pass time used for TreeNets sharing different layers against layer size.
Loading 1511.06314v1…