Source-linked AI summary
Stochastic Multiple Choice Learning for Training Diverse Deep Ensembles
Stefan Lee, Senthil Purushwalkam, Michael Cogswell, Viresh Ranjan, David Crandall, Dhruv Batra
TL;DR
Perception systems often benefit from multiple plausible hypotheses, but existing diverse-solution training methods can require costly retraining or sequential optimization. The paper introduces sMCL, an SGD-based winner-take-gradient method for deep ensembles, and reports broad improvements in oracle performance with interpretable specialization.
Problem
Perception systems may need multiple plausible outputs for ambiguous inputs, while existing diverse-solution methods are poorly suited to general deep architectures and modern training costs.
Method
sMCL trains ensemble members with SGD by assigning each example’s gradient only to its lowest-error predictor under a joint oracle-loss objective.
Results
Across image classification, semantic segmentation, and image captioning, sMCL outperforms classical ensembles and strong baselines, including the 5x slower MCL procedure.
Takeaways & Limitations
sMCL automatically produces diverse, task-specific ensemble specializations while remaining architecture- and loss-function-agnostic and parameter-free.
Abstract
from arXiv · showhide
Many practical perception systems exist within larger processes that include interactions with users or additional components capable of evaluating the quality of predicted solutions. In these contexts, it is beneficial to provide these oracle mechanisms with multiple highly likely hypotheses rather than a single prediction. In this work, we pose the task of producing multiple outputs as a learning problem over an ensemble of deep networks -- introducing a novel stochastic gradient descent based approach to minimize the loss with respect to an oracle. Our method is simple to implement, agnostic to both architecture and loss function, and parameter-free. Our approach achieves lower oracle error compared to existing methods on a wide range of tasks and deep architectures. We also show qualitatively that the diverse solutions produced often provide interpretable representations of task ambiguity.
1 Introduction
Perception systems often need multiple plausible predictions because ambiguity can make a single low-loss output unrealistic or uninformative. The paper frames diverse prediction sets as ensembles whose members collectively cover high-probability alternatives.
- Motivation: Practical perception systems may require a set of plausible hypotheses rather than one output.Examples include semantic segmentation in autonomous vehicles and image captioning for visually impaired users.
- Approach: Ensemble members can specialize on subsets of the data so their combined predictions cover high-probability regions of the solution space.The ensemble mapping is defined as the union of member outputs.
- Approach: The proposed ensemble lets members produce different outputs under a unified loss, reflecting multi-modal beliefs.The method is evaluated on image classification, segmentation, and captioning.
- Motivation: Diverse outputs help expose implicit class confusion and ambiguous evidence that make one correct answer difficult to determine.Multiple predictions can represent overlapping classes or several reasonable guesses when information is incomplete.
- Motivation: Mode-seeking models reduce expected loss by favoring generic or blended outputs that may not reflect the input content well.Examples include mixed-class segmentations, generic captions, and repetitive “I don’t know” responses.
2 Related Work
Prior work obtains diverse predictions either from specialized probabilistic models or ensemble-training procedures that are costly or sequential. The paper targets practical training of diverse ensembles for general deep architectures.
- Generating Multiple Solutions: Existing methods extract multiple diverse solutions from single probabilistic structured-output models but do not directly apply to general deep architectures.This limits their applicability beyond probabilistic structured prediction.
- Generating Multiple Solutions: Multiple Choice Learning minimizes oracle loss over ensemble outputs but uses alternating block coordinate descent with repeated model retraining.The procedure alternates assigning examples to minimum-loss predictors with training models on their assigned partitions.
- Generating Multiple Solutions: Sequential or retraining-based ensemble methods are poorly suited to modern deep architectures that can take weeks to train.The paper introduces an SGD-based alternative to this training burden.
3 Multiple-Choice Learning as Stochastic Block Gradient Descent
The paper formulates ensemble training as minimizing oracle loss, where each example is assigned to its lowest-error predictor. sMCL makes this optimization practical by combining assignments with stochastic gradient updates.
- Objective: The objective trains differentiable ensemble learners so an oracle selecting the lowest-error prediction incurs minimal loss.The ensemble produces multiple outputs for each input.
- Multiple Choice Learning: Multiple Choice Learning replaces the oracle-loss minimum with binary assignment variables indicating each example’s lowest-error predictor.The resulting problem jointly optimizes ensemble parameters and datapoint assignments.
- Multiple Choice Learning: The original alternating algorithm resembles hard-EM or K-Means but is infeasible when training individual deep networks requires weeks or months.It alternates assignments with training models to convergence on assigned partitions.
- Stochastic Multiple Choice Learning: sMCL interleaves assignment steps with batch updates in stochastic gradient descent instead of retraining models repeatedly.It trains all ensemble members under a joint loss within standard SGD solvers.
- Stochastic Multiple Choice Learning: During backpropagation, each example’s gradient reaches only its lowest-error predictor, while other ensemble members receive zero gradient for that example.Ties are broken arbitrarily, and the strategy applies to learners trained by SGD-based solvers.
4 Experiments
Experiments evaluate sMCL across image classification, semantic segmentation, and image captioning, comparing it with independent ensembles, MCL, Dey et al., and beam search. Across tasks, sMCL improves oracle performance while producing diverse, often specialized predictions.
- Experimental setup: sMCL is evaluated with CNN, FCN, and CNN+RNN ensembles across classification, segmentation, and caption generation.The experiments cover three problem domains and multiple deep architectures.
- Interpretable Expertise: In classification, oracle assignments form emergent class specialists, whereas standard-ensemble assignments are nearly uniform.The specialization is not hand-designed or pre-initialized.
- Interpretable Expertise: Segmentation specialists vary in shape, class, and frequency of predicted segments rather than only class labels.The oracle selects the ensemble member with the highest accuracy for each test sample.
- Image Captioning: In captioning, sMCL improves oracle CIDEr-D and the number of unique n-grams over baseline ensembles, while generating diverse captions that capture language and perception ambiguity.For low ensemble sizes, sMCL also outperforms multiple-output decoders.
5 Conclusion
sMCL is an SGD-based method for training diverse deep ensembles using a “winner-take-gradient” strategy. It is broadly applicable, automatically induces task-specific specializations, and requires no parameters beyond adding an sMCL layer.
- sMCL trains diverse deep ensembles with a stochastic gradient descent “winner-take-gradient” strategy.
- The method demonstrates broad applicability and efficacy across experimental settings, outperforming classical ensembles and other strong baselines.
- sMCL automatically generates ensemble-member specializations along different task-specific dimensions.
- The approach is simple to implement, architecture- and loss-agnostic, parameter-free, and adds one sMCL layer to existing ensemble architectures.