Source-linked AI summary
Learning in an Uncertain World: Representing Ambiguity Through Multiple Hypotheses
Christian Rupprecht, Iro Laina, Robert DiPietro, Maximilian Baust, Federico Tombari, Nassir Navab, Gregory D. Hager
TL;DR
Prediction uncertainty can arise from inherently multiple futures, incomplete information, or ambiguous labels, while conditional averages may produce poor outputs for multimodal distributions. The paper introduces MHP, which converts single-output models into multiple-hypothesis systems trained with a meta loss and standard optimization. Across diverse applications, MHP routinely outperforms SHP models and provides information about prediction variability, with some classification settings showing slight score reductions from added noise.
Problem
Conditional averages can poorly represent multimodal output distributions, motivating models that represent multiple plausible predictions.
Method
MHP extends single-output models to multiple hypotheses through a probabilistic Voronoi formulation and a meta loss optimized with backpropagation and gradient descent.
Results
MHP models routinely outperform SHP counterparts across four applications while providing additional insights into prediction variability.
Takeaways & Limitations
MHP offers a principled, simple framework that can retrofit existing models and expose variation across predictions.
Takeaways & Limitations
In multiple object classification, additional hypotheses can slightly reduce scores because they contribute noise when discrete outcomes are already sufficiently represented.
Abstract
from arXiv · showhide
Many prediction tasks contain uncertainty. In some cases, uncertainty is inherent in the task itself. In future prediction, for example, many distinct outcomes are equally valid. In other cases, uncertainty arises from the way data is labeled. For example, in object detection, many objects of interest often go unlabeled, and in human pose estimation, occluded joints are often labeled with ambiguous values. In this work we focus on a principled approach for handling such scenarios. In particular, we propose a framework for reformulating existing single-prediction models as multiple hypothesis prediction (MHP) models and an associated meta loss and optimization procedure to train them. To demonstrate our approach, we consider four diverse applications: human pose estimation, future prediction, image classification and segmentation. We find that MHP models outperform their single-hypothesis counterparts in all cases, and that MHP models simultaneously expose valuable insights into the variability of predictions.
1. Introduction
The paper frames uncertainty as a need to represent multiple plausible outcomes and introduces MHP as a principled extension of single-output prediction. The framework supports diverse tasks, exposes prediction variability, and improves performance over single-hypothesis models in four applications.
- Uncertainty can reflect either inherently ambiguous outcomes or incomplete information, motivating multiple plausible hypotheses.
- MHP extends single-loss, single-output systems to multiple outputs using a probabilistic formulation that yields a loss-induced Voronoi tessellation.
- A meta loss allows MHP models to be trained with standard gradient descent and backpropagation procedures.
- The framework can retrofit CNN architectures and loss functions, expose hypothesis variance, and avoid blurred predictions caused by averaging distinct regression modes.
- MHP models improve over corresponding SHP models across human pose estimation, future frame prediction, multi-label classification, and semantic segmentation.
2. Related Work
Related work covers multimodal regression, multiple-choice prediction, label ambiguity, future prediction, and uncertainty estimation. The paper distinguishes MHP by combining loss agnosticism, a Voronoi-based formulation, shared hypotheses, and applicability across tasks.
- CNNs are widely used across vision tasks, but multiple-hypothesis prediction has been addressed less extensively and under varied names and assumptions.
- Mixture density networks handle multimodal regression with Gaussian mixtures, whereas MHP is loss agnostic and yields a loss-induced Voronoi tessellation across tasks.
- MHP extends Multiple Choice Learning by providing mathematical understanding, supporting regression, introducing a convergence relaxation, and sharing one architecture across hypotheses.
- Prior work addresses label ambiguity through soft probabilistic assignments, multi-label methods, and object-proposal or classifier pipelines.
- Future-prediction studies use multiple predictions for application-specific problems, while other methods estimate uncertainty through dropout sampling or variational inference.
3. Methods
The framework extends single-output predictors to multiple hypotheses by assigning labels to the closest prediction under a chosen loss. Its optimization yields a piecewise-constant, Voronoi-based approximation of conditional output distributions and supports practical CNN training.
- Unambiguous prediction: 18: A single prediction can be a poor representation when the conditional label distribution is multimodal, falling between well-separated modes.For two well-separated Gaussian modes, the conditional average lies where probability density is low.
- Ambiguous prediction: MHP predicts M outputs and computes the loss using the closest hypothesis rather than a single conditional average.The closest-prediction assignment induces a loss over multiple hypotheses.
- Ambiguous prediction: The predicted hypotheses partition label space into Voronoi cells induced by the loss, with each hypothesis representing the conditional average of its cell.Under mild regularity conditions, the generators and predictors form a centroidal Voronoi tessellation.
- Optimization: The model can use CNNs or other learning models, and the meta-loss operates on top of any loss function that is minimized during training.The paper describes a five-step backpropagation procedure and a meta-loss that selects and weights hypotheses through a Kronecker delta.
- Empirical scope: The framework is reported to improve performance across four applications, with no observed regression-task deterioration as the number of hypotheses increases.The applications are human pose estimation, future frame prediction, multi-label classification, and semantic segmentation.
4. Experiments
The experiments evaluate MHP across continuous and discrete prediction tasks, showing improved predictions and useful uncertainty information compared with SHP.
- Experiments: MHP models represent conditional output spaces with multiple predictions across toy, pose, future-frame, classification, and segmentation experiments.The experiments test Voronoi representations, low- and high-dimensional regression, and discrete prediction tasks.
- Human Pose Estimation: Human pose variance is higher for occluded joints and difficult end-effectors than for visible, stable joints, providing a model-confidence indicator.Occluded wrists and joints such as hands and feet vary more than shoulders and hips.
- Human Pose Estimation: MDNs were difficult to train for high-dimensional pose estimation, while MHP handled high-dimensional problems without stability issues.The MDN required RMSProp and exhibited numerical instability; convergence was not achieved for future-frame prediction.
- Future Frame Prediction: MHP predicts distinct possible intersection outcomes, whereas SHP averages them into an unrealistic frame with ghost cars before the decision is made.The network selects different hypotheses depending on whether the car’s exit decision has occurred; ε = 0.05 prevents prediction starvation.
- Future Frame Prediction: More hypotheses produce sharper future-frame images, lower error, and per-pixel variance maps that identify regions with higher uncertainty.For the hat/cap action, the head and shoulders have higher estimated per-pixel uncertainty.
- Classification and Segmentation: MHP improves discrete prediction: all MHP classification models outperform SHP, while 4-MHP segmentation reaches 70.3% mean IoU versus MCL’s 69.1% using one quarter the parameters.Classification performance can decrease at high M because extra hypotheses exceed the number of discrete outcomes and add noise.
5. Conclusions
The paper presents MHP as a principled and simple framework that extends single-hypothesis models, with experiments showing improved performance and richer insight into prediction variability.
- MHP yields a Voronoi tessellation of the output space and can be retrofitted to existing SHP models using standard optimization.The framework is optimized with backpropagation and gradient descent.
- Across extensive experiments, MHP models routinely outperform their SHP counterparts while providing additional insights into the model.
- Variance across hypotheses provides information about prediction variability, while the framework applies to high-dimensional and discrete problems.