Overcoming Limitations of Mixture Density Networks: A Sampling and Fitting Framework for Multimodal Future Prediction

Osama Makansi; Eddy Ilg; Özgün Cicek; Thomas Brox

Source-linked AI summary

Overcoming Limitations of Mixture Density Networks: A Sampling and Fitting Framework for Multimodal Future Prediction

Osama Makansi, Eddy Ilg, Özgün Cicek, Thomas Brox

arXiv:1906.03631v2Jun 9, 2019cs.CV

TL;DR

Existing future-prediction methods are limited in representing uncertain, multimodal outcomes, often producing single hypotheses or unstable constrained mixtures. The paper predicts multiple future samples with Evolving WTA and fits an unconstrained distribution in a second stage. On synthetic and real data, the framework avoids mode collapse and produces more diverse, accurate multimodal distributions than prior work.

Problem
Future prediction needs to model uncertainty and multimodality, but existing approaches often yield single hypotheses or constrained mixtures with training instability and mode collapse.
Method
The framework predicts multiple future samples with Evolving WTA and fits a distribution to those samples using a two-stage network.
Results
The proposed approach outperforms all baselines, prevents mode collapse, and produces more diverse and accurate distributions on synthetic and real data.
Takeaways & Limitations
Combining WTA-based hypothesis sampling with mixture-density fitting yields reasonable estimates of multiple possible future states and applies to mixture density estimation generally.
Takeaways & Limitations
Winner-Takes-All training can be brittle because hypotheses may become trapped in equilibria when one hypothesis pairs with multiple ground truths.

Abstract

from arXiv · show

Future prediction is a fundamental principle of intelligence that helps plan actions and avoid possible dangers. As the future is uncertain to a large extent, modeling the uncertainty and multimodality of the future states is of great relevance. Existing approaches are rather limited in this regard and mostly yield a single hypothesis of the future or, at the best, strongly constrained mixture components that suffer from instabilities in training and mode collapse. In this work, we present an approach that involves the prediction of several samples of the future with a winner-takes-all loss and iterative grouping of samples to multiple modes. Moreover, we discuss how to evaluate predicted multimodal distributions, including the common real scenario, where only a single sample from the ground-truth distribution is available for evaluation. We show on synthetic and real data that the proposed approach triggers good estimates of multimodal distributions and avoids mode collapse. Source code is available at $\href{https://github.com/lmb-freiburg/Multimodal-Future-Prediction}{\text{this https URL.}}$

1. Introduction

Future prediction must represent uncertainty and multiple plausible outcomes while using contextual information, but existing methods often produce single or unstable, constrained predictions. The paper proposes an efficient framework that samples and fits unconstrained multimodal distributions and evaluates them on synthetic and real data.

Motivation: Non-deterministic environments require predictions that capture multimodality, uncertainty, and conditional context rather than a uniform distribution.Traffic rules, past movement statistics, bicycle orientation, and speed can shape which futures are more likely.
Limitations of prior work: Most existing future-prediction methods produce a single future state, often approximating the mean and ignoring alternative outcomes.This limitation is viable in approximately deterministic settings but does not represent other possibilities in non-deterministic environments.
Limitations of prior work: Winner-Takes-All methods generate diverse hypotheses but provide no complete distribution, while cVAEs produce multiple futures without complete distributions.Other mixture approaches constrain components to predefined actions or lanes.
Limitations of prior work: General unconstrained mixture distributions require specialized initialization and training and remain vulnerable to numerical instability and mode collapse.These difficulties persist despite sequential training procedures designed to improve MDN optimization.
Proposed approach: The proposed framework uses Evolving WTA to generate samples, then fits a distribution in a second stage, producing unconstrained multimodal outputs in one forward pass.The authors present it as a generic method for future prediction and mixture density estimation.
Evaluation: Evaluation combines the synthetic CPI dataset, where full ground-truth distributions enable Earth Mover’s Distance, with real SDD data evaluated from a single ground-truth sample.The authors report that the approach outperforms baselines, prevents mode collapse, and yields more diverse and accurate distributions.

3. Multimodal Future Prediction Framework

The framework separates multimodal prediction into sampling diverse hypotheses and fitting an unconstrained mixture distribution, addressing training instability and mode collapse in direct MDN optimization.

Framework overview: The network predicts a multimodal distribution p(y|x) for an object’s future location from past images and bounding-box masks.The target is the annotated object’s location at a fixed future time t + ∆t.
Framework overview: Training data provides only one future sample per input, so the framework generalizes multimodality across differing input conditions.Repeated identical inputs with multiple futures are considered very unlikely for high-dimensional inputs.
MDN baseline: Direct MDN optimization jointly learns mixture weights, means, and variances but is numerically unstable, difficult in higher dimensions, and prone to degenerate predictions and mode collapse.The authors sequentially learn means, variances, and then all parameters jointly, yet report that MDNs still suffer from mode collapse.
Two-stage framework: The proposed pipeline splits prediction into sampling and distribution fitting, using an improved WTA loss followed by a second network that fits a mixture distribution.The fitting network replaces a typical EM-based fitting step, enabling faster runtime and end-to-end fine-tuning.
Sampling - EWTA: EWTA gradually decreases the number of winners from all hypotheses to one, releasing hypotheses from equilibria and leaving fewer unused hypotheses than WTA and RWTA.Final spurious modes are removed after adding the fitting stage and jointly fine-tuning the pipeline.
Fitting - MDF: MDF estimates soft assignments from hypotheses to mixture components and derives mixture parameters before optimizing the resulting negative log-likelihood.The assignments are produced from an M-dimensional output vector for each hypothesis.

4. Car Pedestrian Interaction Dataset

The CPI dataset is a synthetic benchmark for evaluating multimodal future predictions when the ground-truth distribution is known.

CPI simulates interacting car and pedestrian motion in a static environment with realistic, multimodal policies.Because the policies are known, the dataset provides ground-truth distributions p(y|x) for evaluation.

5. Evaluation Metrics

The evaluation uses different metrics depending on whether a full ground-truth distribution or only one ground-truth sample is available.

Oracle Error selects the hypothesis or mode closest to the ground truth, rewarding diverse outputs without penalizing unreasonable ones.
NLL measures how well a ground-truth sample fits the predicted distribution and supports evaluation when only one sample is available.Missing and inconsistent modes are penalized when NLL is averaged over the dataset.
EMD measures differences between predicted and ground-truth distributions when the full ground-truth distribution is available.It can be interpreted as the energy required to move probability mass from one distribution to the other.
SEMD measures the distance between secondary modes and the primary MAP mode to quantify mixture multimodality.Large SEMD indicates strong multimodality, while small SEMD indicates unimodality; it is intended as a secondary metric besides NLL.

6. Experiments

Experiments evaluate hypothesis generation and distribution fitting on synthetic CPI and real Stanford Drone data. EWTA improves distributional coverage, while the two-stage model avoids the mode collapse observed in standard MDNs.

6. Experiments: K = 40 hypotheses and M = 4 mixture components are used, with sequential training followed by end-to-end fine-tuning.The sampling and fitting networks are trained separately before the EWTA loss is removed for joint fine-tuning.
6. Experiments: The CPI experiments use 20k training samples, 54 test samples, and a future offset of ∆t = 20 frames.CPI permits quantitative distribution evaluation because its ground-truth distribution is available.
6. Experiments: The Stanford Drone experiments use a 50/10 video train/test split with a future offset of ∆t = 5 sec.Laplace mixtures are used for SDD because their negative log-likelihood corresponds to L1 distance and is more robust to outliers.
6.3. Hypotheses prediction: EWTA produces hypotheses with lower EMD than WTA and RWTA, although WTA and RWTA achieve slightly better oracle error.EWTA better represents the true distribution, whereas WTA prioritizes a very good individual hypothesis.
6.3. Hypotheses prediction: Dropout produces insufficiently diverse hypotheses, while EWTA approximates the CPI ground-truth distribution and captures the car–pedestrian interaction.In the illustrated scenario, EWTA and dropout learn that the pedestrian stops when the car enters, but dropout fails to estimate the car’s future.
6.4. Mixture Density Estimation: The two-stage network clearly outperforms other methods on CPI and avoids the frequent mode collapse of standard MDNs.
6.4. Mixture Density Estimation: On SDD, only a single ground-truth sample is available, so evaluation uses NLL rather than EMD; the two-stage network outperforms standard MDN.SEMD indicates that the proposed approach avoids MDN mode collapse for parametric multimodal distributions.
6. Experiments: Supplementary material includes qualitative examples, failure cases, and ablation studies of design choices.

7. Conclusion

The paper combines hypothesis sampling with mixture fitting to estimate multimodal future distributions and avoid common MDN mode collapse. Experiments report reasonable multimodal estimates and broader applicability beyond future prediction.

Qualitative SDD examples show both proposed variants capturing multimodality better than MDN and non-parametric methods, which exhibit overfitting and mode collapse.
The framework combines WTA-based hypothesis sampling with MDN-style mixture fitting in a two-stage approach designed to avoid mode collapse.
The evolving hypothesis-generation strategy is the framework’s major component for learning multiple possible future states.
Experiments show that the overall framework learns interactions between objects and produces reasonable estimates of multiple possible future states.
The authors suggest that multimodal distribution prediction may also apply to domains beyond future prediction where distribution estimation matters.

Supplementary Material for: Overcoming Limitations of Mixture Density Networks: A Sampling and Fitting Framework for Multimodal Future Prediction

The supplementary material defines the synthetic CPI dataset through interacting pedestrian and car states, regions, histories, and action distributions. It also describes generating training sequences from these state-based dynamics.

The CPI dataset models future prediction conditioned on interactions between a car and a pedestrian.
The dataset evaluates whether methods constrain the actors’ futures jointly rather than predicting independent possibilities.
The synthetic environment defines actor locations using bounding boxes and regions that constrain valid pedestrian and car positions.
Pedestrian and car dynamics are represented through states, histories, world state, and state-transition functions.
Given the states, distributions over possible actions are sampled to update actor locations.
The training set contains 20k sequences with three image frames and 10 random futures per sequence, totaling 200k samples.

2. Architecture

The architecture uses a FlowNetS encoder as its base, while the supplementary material lists possible car states used in the synthetic setting.

The base architecture is built on the FlowNetS encoder, with further architecture details provided in a supplementary table.
The supplementary material includes a table listing possible car states.

3. Baselines

The baselines include classical filtering, single-point and unimodal distribution predictors, non-parametric prediction, and the proposed sampling-fitting variants. Their settings cover motion assumptions, noise, soft labels, kernel width, history length, and output parameterization.

Classical Future Prediction: The Kalman filter baseline recursively alternates prediction and update phases while modeling location, velocity, and uncertainty.
The supplementary tables specify state-to-distribution parameter mappings for pedestrians and cars and describe the architecture used for MDNs and the first-stage network.
Classical Future Prediction: The Kalman filter assumes constant motion, no external control, and process and observation noises both set to 2.0.
The supplementary comparisons vary non-parametric kernel width, proposed-variant choice, and the history length used by EWTAD-MDF.
For single-point prediction, the baseline outputs one future position and uses Euclidean Distance loss.
For unimodal distribution prediction, the baseline outputs mean and variance and is trained with negative log-likelihood loss.
The non-parametric baseline discretizes future locations into pixels and predicts a probability for each pixel with a softmax.
Because one-hot targets are too peaked, the non-parametric baseline blurs them with a Gaussian before minimizing cross-entropy.

4. Training Details

Training is stabilized by constraining the predicted variance during optimization.

The predicted σ is passed through a scaled sigmoid function to impose an upper bound during training.The sigmoid’s center slope is scaled to 1.

5. Ablation Studies

Ablations examine framework variants, training schedules, history length, prediction horizon, and the number of generated hypotheses.

Variants of the sampling-fitting framework: EWTAD-MDF performs better than EWTAP-MDF on both CPI and SDD.Using WTA with lNLL and predicted uncertainties in the MDF stage is reported as better than using WTA with lED.
Effect of history length: Longer history improves CPI accuracy, while SDD benefits mainly from adding one history frame rather than using two.The largest SDD improvement occurs from h = 0 to h = 1, with only a slight difference at h = 2.
Effect of future time horizon: Longer prediction horizons increase error for EWTAD-MDF on CPI and SDD.The ablation attributes this to longer-horizon prediction being more complex.
Effect of number of hypotheses: Generating more hypotheses usually improves predictions, but the fitting network’s capacity limits the benefit.Results are slightly worse for K = 80, motivating deeper and more complex fitting networks.

6. Qualitative WTA variant comparison

EWTA distributes hypotheses across clusters more effectively than WTA and RWTA, while EWTAD-MDF still exhibits failures under unfamiliar or imbalanced data.

WTA variant comparison: EWTA produces hypotheses close to a Voronoi Tessellation and assigns equal numbers of hypotheses to each cluster.This allocation is relevant for fitting a distribution.
WTA variant comparison: WTA leaves many hypotheses untouched, while RWTA leaves eight hypotheses at the mean position.RWTA can generate an artificial incorrect mode and represent ground-truth samples poorly.
Failure cases: On unseen scenes, both MDN and EWTAD-MDF generalize poorly, although EWTAD-MDF predicts a more reasonable variance.The comparison concerns failure cases on SDD.
Failure cases: EWTAD-MDF can miss a mode when the corresponding behavior is rare in the imbalanced training data.A similar limitation occurs for cars, which are under-sampled in SDD and appear in fewer than 1% of scenes.