Source-linked AI summary
Uncertainty Estimates and Multi-Hypotheses Networks for Optical Flow
Eddy Ilg, Özgün Çiçek, Silvio Galesso, Aaron Klein, Osama Makansi, Frank Hutter, Thomas Brox
TL;DR
Optical-flow networks need uncertainty estimates because single predictions can be unreliable in difficult or safety-critical scenarios. The paper compares several uncertainty-estimation strategies and introduces a WTA-based multi-hypotheses network that produces and merges alternatives efficiently. The study finds strong uncertainty estimates overall, with the proposed architecture offering very good performance at interactive speeds.
Problem
Optical-flow networks typically output a single prediction rather than distribution parameters, limiting information about reliability and alternative hypotheses.
Method
The paper compares empirical, predictive, and predictive-ensemble uncertainty methods and introduces a WTA multi-hypotheses network with a merging network.
Results
Uncertainty estimates are generally strong; FlowNetH-Pred-Merged is on-par with BootstrappedEnsemble-Pred-Merged in AUSE and EPE while offering the best runtime trade-off.
Takeaways & Limitations
Networks can estimate their own uncertainty reliably in real time, supporting more informed use of optical-flow predictions.
Takeaways & Limitations
Sampling-based ensemble approaches increase computational load and memory linearly with sample count, making them unsuitable for real-time use.
Abstract
from arXiv · showhide
Optical flow estimation can be formulated as an end-to-end supervised learning problem, which yields estimates with a superior accuracy-runtime tradeoff compared to alternative methodology. In this paper, we make such networks estimate their local uncertainty about the correctness of their prediction, which is vital information when building decisions on top of the estimations. For the first time we compare several strategies and techniques to estimate uncertainty in a large-scale computer vision task like optical flow estimation. Moreover, we introduce a new network architecture utilizing the Winner-Takes-All loss and show that this can provide complementary hypotheses and uncertainty estimates efficiently with a single forward pass and without the need for sampling or ensembles. Finally, we demonstrate the quality of the different uncertainty estimates, which is clearly above previous confidence measures on optical flow and allows for interactive frame rates.
1 Introduction
Deep optical-flow networks offer strong accuracy and speed but usually provide only a single prediction, leaving uncertainty and alternative hypotheses unmodeled. The paper compares uncertainty-estimation strategies and introduces a multi-hypotheses architecture designed for efficient deployment.
- Deep networks achieve high-accuracy optical flow at interactive frame rates, supporting automotive and activity-understanding applications.
- Black-box optical-flow predictions require reliability estimates because errors in autonomous-driving decisions can be fatal.
- Standard vision networks typically output one preferred prediction rather than parameters of a predictive distribution.
- The paper systematically compares empirical ensembles, predictive models, and predictive ensembles for uncertainty estimation in high-resolution regression networks.Evaluated techniques include MC dropout, bootstrapped ensembles, and snapshot ensembles.
- Sampling-based ensembles improve uncertainty estimation but increase computational load and memory footprint linearly with sample count, preventing real-time use.
- A WTA-based multi-headed network generates diverse hypotheses in one network, while a second network merges them and estimates final uncertainty.The architecture avoids sampling and is intended to combine reliability with interactive runtime.
3 Uncertainty Estimation with Deep Networks
The paper estimates optical-flow uncertainty using empirical ensembles, parametric predictive distributions, and combinations of both. These approaches extend deterministic FlowNet predictions with distributional information such as variance.
- 3 Uncertainty Estimation with Deep Networks: Deterministic FlowNet produces a single preferred optical-flow prediction, while this paper investigates estimating its predictive variance as uncertainty.The network maps input images to optical flow, and the preferred output typically corresponds to the mean of p(y|x, D).
- 3 Uncertainty Estimation with Deep Networks: Three uncertainty strategies use ensemble variance, a parametric output distribution, or a combination of both.The paper categorizes these as empirical ensembles, predictive models, and predictive ensembles.
- 3.1 Empirical Uncertainty Estimation: Ensembles approximate the mean and variance of p(y|x, D) from independently trained models’ predictions.Dropout, bootstrapping, and SGDR snapshots are investigated as ensemble-construction methods.
- 3.1 Empirical Uncertainty Estimation: Ensemble approaches require multiple forward passes, increasing training and test-time computation and memory linearly with the number of samples.This runtime cost prevents such approaches from being applicable in real time.
- 3.2 Predictive Uncertainty Estimation: Parametric predictive models output distribution parameters and use negative log-likelihood, with a Laplacian chosen because FlowNet’s EPE is more robust to outliers.For the Laplace model, the predictive variance is σ2 = 2b2.
- 3.3 Predictive Ensembles: Predictive ensembles combine per-network parametric distributions with ensemble-based aggregation of their means and variances.The combined approach can use dropout, bootstrapping, or SGDR variants.
4 Predicting Multiple Hypotheses within a Single Network
The paper uses a multi-headed network and Winner-Takes-All loss to predict diverse optical-flow hypotheses in one network. A second network merges these hypotheses into a final flow prediction and uncertainty estimate.
- 4 Predicting Multiple Hypotheses within a Single Network: The multi-hypothesis method generates several optical-flow predictions within one network, avoiding the multiple forward passes required by sampling-based methods.The hypotheses are subsequently merged into a single prediction and variance by a second network.
- 4 Predicting Multiple Hypotheses within a Single Network: Winner-Takes-All loss penalizes only the best hypothesis for each pixel, encouraging different predictions when the correct solution is uncertain.The best hypothesis is selected using ground-truth endpoint error.
- 4 Predicting Multiple Hypotheses within a Single Network: The loss encourages at least one hypothesis to match the ground truth and promotes diverse likely solutions where multiple flow outcomes exist.A one-sided-difference term encourages similar solutions to belong to the same hypothesis.
- 4 Predicting Multiple Hypotheses within a Single Network: Although hypothesis collapse is possible in principle, the authors report that training variability made collapse very unlikely and that all hypotheses were used.Oracle merging showed that the hypotheses contributed more or less equally.
5 Experiments
Experiments evaluate uncertainty estimation strategies for optical flow using FlowNet-based models, sparsification metrics, and comparisons across datasets and architectures. The multi-hypotheses FlowNetH-Pred-Merged model provides strong uncertainty estimates with an interactive-rate accuracy–runtime trade-off.
- 5.1 Training Details: The experiments use FlowNetC as a tractable base model, with training modifications including Batch Normalization and cosine learning-rate scheduling.The evaluation focuses on uncertainty estimation rather than improving optical-flow accuracy over the base model.
- 5.2 Evaluation Metrics: Sparsification plots progressively remove pixels with the highest estimated uncertainties and compare the resulting error curve with an oracle ranked by ground-truth error.Sparsification Error measures the difference from the oracle, enabling fair comparison across methods; its area under the curve provides a single-number summary.
- 5.3 Comparison among Uncertainties from CNNs: Predictive uncertainty models outperform empirical ensembles in uncertainty quality, while empirical ensembles generally achieve lower EPE and Oracle EPE.Predictive training can explain outliers through loss attenuation, whereas empirical ensembles retain greater within-ensemble diversity.
- 5.3 Comparison among Uncertainties from CNNs: FlowNetH-Pred-Merged is on-par with BootstrappedEnsemble-Pred-Merged in AUSE and EPE, while offering the best trade-off when runtime is included.Only FlowNetC and FlowNetH-Pred-Merged among the compared approaches allow deployment at interactive frame rates.
- 5.4 Comparison to Energy-Based Uncertainty Estimation: FlowNetH-Pred-Merged outperforms ProbFlow in uncertainty estimation, despite having a similar oracle on Sintel, and also outperforms it in EPE on KITTI.The comparison indicates that the advantage is attributable to better uncertainty estimates rather than simply weaker optical-flow predictions.
- 5.4 Comparison to Energy-Based Uncertainty Estimation: Stacking refinement networks does not negatively affect uncertainty estimation, even as the predicted flow fields improve.The uncertainty approach also performs well qualitatively outside motion boundaries and across other causes of brittle optical-flow estimates.
6 Conclusion
The paper finds that deep regression networks can estimate optical-flow uncertainty effectively, including with a single network, and presents a multi-hypotheses network that operates faster than sampling-based approaches and ensembles.
- SGDR and Bootstrapped Ensembles outperform the commonly used dropout technique for optical-flow uncertainty estimation.
- A single network estimates its own uncertainty surprisingly well and outperforms every empirical ensemble.
- The multi-hypotheses network shows very good performance while running faster than sampling-based approaches and ensembles.
- Reliable real-time uncertainty estimation is described as practically relevant because it can increase trust in trained networks.
1 Video
The supplementary video provides qualitative results on diverse real-world video sequences and compares the method with ProbFlow.
- The supplementary video presents qualitative results on diverse real-world video sequences.
- It also includes a comparison with ProbFlow.
2 Color Coding
The paper visualizes optical flow through direction-and-magnitude color coding and visualizes uncertainty through predicted entropy from a Laplace distribution model.
- Hue represents displacement direction, while color intensity represents displacement magnitude; white denotes no motion.
- Flow fields are independently rescaled for each displayed image pair and video fragment because motion ranges differ across sequences.
- Uncertainty visualizations use predicted entropy computed from estimated Laplace-distribution scale parameters for the x and y dimensions.
- Oracle entropy sets the scale parameters to the absolute endpoint errors, enabling comparison with the model’s estimated entropies.
3 Sparsification Plots
Sparsification evaluates whether removing the most uncertain pixels proportionally reduces remaining endpoint error, while dataset-wise normalization changes outlier visibility and high-uncertainty behavior.
- Sparsification progressively discards the pixels judged most uncertain and checks whether remaining average endpoint error decreases proportionally.
- Flow-field color coding maps each pixel’s displacement vector to hue and magnitude for visualization.
- Image-wise sparsification averages individually normalized image plots, giving easy and hard uncertainty-estimation cases equal weight.
- Dataset-wise sparsification ranks pixels across the whole dataset before normalization, making outlier effects more visible through larger curve slopes.
- FlowNetH-Pred-Merged performs slightly worse than other ensembles at very high uncertainties under dataset-wise sparsification but remains best by AUSE.
4 Effect of Pre-Cycles for SGDR Ensembles
The study examines how discarded earlier SGDR ensemble cycles affect optical-flow accuracy and uncertainty quality. Later models improve EPE without penalizing AUSE.
- 4 Effect of Pre-Cycles for SGDR Ensembles: Later SGDR models improve EPE without penalizing AUSE for ensembles with M = 8.The comparison covers pre-cycle counts from 0 to 8 under both image-wise and dataset-wise sparsification normalization.
5 Evaluation on KITTI and Comparison to ProbFlow
The KITTI evaluation compares FlowNetH-Pred-Merged with ProbFlow after fine-tuning and examines the associated uncertainty and refinement stack. FlowNetH outperforms ProbFlow in both flow accuracy and uncertainty quality, while the full stack uses staged refinement training.
- 5 Evaluation on KITTI and Comparison to ProbFlow: Fine-tuning significantly reduces FlowNetH-Pred-Merged endpoint error and AUSE on the joint KITTI2012 and KITTI2015 training split.The evaluation uses a 75%/25% training-test split after mixing the two KITTI datasets.
- 5 Evaluation on KITTI and Comparison to ProbFlow: FlowNetH outperforms ProbFlow in both EPE and AUSE, and its uncertainty quality surpasses ProbFlow independently of flow accuracy.Figure 4 reports that fine-tuning does not drastically change sparsification error while EPE decreases significantly.
- 5 Evaluation on KITTI and Comparison to ProbFlow: Two stacked refinement networks estimate residual flow and uncertainty on top of FlowNetH-Merged, with residual connections yielding much lower convergence times.The refinement networks are trained step by step, while lower networks are fixed after their training stage.
7 Qualitative Evaluation
Qualitative evaluations compare FlowNetH-Pred-Merged and ProbFlow across real-world, Sintel, and KITTI examples. FlowNetH produces especially diverse hypotheses in difficult cases while displaying predicted and oracle uncertainty outputs.
- 7 Qualitative Evaluation: Qualitative comparisons cover real-world videos, Sintel train clean, KITTI2012, and KITTI2015 datasets.The figures show original image pairs or sequences, ground-truth flow where available, predicted flow, and entropy estimates.
- 7 Qualitative Evaluation: FlowNetH hypotheses generate the most diverse results among the compared ensembles.This diversity is particularly visible in the hard Sintel example, whereas easy examples show little output variety.
- 7 Qualitative Evaluation: For easy Sintel examples, the networks are certain and their outputs show little variety.The predictive and empirical tables report averaged flows and computed entropies for these cases.
- 7 Qualitative Evaluation: For hard Sintel examples, each method shows some output variety, while FlowNetH provides much more varied predictions.The predictive results identify the FlowNetH-Pred-Merged hypothesis as the most diverse.