Source-linked AI summary
Efficient Uncertainty Estimation for Semantic Segmentation in Videos
Po-Yu Huang, Wan-Ting Hsu, Chun-Yueh Chiu, Ting-Fan Wu, Min Sun
TL;DR
Uncertainty estimates matter in applications where decisions depend on model confidence, but repeated MC dropout sampling can be slow. The paper uses video temporal information for region-based aggregation, reporting comparable CamVid segmentation accuracy with less than 2% mean-IoU drop and better frame-level uncertainty metrics.
Problem
Repeated MC dropout sampling makes uncertainty estimation slow for real-time applications, while segmentation works often lack probabilistic outputs with model uncertainty.
Source: §1 · p. 1, §2.1 · p. 3Method
RTA uses video temporal information and optical flow to aggregate prior outputs, adjusting incoming-data weights using reconstruction error.
Source: §3.2 · p. 6, §3.3 · p. 8Results
Less than 2% drop on mean IoU is reported for RTA on CamVid, with comparable video-segmentation accuracy and frame-level metrics that outperform MC dropout.
Source: §1 · p. 3Takeaways & Limitations
The authors report that temporal aggregation can speed uncertainty estimation by using video information rather than repeated sampling.
Source: §1 · p. 3, §3.2 · p. 6When optical flow is inaccurate, particularly for fast-moving or occluded regions, TA-MC uncertainty estimates can become inaccurate.
Source: §3.3 · p. 7
Abstract
from arXiv · showhide
Uncertainty estimation in deep learning becomes more important recently. A deep learning model can't be applied in real applications if we don't know whether the model is certain about the decision or not. Some literature proposes the Bayesian neural network which can estimate the uncertainty by Monte Carlo Dropout (MC dropout). However, MC dropout needs to forward the model $N$ times which results in $N$ times slower. For real-time applications such as a self-driving car system, which needs to obtain the prediction and the uncertainty as fast as possible, so that MC dropout becomes impractical. In this work, we propose the region-based temporal aggregation (RTA) method which leverages the temporal information in videos to simulate the sampling procedure. Our RTA method with Tiramisu backbone is 10x faster than the MC dropout with Tiramisu backbone ($N=5$). Furthermore, the uncertainty estimation obtained by our RTA method is comparable to MC dropout's uncertainty estimation on pixel-level and frame-level metrics.
1 Introduction
Uncertainty estimation matters for applications such as self-driving, but repeated MC dropout sampling slows inference. The paper proposes temporal aggregation methods that use video information to estimate uncertainty more efficiently while maintaining performance.
1 Introduction
- MC dropout estimates uncertainty by averaging stochastic network passes, but repeated sampling increases computation cost and can hinder real-time use.The introduction notes that some uncertainty-estimation methods require multiple samples.Source: §1 · pp. 1–2
- TA averages outputs from consecutive frames to approximate MC dropout sampling, using optical flow to aggregate pixels at their moved positions.For static objects, averaging N consecutive frames has the same effect as MC dropout with N samples.Source: §1 · p. 2
- RTA adjusts the incoming-data weight using reconstruction error, reducing reliance on previous predictions where flow estimation is poor.The method is designed for objects with large displacements that can lead to incorrect flow and estimates.Source: §1 · pp. 2–3
- More than 10 times faster inference is reported for TA compared with MC dropout.The speedup depends on the backbone model.Source: §1 · p. 3
- Less than 2% drop on CamVid mean IoU accompanies comparable video-segmentation accuracy with RTA.The authors also report that their uncertainty estimation outperforms MC dropout on frame-level metrics.Source: §1 · p. 3
2 Related Work
Related work covers uncertainty estimation, semantic-segmentation architectures, and the use of temporal information in video tasks. The paper positions its approach within these strands.
2.1 Uncertainty Estimation
- Most segmentation works do not generate probabilistic outputs with a measure of model uncertainty, while MC dropout estimates uncertainty through repeated stochastic passes.The section describes Bayesian neural networks as modeling distributions over weights and MC dropout as an approximation of variational inference.Source: §2.1 · p. 3
2.2 Semantic Segmentation
- The paper selects Bayesian SegNet and Tiramisu, both encoder-decoder architectures, to demonstrate its approach.The related-work discussion also surveys U-net, MobileNets-based segmentation, PSP-Net, and DeepLab.Source: §2.2 · p. 4
2.3 Leverage Temporal Information
- Prior video methods use superpixels, patches, object proposals, or optical flow to exploit temporal information; the authors claim to be the first to use optical flow to speed uncertainty estimation.Examples include warping nearby-frame feature maps to a reference frame for aggregation.Source: §2.3 · p. 4
3 Method
The method replaces repeated MC-dropout sampling with temporal aggregation, then adapts aggregation per pixel using optical-flow reconstruction error.
3.1 Preliminary: Bayesian Neural Network with MC dropout
- Bayesian neural networks model distributions over weights; MC dropout approximates predictions with N stochastic test-time passes, making inference N times slower than one pass.Dropout is treated as an approximation to the weight posterior.Source: §3.1 · pp. 5–6
3.2 Temporal Aggregation MC Dropout (TA-MC)
- TA-MC approximates MC sampling by aggregating outputs across consecutive video frames, warping prior predictions with optical flow before weighted combination.Entropy and variation ratio use the aggregated prediction directly, while BALD and Mean STD require additional quantities to be aggregated.Source: §3.2 · pp. 6–7
3.3 Region-Based Temporal Aggregation MC Dropout (RTA-MC)
- RTA sets per-pixel α from reconstruction error: high-error regions receive a larger α and rely more on current outputs, attenuating mismatched aggregation.The method uses a threshold λ to distinguish flow quality and assigns αerror higher than αacc.Source: §3.3 · p. 8
4 Experiment
On CamVid, the experiments compare TA-MC and RTA-MC with MC dropout using SegNet and Tiramisu backbones. RTA is reported to preserve comparable segmentation accuracy while reducing inference time, with uncertainty estimates comparable at pixel level and stronger on some frame-level metrics.
4.1 Dataset
- CamVid contains 701 labeled road-scene frames across 11 classes, split into 367 training, 101 validation, and 233 test frames.Frames are labeled every second from four 30Hz videos and resized to 360x480 pixels.Source: §4.1 · p. 8
4.3 Results of Video Segmentation
- At Tiramisu’s matched-accuracy setting with MC dropout at N=5, the methods remain 10x faster; SegNet RTA has only a 1.2% mean IoU drop.TA-MC and RTA-MC have the same inference speed; the reported Tiramisu comparison reduces MC dropout sampling to N=5.Source: Table 1 · p. 9, §4.3 · p. 10
4.2 Implementation Details
- The experiments use Bayesian SegNet and Tiramisu, FlowNet 2.0 for optical flow, and a flow-error threshold of 10 for RTA.TA uses α=0.2; RTA uses α_acc=0.2 and α_err=0.7, with timing measured on a GTX 1080.Source: §4.2 · p. 9
4.4 Results of Uncertainty Estimation
- As the retained most-certain pixel fraction falls from 1 to 0.5, mean IoU rises for all methods; TA-based estimates remain comparable to MC dropout.MC dropout has the highest accuracy at almost all percentages, while TA-MC and RTA-MC have similar pixel-level curves.Source: Figure 4 · p. 10, §4.4 · p. 11, Figure 5 · p. 11
- For frame-level ranking, RTA-MC is comparable overall to MC dropout and leads for Entropy and Variation Ratio; at 30% retrieval, SegNet RTA reaches 69.6% versus MC dropout’s 66.7%.The paper reports similar Entropy and Variation Ratio results for Tiramisu; Table 3 also reports RTA-MC’s highest Variation Ratio scores at 30%, 50%, and 70% retrieval for both backbones.Source: §4.4 · pp. 11–12, Table 2 · p. 13, Table 3 · p. 13
- 10.97x faster inference with a 1.2% mean IoU drop, while RTA’s uncertainty is comparable on pixel-level metrics and better on frame-level metrics for Entropy and Variation Ratio.The conclusion reports these results for RTA on CamVid.Source: §5 · p. 13