Source-linked AI summary
Occlusions, Motion and Depth Boundaries with a Generic Network for Disparity, Optical Flow or Scene Flow Estimation
Eddy Ilg, Tonmoy Saikia, Margret Keuper, Thomas Brox
TL;DR
Occlusions make correspondence estimation unreliable and also mark depth or motion boundaries relevant to downstream tasks. The paper jointly predicts occlusions, boundaries, disparity, and optical flow within efficient refinement networks. This approach improves occlusion and boundary estimates, motion segmentation, scene-flow estimation, and benchmark performance, including state-of-the-art results on KITTI.
Problem
Post-hoc occlusion estimation is suboptimal because occlusions and correspondences mutually depend on each other, while occlusions and boundaries matter for correspondence and higher-level tasks.
Method
The paper integrates occlusion and depth or motion-boundary prediction into FlowNet 2.0-based disparity and optical-flow refinement networks, with joint training.
Results
The networks improve occlusion and boundary estimates, achieve state-of-the-art results on KITTI, and improve motion segmentation and scene-flow estimation.
Takeaways & Limitations
Jointly estimated occlusions provide useful outputs for motion segmentation and scene-flow estimation alongside disparity and optical flow.
Takeaways & Limitations
The architecture conjectures that prediction scaling interacts with weight decay and can affect results when internal activation ranges differ.
Abstract
from arXiv · showhide
Occlusions play an important role in disparity and optical flow estimation, since matching costs are not available in occluded areas and occlusions indicate depth or motion boundaries. Moreover, occlusions are relevant for motion segmentation and scene flow estimation. In this paper, we present an efficient learning-based approach to estimate occlusion areas jointly with disparities or optical flow. The estimated occlusions and motion boundaries clearly improve over the state-of-the-art. Moreover, we present networks with state-of-the-art performance on the popular KITTI benchmark and good generic performance. Making use of the estimated occlusions, we also show improved results on motion segmentation and scene flow estimation.
1 Introduction
The paper jointly estimates occlusions and depth or motion boundaries with disparity or optical-flow networks, addressing the limitations of post-hoc occlusion estimation. It reports improved occlusion, boundary, motion-segmentation, benchmark, and scene-flow performance.
- Motivation: Post-hoc occlusion estimation is unreliable because occlusions and correspondences mutually influence one another.Computing correspondences in both directions and checking consistency after estimation is suboptimal when occlusions already degrade correspondence estimates.
- Approach: The network explicitly outputs occlusions together with disparity or optical flow, as well as depth or motion boundaries.The architecture integrates these predictions with FlowNet 2.0-based disparity and optical-flow networks.
- Applications: Estimated occlusions and boundaries improve motion segmentation and support scene-flow estimation by addressing erroneous or missing information in occluded regions.The scene-flow extension uses predicted occlusions to interpolate occluded areas before computing motion-compensated disparity differences.
- Results: The networks achieve state-of-the-art results on KITTI while retaining good generic performance without scenario-specific fine-tuning.The paper also reports good scene-flow performance at high frame-rate using only 2D motion information.
- Approach: Joint training resolves the occlusion–correspondence dependency rather than attempting to disentangle it.The paper reports that training occlusions jointly with disparity or optical flow is clearly beneficial.
3 Network Architectures
The architecture extends FlowNet-style refinement stacks to jointly estimate flow, occlusions, and boundaries, using residual refinement and forward/backward information. Variants include joint streams and mutually warped directional estimates, with a related disparity and scene-flow extension.
- Base architecture: The network builds on FlowNet encoder-decoder and FlowNet 2.0 refinement-stack architectures, with analogous modifications for disparity.The modifications are summarized in Figure 1.
- Refinement design: Residual connections and warped images are added to refinement networks, while the small-displacement network is omitted.The remaining fusion network performs refinement, including smoothing and sharpening, without seeing the second image.
- Occlusions and boundaries: Occlusions are predicted from the beginning, whereas depth or motion boundaries are introduced only in later refinement stages.Earlier boundary prediction improved detail but made the networks fail more rigorously when errors occurred.
- Bidirectional estimation: One variant jointly estimates forward and backward flows and occlusions using a second correlation between features from the two images.The outputs of the correlations are concatenated before further processing.
- Bidirectional estimation: Another variant separates forward and backward streams and mutually warps each direction after every network.Warping aligns the opposite-direction estimate to the corresponding image coordinates before it is supplied as input.
- Scene flow: For scene flow, disparity change is computed from disparities across time, while predicted occlusions guide interpolation where warping is incorrect or undefined.The construction combines disparity at t = 0 with optical flow and disparity change.
4 Experiments
Experiments show that CNNs can estimate occlusions from images, improve occlusion and motion-boundary quality over prior methods, and retain strong disparity and flow performance across benchmarks.
- Joint estimation: Jointly estimating occlusions neither improves nor degrades flow performance, and explicit occlusion outputs can be obtained without affecting the flow.The refinement experiments similarly find that adding occlusions generally does not outperform estimating flow alone.
- Estimating occlusions with CNNs: Occlusion estimation is possible from two images alone, while providing optical flow improves results.The experiments compare image-only inputs with versions that additionally provide forward or backward flow information.
- Estimating occlusions with CNNs: Adding backward flow marginally improves results, whereas supplying it directly does not help; warping or consistency provides a small benefit.The authors attribute the limited benefit of direct backward flow to the difficulty of looking up information at the target location.
- Comparisons to prior methods: The proposed method estimates most fine occlusion details and outperforms existing methods by large margins, especially for optical-flow occlusions and motion boundaries.Qualitative comparisons report finer details than consistency checking, S2DFlow, and MirrorFlow; motion boundaries also outperform Weinzaepfel et al.
- Benchmark results: The resulting networks achieve strong benchmark performance: larger models rank second on KITTI 2015, top on KITTI 2012 and Sintel, and set optical-flow state of the art on both KITTI datasets.The smaller versions provide a speed/accuracy trade-off, while the larger networks deliver the strongest rankings reported here.
5 Conclusion
The paper concludes that CNNs can jointly estimate occlusions and depth or motion boundaries while achieving strong disparity, flow, and scene-flow performance.
- CNNs estimate occlusions and depth or motion boundaries more effectively than traditional methods, with performance surpassing them by a large margin.
- The generic network architecture performs well for disparity and flow estimation and achieves state-of-the-art results on benchmarks.
- Scene-flow benchmarking compares the method with prior approaches, including slower state-of-the-art methods using 2D information.
1 Video
The paper provides a supplementary video containing qualitative results on multiple video sequences.
- The supplementary video contains qualitative results.
- The results cover a number of video sequences.
- The paper directs readers to the supplementary video for these examples.
2 Visual Examples
Visual examples show effective motion-boundary, occlusion, flow, depth-boundary, disparity, and scene-flow estimation across real images and Sintel examples.
- Motion and flow examples: FlowNet-CSS already estimates motion boundaries well, while FlowNet-CSSR-ft-sd produces smoother flow and slightly more detail.
- Depth and disparity examples: On Sintel examples, disparity and occlusion estimates are close to ground truth, including large occlusions and fine details.
- Scene-flow examples: For scene flow, the proposed network removes hallucination effects in occluded areas and meaningfully extrapolates missing boundary values.
3 Losses
The losses modify prediction scaling and spatial weighting to improve small-displacement behavior and emphasize occlusion or boundary regions during training.
- Prediction scaling: Removing the original factor-of-20 scaling avoids very small internal values and improves results for small displacements.The paper reports no major EPE effect from changing the scaling, with a slight EPE decrease for disparity.
- Prediction scaling: The proposed scaling changes the internal relation from y = 20 · f to y = f.
- Occlusion weighting: Occlusion cross-entropy learns large regions well but tends to ignore thin regions because they contribute less to the total loss.
- Occlusion weighting: The weighting scheme addresses class imbalance while reducing overflow effects around thin occlusion regions.
- Architecture details: Changing the correlation layer position in DispNetC produces a significant performance improvement when placed after the third convolution.
- Weight maps: A pixel receives greater weight when surrounding pixels have different occlusion values, with distance handled by a Gaussian parameter σ.The same weighting is applied to boundary estimation.
4 Ablation Study of Network Configurations
The ablation study evaluates architectural changes to the disparity and flow networks, finding that residual stacking improves performance while scaling has little effect on EPE.
- Moving the DispNet correlation layer up one level lowers endpoint error by almost 50%.This change mainly produces larger strides and correlation distances.
- Scaling the networks has no big effect on EPE, although it improves disparity results.
- Adding a residual-connected second network clearly improves performance over normal stacking.
- Initializing the third network with the second network’s weights is beneficial because both networks have similar tasks.
- The final FlowNet-CSS matches the result from the full training schedule, while the disparity stack significantly improves over DispNetC.
5 Motion Segmentation Results
The motion-segmentation evaluation shows that the approach performs better under both dense and 4px density evaluation.
- The approach performs better in the 4px density evaluation, with behavior similar to the dense version.
6 Training Settings for KITTI
The KITTI training procedure fine-tunes each network in sequence, while qualitative examples illustrate joint predictions of flow, disparity, occlusions, and boundaries across datasets.
- Training procedure: Each network in the stack is fine-tuned individually on KITTI, fixing earlier networks before training subsequent ones.
- Evaluation: The motion-segmentation table reports results on FBMS-59 at sampling densities of 8 and 4px, with the authors’ flow and occlusions achieving the best performance.
- Training procedure: Training combines KITTI 2012 and 2015 data split into 75% training and 25% validation sets.
- Training procedure: Because KITTI lacks motion-boundary ground truth, the last network is tied to precomputed boundary features with an L2 loss.
- Qualitative results: Qualitative examples show joint flow, occlusion, and motion-boundary estimates, including sharper details and visible object boundaries.
- Qualitative results: A Sintel example presents joint depth-boundary, occlusion, and disparity estimates that are generally close to ground truth.
- Qualitative results: The scene-flow visualization includes optical flow, disparity at t = 0, and disparity at t = 1 warped to t = 0 using forward flow.