Source-linked AI summary
Optical Flow with Semantic Segmentation and Localized Layers
Laura Sevilla-Lara, Deqing Sun, Varun Jampani, Michael J. Black
TL;DR
Existing optical flow methods use generic, spatially homogeneous assumptions even though motion differs across object classes and remains difficult near fast motions, low texture, and occlusion boundaries. The paper combines semantic segmentation with class-specific motion models and localized layers, achieving the lowest error among published monocular methods on KITTI-2015 and qualitative improvements across varied videos.
Problem
Existing optical flow methods make generic spatial assumptions despite class-dependent motion and poor performance in fast-motion, low-texture, and occlusion-boundary regions.
Method
The method uses semantic segmentation to assign class-specific motion models and composes them within spatially varying localized layers.
Results
The method achieves the lowest error among published monocular methods on KITTI-2015 flow and qualitatively improves flow and segmentation across varied videos.
Takeaways & Limitations
Semantic information about objects and regions helps estimate their motion, while motion can also improve segmentation over multiple frames.
Takeaways & Limitations
The method assumes at most two major motions in detected regions and may fail when class segmentation groups multiple objects together; instance-level segmentation or more layers are suggested.
Abstract
from arXiv · showhide
Existing optical flow methods make generic, spatially homogeneous, assumptions about the spatial structure of the flow. In reality, optical flow varies across an image depending on object class. Simply put, different objects move differently. Here we exploit recent advances in static semantic scene segmentation to segment the image into objects of different types. We define different models of image motion in these regions depending on the type of object. For example, we model the motion on roads with homographies, vegetation with spatially smooth flow, and independently moving objects like cars and planes with affine motion plus deviations. We then pose the flow estimation problem using a novel formulation of localized layers, which addresses limitations of traditional layered models for dealing with complex scene motion. Our semantic flow method achieves the lowest error of any published monocular method in the KITTI-2015 flow benchmark and produces qualitatively better flow and segmentation than recent top methods on a wide range of natural videos.
1. Introduction
The paper uses semantic segmentation to assign object-specific motion models and introduces localized layers whose spatial extent varies across the scene. This improves optical flow and segmentation, including at occlusion and motion boundaries.
- Motivation: Semantic segmentation supplies object boundaries, class-specific motion expectations, depth-order cues, and temporal identity cues for optical flow.
- Localized layers: Localized layers allow each pixel to belong to one or more layers with spatially varying extent, rather than imposing a small set of global layers.
- Semantic motion models: The model assigns homographies to planar regions, affine motion with deviations to independently moving objects, and other models to diverse regions such as vegetation and buildings.
- Results: Semantic optical flow achieves the lowest error among published monocular methods on KITTI-2015 flow.
- Results: Motion estimation can also improve semantic segmentation, producing precise segmentation when computed over several frames.
- Conclusion: The paper concludes that semantic information about what and where things are helps estimate how they move.
2. Related Work
Prior work uses generic region, motion, and occlusion cues, while this paper distinguishes itself by using semantic information to select motion models. Related layered approaches also face complexity and representational limits in realistic scenes.
- Motion estimation and segmentation: Earlier methods segment scenes into color- or cue-based patches and fit parametric flow models without using semantic information to choose the model.
- Motion estimation and segmentation: Generic methods reason about coherent moving regions or superpixel occlusions without knowing the identities of the segmented objects.
- Combining flow models: Combining different flow algorithms has been studied, but prior combinations are generic and do not use semantic information.
- Occlusion reasoning and figure-ground: The paper reverses the usual boundary-detection direction by using object detection to estimate motion boundaries more accurately.
- Layered optical flow: Traditional layered models suit simple scenes but struggle with complex 3D occlusion relationships and combinatorial depth ordering, which scales as K!.
- Layered optical flow: Object-layer methods typically define an object as a coherently moving image region rather than an entity with a semantic identity.
- Video segmentation: Video segmentation methods generally rely on bottom-up region and motion analysis, often tracking superpixels over time for temporal coherence.
3. Model and Methods
The method combines semantic scene segmentation with class-specific motion models and localized two-layer flow estimation. It jointly refines motion and segmentation, then composites regional estimates into a coherent flow field.
- Semantic classes and regional modeling: Semantic segmentation assigns regions to Things, Planes, and Stuff so each class can receive a distinct motion treatment.Things include independently moving objects; Planes include roads, sky, and water; Stuff includes vegetation, buildings, and unknown regions.
- Semantic classes and regional modeling: Thing regions use affine motion plus smooth deformation, while Planes use homographies and Stuff retains the initial dense flow.The localized Thing model accounts for occlusions, disocclusions, complex geometry, and deformations.
- Localized layered estimation: The localized layered formulation estimates foreground and background motion within detected object regions, allowing the number and spatial extent of layers to vary across the scene.The formulation uses two layers locally and jointly estimates motion and layer assignments over image sequences.
- Localized layered estimation: The energy combines appearance data, temporal consistency, semantic-layer coupling, and spatial contiguity to refine flow and segmentation.The data term handles visibility and occlusion, while the layer term aligns foreground segmentation with the semantic Thing mask.
- Flow-field composition: Regional motions are composited by directly pasting foreground flow and selecting or combining background estimates where localized layers overlap Planes and Stuff.Stuff-overlap regions use a weighted average of Stuff flow and layer flow.
4. Experiments
SOF is evaluated on KITTI 2015 and natural YouTube sequences, where it improves flow quality but exposes a foreground-accuracy and motion-complexity tradeoff.
- 4. Experiments: SOF is tested on KITTI 2015 and natural YouTube sequences, with quantitative evaluation available only for KITTI.The YouTube videos contain moving Pascal VOC classes but lack ground-truth flow.
- 4.1. KITTI 2015: 16.81% overall outliers versus 22.38% for DiscreteFlow demonstrates SOF’s lower KITTI 2015 error.The improvements mainly come from refined Plane motion and interpolated occluded background motion.
- 4.1. KITTI 2015: SOF has slightly more foreground outliers, revealing a tradeoff between segmentation accuracy and foreground flow accuracy.Restricting foreground motion more strongly improves segmentation but worsens the flow estimate.
- 4.3. Runtime: SOF requires up to 6 minutes per KITTI frame, dominated by DiscreteFlow initialization and motion estimation for Things.Semantic segmentation takes 10 seconds, initial motion estimation 3 minutes, Plane motion 2 seconds, and Things motion averages 1–2 minutes.
- 4.2. Natural Sequences: On YouTube sequences, SOF corrects errors in large planar regions, improves motion boundaries, and refines segmentation near object boundaries and thin regions.These results are qualitative because the natural sequences lack ground truth.
5. Conclusion and Future work
The paper concludes that semantic labels and localized layers improve motion estimation and can also refine segmentation, while future work targets richer motion and joint estimation.
- 5. Conclusion: Figure 5 shows SOF gains concentrated on roads, especially occluded regions, and near cars, compared with DiscreteFlow.The figure compares initial segmentation, SOF flow, and color-coded outlier agreement.
- 5. Conclusion: Figure 6 compares initial and SOF segmentations, DiscreteFlow and SOF flow, and ground-truth flow to show recovered details.The comparison presents segmentation and flow outputs alongside ground truth.
- 5. Conclusion: SOF assigns motion models using three semantic groups: independently moving Things, roughly planar Planes, and all remaining Stuff.The paper focuses localized layered estimation on Things and applies layered flow only around selected objects.
- 5. Conclusion: Figure 7 identifies failure when a region contains more than two dominant motions, contrary to the layered method’s assumption.This limitation is linked to grouping multiple objects within class-level segmentation.
- 5. Conclusion: The study reports the lowest monocular KITTI flow error and qualitative improvements in flow and segmentation across varied natural videos.It frames these results as evidence that knowing what and where objects are helps estimate how they move.
- 5. Future work: Future work includes jointly integrating flow estimation with semantic segmentation and using flow for class recognition as well as segmentation.The paper describes this integration as useful but challenging.