Source-linked AI summary
Flow Fields: Dense Correspondence Fields for Highly Accurate Large Displacement Optical Flow Estimation
Christian Bailer, Bertram Taetz, Didier Stricker
TL;DR
Large-displacement optical flow needs reliable dense initialization, but ANNF fields are outlier-prone and sparse matches leave gaps. The paper introduces dense Flow Fields using a data-based hierarchical search and enhanced outlier filtering. Flow Fields improve optical-flow estimation across MPI-Sintel, KITTI, and Middlebury, including nearly 0.5-pixel and 0.4-pixel EPE reductions on MPI-Sintel final and clean, respectively.
Problem
Large-displacement optical flow relies on sparse descriptor matches or dense ANNF fields, whose sparsity, gaps, or outliers limit dense initialization quality.
Method
Flow Fields use a purely data-based hierarchical correspondence search with flow-space locality and enhanced two-way, region-based, and density-based outlier filtering.
Results
The approach clearly outperforms ANNF and descriptor-matching approaches, with nearly 0.5-pixel lower EPE on MPI-Sintel final and nearly 0.4-pixel lower EPE on clean than original EpicFlow.
Takeaways & Limitations
Flow Fields are presented as a dense correspondence approach better suited than ANNF and state-of-the-art descriptor matching for optical-flow estimation.
Takeaways & Limitations
The authors identify testing more advanced data terms and patch-deformation estimation as future work that could further improve the approach.
Abstract
from arXiv · showhide
Modern large displacement optical flow algorithms usually use an initialization by either sparse descriptor matching techniques or dense approximate nearest neighbor fields. While the latter have the advantage of being dense, they have the major disadvantage of being very outlier prone as they are not designed to find the optical flow, but the visually most similar correspondence. In this paper we present a dense correspondence field approach that is much less outlier prone and thus much better suited for optical flow estimation than approximate nearest neighbor fields. Our approach is conceptually novel as it does not require explicit regularization, smoothing (like median filtering) or a new data term, but solely our novel purely data based search strategy that finds most inliers (even for small objects), while it effectively avoids finding outliers. Moreover, we present novel enhancements for outlier filtering. We show that our approach is better suited for large displacement optical flow estimation than state-of-the-art descriptor matching techniques. We do so by initializing EpicFlow (so far the best method on MPI-Sintel) with our Flow Fields instead of their originally used state-of-the-art descriptor matching technique. We significantly outperform the original EpicFlow on MPI-Sintel, KITTI and Middlebury.
1. Introduction
Large-displacement optical flow is difficult because visual similarity is unreliable and local optimization can fail when coarse-scale motion is inaccurate. The paper introduces dense Flow Fields whose search strategy reduces outliers, adds hierarchical and filtering enhancements, and improves optical-flow initialization.
- Visual similarity is unreliable under illumination changes, deformations, repetitive patterns, low texture, occlusions, and blur, motivating prior assumptions in dense optical flow.
- Local-smoothness variational methods rely on local linearization and image pyramids, which can fail when lower-scale motion is not close to the correct higher-scale motion.
- Dense approximate nearest neighbor fields provide global, full-resolution search but usually contain many difficult-to-identify outliers, while sparse matches leave larger gaps.
- Flow Fields reduce optical-flow outliers through a purely data-based search strategy that finds most inliers without explicit regularization, smoothing, or a new data term.
- The hierarchical search combines non-locality in image space with flow-space locality and uses scales as outlier sieves, improving results even for tiny objects and details.
- The method extends forward-backward consistency with two-way consistency, region-based filtering, and density-based filtering.
- The approach clearly outperforms ANNF, achieving the best result on MPI-Sintel and the second-best result on KITTI.
2. Related Work
Related work combines variational optical flow with sparse or dense correspondence initialization. The paper positions Flow Fields against ANNF-based methods, descriptor matching, edge-aware variants, and related locality-based correspondence strategies.
- Sparse descriptor matching was integrated into optical flow to improve large-displacement performance, followed by several related approaches.
- Dense ANNF methods have achieved strong results, but Deep Matching was reported to clearly outperform ANNF for one optical-flow initialization approach.
- Edge-aware correspondence methods improve motion-boundary results while retaining ANNF-based correspondence determination, which the paper describes as unfavorable for optical flow.
- Region growing in 3D reconstruction shares the locality idea used to avoid outliers, but is usually computationally expensive; a faster prior alternative lacked optical-flow-specific aspects.
3. Our Approach
Flow Fields combine kd-tree initialization with propagation, random search, and hierarchical processing to obtain dense, less outlier-prone optical-flow correspondences. The hierarchy preserves tiny objects and details while filtering persistent outliers across scales.
- Basic Flow Fields: Flow Fields initialize dense correspondences by selecting the lowest-error candidate among kd-tree leaf matches for every pixel.Patches are represented by 27-dimensional WHT vectors using nine bases across three CIELab channels.
- Basic Flow Fields: Small kd-tree leaves limit resistant outliers, while propagation distributes rare correct seeds because valid optical flow is locally smooth.Outliers often fail to form a smooth surface around their locations.
- Basic Flow Fields: Four propagation steps alternate with three random-search steps, using multiple directions and subpixel offsets to refine matches without introducing new outliers.A candidate flow is replaced when a uniformly sampled offset decreases its robust matching error.
- Flow Fields: The hierarchical approach initializes only every nth pixel, increases patch radius and search distance across levels, and uses lower levels as outlier sieves.Subsampled patches are evaluated on smoothed full-resolution images, avoiding upsampling errors while retaining progressively finer accuracy.
- Flow Fields: Hierarchical propagation spreads flow across large image-space gaps, even when initialization contains only two flow values separated by 52 pixels.The method distributes a flow throughout nearly the whole image despite poor initialization and barriers to local propagation.
- Flow Fields: Hierarchical matching preserves tiny objects and details better than the basic approach, with k = 3 generally effective but k > 3 reducing tiny-object preservation.The paper attributes preservation to coarse-level availability, unscaled WHT initialization, and local minima around tiny objects.
- Outlier filtering: An extended outlier filter checks consistency against backward flows computed with two patch radii and deletes pixels inconsistent with both.The authors motivate this by observing that outliers often diverge into different directions across patch radii.
4. Evaluation
The evaluation compares Flow Fields with ANNF and prior pipelines across MPI-Sintel, Middlebury, and KITTI, showing strong accuracy and robustness to large motions, blur, and small objects.
- Datasets: The evaluation covers MPI-Sintel, Middlebury, and KITTI, spanning large motions, subpixel accuracy, and driving-scene motion.MPI-Sintel includes motions up to 400 pixels per frame, Middlebury emphasizes subpixel EPE, and KITTI contains city-street imagery with potentially large motion.
- MPI-Sintel experiments: Nearly 90% of non-occluded MPI-Sintel training pixels achieve EPE below 3 pixels with a purely data-based search.The result uses correspondence fields that consider every image position as a possible correspondence.
- MPI-Sintel experiments: More hierarchy levels improve EPE through k = 3, while weighted median filtering can raise the below-3-pixel percentage but may remove details important to EpicFlow.The hierarchy provides required details, whereas the filtering changes the flow distribution unfavorably for EpicFlow.
- MPI-Sintel results: The method clearly outperforms original EpicFlow on MPI-Sintel, reducing final EPE by nearly 0.5 pixels and clean EPE by nearly 0.4 pixels.Most of the advance comes from non-occluded areas; the method also performs well near occlusion boundaries and for motions above 40 pixels.
- Middlebury results: On Middlebury, Flow Fields achieve average rank 38.0 versus EpicFlow's 52.2 and average EPE 0.33 versus 0.39.The approach improves some datasets despite EpicFlow's limitations on subpixel Middlebury evaluation.
- KITTI results: Hierarchical matching remains effective on KITTI despite using very small patches, demonstrating that enlarged, blurred patches can support difficult matching cases.The authors report that the concept works even when matching large patches fails.
5. Conclusion and Future Work
The paper concludes that Flow Fields provide a promising dense correspondence approach for optical flow, outperforming ANNF and descriptor-matching alternatives when used with EpicFlow. Future improvements include testing more advanced data terms and patch-deformation estimation.
- Conclusion: Flow Fields are clearly superior to ANNF and better suited than state-of-the-art descriptor matching for optical flow estimation.The approach was evaluated through EpicFlow initialization and produced promising optical flow results.
- Conclusion: Advanced outlier filtering strengthens the Flow Fields approach for optical flow estimation.The paper presents Flow Fields together with advanced filtering within a state-of-the-art optical flow pipeline such as EpicFlow.
- Future Work: More advanced data terms remain a future direction for improving the approach.The authors specifically identify improvements to the census transform as potentially useful.
- Future Work: Random-search patch deformations are another proposed avenue for extending the method.The paper notes that this strategy has worked well for patch normals in 3D reconstruction.