Source-linked AI summary
EpicFlow: Edge-Preserving Interpolation of Correspondences for Optical Flow
Jerome Revaud, Philippe Weinzaepfel, Zaid Harchaoui, Cordelia Schmid
TL;DR
Optical-flow estimation remains difficult for occlusions, motion discontinuities, and large displacements. EpicFlow interpolates sparse matches with an edge-aware distance and uses the dense field to initialize one-level variational minimization. It outperforms coarse-to-fine approaches and achieves strong results on MPI-Sintel, while performance depends on match and contour quality.
Problem
Optical-flow estimation remains challenging because real-world videos contain occlusions, motion discontinuities, and large displacements.
Method
EpicFlow performs edge-aware sparse-to-dense interpolation of matches, approximates the distance for faster computation, and initializes one-level variational energy minimization.
Results
EpicFlow outperforms current coarse-to-fine approaches and performs best on MPI-Sintel while remaining competitive on Kitti and Middlebury.
Takeaways & Limitations
The approach offers an alternative initialization strategy that leverages contour cues and dense correspondence fields for optical-flow estimation.
Takeaways & Limitations
EpicFlow can fail when sparse matches are erroneous or contours are extracted incorrectly, including missing matches on thin elements and motion spreading across boundaries.
Abstract
from arXiv · showhide
We propose a novel approach for optical flow estimation , targeted at large displacements with significant oc-clusions. It consists of two steps: i) dense matching by edge-preserving interpolation from a sparse set of matches; ii) variational energy minimization initialized with the dense matches. The sparse-to-dense interpolation relies on an appropriate choice of the distance, namely an edge-aware geodesic distance. This distance is tailored to handle occlusions and motion boundaries -- two common and difficult issues for optical flow computation. We also propose an approximation scheme for the geodesic distance to allow fast computation without loss of performance. Subsequent to the dense interpolation step, standard one-level variational energy minimization is carried out on the dense matches to obtain the final flow estimation. The proposed approach, called Edge-Preserving Interpolation of Correspondences (EpicFlow) is fast and robust to large displacements. It significantly outperforms the state of the art on MPI-Sintel and performs on par on Kitti and Middlebury.
1. Introduction
EpicFlow addresses optical-flow failures involving occlusions, motion discontinuities, and large displacements by interpolating sparse matches with an edge-aware distance before one-level energy minimization. It avoids coarse-to-fine initialization errors and achieves strong benchmark performance.
- Occlusions, motion discontinuities, and large displacements remain central challenges for accurate optical-flow estimation.
- Coarse-to-fine energy minimization can become trapped in local minima and propagate errors from low-resolution levels.
- EpicFlow interpolates sparse matches densely using an edge-aware distance, then initializes one-level energy minimization with the resulting field.
- The method approximates its edge-aware distance to fit one affine model per input match instead of one per pixel, accelerating interpolation without performance loss.
- EpicFlow is reported as best on MPI-Sintel and competitive on Kitti and Middlebury, while outperforming coarse-to-fine-based optical-flow estimation empirically.
2. Related Work
Prior optical-flow methods commonly combine variational optimization with coarse-to-fine processing, but this can accumulate errors and lose details. Related alternatives use matching, edge affinities, or sparse-to-dense interpolation, each with distinct limitations.
- Variational optical-flow methods commonly use coarse-to-fine minimization, which can get stuck in local minima and accumulate errors across scales for large displacements.
- Earlier methods integrated descriptor or matching cues into coarse-to-fine energies, but retained intrinsic coarse-scale flaws such as lost details and boundary overlap.
- Matching-focused approaches include superpixel-based nearest-neighbor fields and edge-based affinities for piecewise-affine flow.
- Ren’s edge-affinity approach is limited to small displacements because it discretizes the optical-flow constraint.
3. Sparse-to-dense interpolation
EpicFlow converts sparse matches into a dense correspondence field using local interpolation weighted by an edge-aware geodesic distance, then uses that field to initialize one-level variational flow estimation. A graph-based approximation makes the geodesic computation efficient while preserving piecewise, edge-respecting interpolation.
- Pipeline: EpicFlow computes sparse matches, densifies them through interpolation, and performs one-level variational energy minimization initialized by the dense field.The sparse matches can come from a state-of-the-art matching algorithm; the final minimization produces the optical flow estimate.
- Interpolation model: The interpolation estimates correspondence fields with either Nadaraya-Watson weighting or locally weighted affine transformations.Both estimators weight matches according to their proximity under a chosen distance; the affine estimator fits a local transformation.
- Local interpolation: Restricting interpolation to the K nearest matches prevents remote correspondences from influencing pixels across differently moving objects.The same neighborhood restriction is used for the weighted sum and the affine least-squares system.
- Edge-preserving distance: The geodesic distance assigns high crossing costs to motion boundaries, keeping pixels within the same motion layer close and pixels across boundaries far apart.In practice, image edges approximate motion boundaries, so edge-aware interpolation respects regions when each region has enough matches.
- Fast approximation: The approximation assigns each pixel to its nearest match, builds a neighborhood graph over matches, and estimates inter-match distances with restricted geodesic paths and Dijkstra’s algorithm.This replaces the costly computation of distances between all pixels and all matches with a graph-based approximation.
- Fast approximation: The resulting interpolation is piecewise constant for Nadaraya-Watson or piecewise affine for locally weighted affine, enabling fast propagation of match-level estimates.The paper states that this piecewise property is crucial for speed and does not impact accuracy in experiments.
4. Optical Flow Estimation
EpicFlow replaces coarse-to-fine initialization with a dense correspondence field from sparse-to-dense interpolation, followed by full-scale variational minimization. Its boundary-aware cost map provides a prior on boundary locations, while the final solver uses the interpolated field as initialization.
- Coarse-to-fine vs. EpicFlow: EpicFlow uses dense sparse-to-dense correspondences to initialize full-scale variational energy minimization instead of a coarse-to-fine scheme.The dense field is produced by interpolation and then refined through energy minimization.
- Coarse-to-fine vs. EpicFlow: Coarse-to-fine minimization lacks a theoretical proof or guarantee of accurately minimizing the full-scale energy.The paper therefore characterizes coarse-to-fine optimization as a heuristic for initialization.
- Coarse-to-fine vs. EpicFlow: EpicFlow's cost map acts as a prior on boundary locations, unlike coarse-scale local smoothness weights whose interpretation is difficult.The passage notes that a similar prior could be incorporated into coarse-to-fine minimization, but would be difficult to interpret at coarse scales.
- Optimization: Using SED instead of the stated setting with κ = 5 produced similar performance.This is reported as an experimental comparison of alternative edge-cost choices.
- Optimization: The minimization starts from the sparse-to-dense interpolation and omits coarse-to-fine processing.The implementation performs 5 fixed-point iterations, with flow updates solved using 30 successive-over-relaxation iterations.
5. Experiments
Experiments evaluate EpicFlow across MPI-Sintel, Kitti, and Middlebury, examining matches, interpolation, distance choices, initialization, benchmarks, speed, and failure cases. Results show strong performance when matches are sufficiently dense and accurate, with limitations from sparse-match and contour errors.
- Datasets and setup: EpicFlow is evaluated on MPI-Sintel, Kitti, and Middlebury, which respectively emphasize large or rapid motion, real-world driving scenes, and limited displacements.Parameters are optimized on training data before test-set comparison.
- Matches and interpolation: DeepMatching outperforms KPM by 2 pixels on MPI-Sintel and 8 pixels on Kitti, while results are comparable and below 1 pixel on Middlebury.The authors attribute KPM’s weaker Kitti performance to mismatches in repetitive textures.
- Matches and interpolation: Variational minimization improves interpolated-flow performance by around 0.5 pixel while preserving motion discontinuities and small details.The refinement smooths and further refines the initialized flow.
- Distance and contours: Using Euclidean distance causes a 1-pixel drop on MPI-Sintel, while mixing Euclidean neighborhoods with geodesic weights reduces performance by around 0.3 pixels on MPI-Sintel and Kitti.The results support using the edge-preserving geodesic distance throughout the interpolation pipeline.
- Comparison with coarse-to-fine: EpicFlow outperforms the coarse-to-fine baseline and is faster, but it performs unsatisfactorily for low-density or strongly corrupted matches.Its advantage appears when matching is sufficiently dense for a given error rate.
- Comparison with the state of the art: On MPI-Sintel, EpicFlow leads the second-best method by 0.5 pixel in AEE and runs an order of magnitude faster; it is competitive on Kitti and Middlebury.It achieves AEE below 0.4 pixel on Middlebury and runs in 16.4 seconds per MPI-Sintel image pair on one CPU core.
- Failure cases: EpicFlow can fail when sparse matches are missing on thin elements or contour extraction is incorrect, causing surrounding-region motion to spread onto those structures.These failure cases affect thin objects and poorly detected boundaries.
6. Conclusion
EpicFlow combines edge-aware sparse-to-dense interpolation with one-level variational refinement for optical flow estimation. Its performance depends on both sparse matches and contour estimates, with future work targeting improvements to these components.
- EpicFlow interpolates sparse matches into a dense correspondence field using contour cues and an edge-aware geodesic distance.
- The resulting dense field initializes a one-level variational energy minimization that produces the final optical flow estimate.
- The method assumes that image contours often coincide with motion discontinuities, making contour estimates central to its interpolation strategy.
- Sparse matches and contour estimates are both key components, and the paper identifies improving them separately or jointly as future work.