Source-linked AI summary
Learning to Estimate Hidden Motions with Global Motion Aggregation
Shihao Jiang, Dylan Campbell, Yao Lu, Hongdong Li, Richard Hartley
TL;DR
Two-frame optical flow has difficulty estimating motions for points that become occluded because local evidence is missing or unreliable. The paper introduces global motion aggregation, which uses reference-image self-similarity to propagate motion through long-range dependencies. It reports improved occluded-region flow without harming non-occluded regions and state-of-the-art Sintel results.
Problem
Occlusions make two-frame optical flow difficult because brightness constancy and appearance correlations provide insufficient guidance for motions absent from the second frame.
Method
A transformer-inspired global motion aggregation module uses reference-frame self-similarities to aggregate motion features and augment RAFT.
Results
13.6% lower average EPE on Sintel Final and 13.7% lower average EPE on Sintel Clean were reported relative to RAFT.
Takeaways & Limitations
Global aggregation improves optical-flow predictions in occluded regions without damaging non-occluded performance and achieves first-place results on Sintel at submission.
Takeaways & Limitations
On KITTI 2015, results were on par with RAFT, with insufficient training data identified as a likely reason for the lack of improvement.
Abstract
from arXiv · showhide
Occlusions pose a significant challenge to optical flow algorithms that rely on local evidences. We consider an occluded point to be one that is imaged in the first frame but not in the next, a slight overloading of the standard definition since it also includes points that move out-of-frame. Estimating the motion of these points is extremely difficult, particularly in the two-frame setting. Previous work relies on CNNs to learn occlusions, without much success, or requires multiple frames to reason about occlusions using temporal smoothness. In this paper, we argue that the occlusion problem can be better solved in the two-frame case by modelling image self-similarities. We introduce a global motion aggregation module, a transformer-based approach to find long-range dependencies between pixels in the first image, and perform global aggregation on the corresponding motion features. We demonstrate that the optical flow estimates in the occluded regions can be significantly improved without damaging the performance in non-occluded regions. This approach obtains new state-of-the-art results on the challenging Sintel dataset, improving the average end-point error by 13.6% on Sintel Final and 13.7% on Sintel Clean. At the time of submission, our method ranks first on these benchmarks among all published and unpublished approaches. Code is available at https://github.com/zacjiang/GMA
1. Introduction
Two-frame optical flow struggles to estimate motions for points that become occluded, because local matching evidence is absent or unreliable. The paper addresses this by propagating motion through image self-similarities and long-range connections, improving Sintel accuracy.
- Problem: Occlusion is especially difficult because points visible in the reference frame may disappear through out-of-frame motion or occlusion by objects.The paper defines occluded points as 3D points imaged in the reference frame but not visible in the matching frame.
- Problem: Brightness constancy and appearance correlations provide little reliable guidance for learning motions in occluded regions.Existing approaches instead use smoothness interpolation or CNNs to infer motion from neighboring pixels, but these methods can fail under substantial occlusion.
- Approach: Long-range connections weighted by reference-frame self-similarities allow motion from non-occluded points to be propagated to occluded points.The method assumes points on the same object often have homogeneous motion and uses attention to identify related pixels.
- Approach: The global motion aggregation module augments RAFT by aggregating motion features, then combining them with local motion and visual context features.The decoder can therefore use local, global, or combined features depending on the pixel location.
- Results: 13.6% lower average EPE on Sintel Final and 13.7% lower average EPE on Sintel Clean were reported relative to RAFT.The corresponding values were 2.86 → 2.47 on Sintel Final and 1.61 → 1.39 on Sintel Clean.
2. Related Works
Prior optical-flow methods address occlusions through robust objectives, interpolation, explicit modeling, or learned local propagation. This paper instead uses implicit non-local motion aggregation based on image self-similarity and transformer-inspired attention.
- Prior approaches: Traditional methods treat occlusions as outliers and use robust objectives, regularization, or truncated data terms.These approaches include robust potentials, total-variation regularization, and discrete optimization formulations.
- Prior approaches: MRF-based methods estimate forward and backward flow, identify occlusions through consistency checks, and interpolate their motion as post-processing.The interpolation step fills flow in regions detected as occluded.
- Prior approaches: Other methods jointly optimize occlusion-related objectives, including forward-backward consistency, occlusion-disocclusion symmetry, or local depth relationships.These approaches explicitly incorporate occlusion reasoning into optimization or scene modeling.
- This work: This paper avoids explicit occlusion reasoning in the loss and instead performs implicit global motion aggregation for occluded regions.The approach is characterized as non-local interpolation rather than local interpolation.
- Attention design: The method uses context-derived query and key vectors with motion-derived value vectors, rather than conventional self-attention over one feature type.The context features model image appearance, while motion features encode the correlation volume.
3. Method
The method augments RAFT with global motion aggregation, using appearance self-similarity and positional information to propagate motion features over long ranges. The aggregated features are combined with local motion features before GRU-based residual flow decoding.
- RAFT backbone: RAFT provides all-pairs correlations and iterative GRU residual refinement, but convolutional propagation remains limited when occlusions lack local evidence.The final flow is formed by summing iteratively predicted residual flows.
- Positional information: The attention matrix combines appearance similarity with relative positional embeddings to condition global motion aggregation on feature location.Separate vertical and horizontal offset embeddings are summed, allowing the model to learn distance-dependent aggregation behavior.
- Global motion aggregation: Global motion aggregation uses attention over context-feature self-similarities to propagate motion information between pixels.Query and key projections model appearance similarity, while value projections encode motion features from the correlation volume.
- Attention formulation: A learned scalar and query, key, and value projection functions parameterize the attention-based feature update.The module learns Wqry, Wkey, Wval, and α for its projections and aggregation strength.
- Feature decoding: The final decoder input concatenates local motion features, aggregated motion features, and context features before GRU decoding.This lets the network select or combine motion information rather than prescribing when aggregated motion must be used.
- Ablation variants: A positional-only variant performs long-range aggregation without reasoning about image content, providing a comparison against self-similarity-based attention.The variants are reported as Ours (+p) and Ours (p only).
4. Experiments
Experiments evaluate GMA on Sintel and KITTI, including region-specific occlusion analyses, comparisons with RAFT, qualitative attention visualizations, ablations, and computational cost. GMA improves Sintel flow accuracy, especially for occluded regions, while remaining close to RAFT on KITTI with modest overhead.
- 4.1. Experimental Setup: Sintel evaluations partition errors into non-occluded, in-frame occluded, and out-of-frame occluded regions across Clean, Final, and Albedo rendering passes.Clean and Final are training-set evaluations, while Albedo is reserved for evaluation and removes illumination effects except in occluded regions.
- 4.2. Occlusion Analysis: GMA’s relative Sintel improvement is predominantly attributable to better predictions for occluded points, including the more difficult out-of-frame cases.Non-occluded performance also improves, with the authors associating this with resolving ambiguities from reflections, blur, and other brightness variations.
- 4.3. Comparison with Prior Works: 13.6% improvement reduces AEPE from 2.86 to 2.47 on Sintel Final against RAFT.The method ranks first on the Sintel Final benchmark at submission and also leads on the EPE unmatched metric.
- 4.3. Comparison with Prior Works: 13.7% improvement reduces AEPE from 1.61 to 1.39 on Sintel Clean against RAFT.The method ranks first on Sintel Clean and on EPE unmatched for both Clean and Final.
- 4.3. Comparison with Prior Works: On KITTI 2015, GMA is on par with RAFT, while positional-only attention outperforms RAFT and content self-similarity attention slightly underperforms.The authors attribute the lack of improvement to insufficient training data for learning high-level appearance similarities.
- 4.6. Timing, Parameter Counts and Memory: 72ms inference, 5.9M parameters, and 17.2GB training memory characterize GMA versus RAFT’s 60ms, 5.3M parameters, and 16.0GB.Measurements use a single RTX 3090 GPU; the authors describe the resulting computational overhead as modest.
5. Discussion
Long-range, self-similarity-weighted connections can resolve occluded-point flow, but additive aggregation depends on approximately homogeneous attended flows.
- Long-range connections weighted by image self-similarities effectively resolve optical flow for occluded 3D points.The approach aggregates motions from points with similar motion characteristics to in-paint occluded motion.
- The method relies on similar flow vectors frequently occurring among points belonging to the same class or scene object.This reflects approximate homogeneous motion in 3D.
- Additive aggregation is helpful only when the attended locations have approximately homogeneous flow.General object and camera motions can violate this condition, even on a single rigid object.
6. Conclusion
The paper addresses occlusion as a major source of optical-flow error with a global motion aggregation module based on first-image appearance self-similarity. Experiments show improved occluded-region predictions and strong Sintel performance.
- Occlusion is a major source of error in optical flow estimation.
- The global motion aggregation module aggregates motion features using appearance self-similarity from the first image.The method uses long-range connections to introduce higher-order reasoning into optical flow.
- Experiments show significantly improved optical-flow predictions in occluded regions, including large reductions in EPE on Sintel Clean and Final.
- The aggregation approach is applicable to any supervised flow network.
7. Screenshots of Sintel Server Results
GMA achieves the best reported Sintel Clean and Final server results under both overall and unmatched-pixel EPE metrics.
- GMA obtains the best overall results under the ‘EPE all’ metric on Sintel Clean and Final.
- GMA also obtains the best ‘EPE unmatched’ results by a large margin over previous approaches.The authors associate this result with effectiveness in addressing occlusion.
8. Additional Qualitative Results
The paper supplements benchmark results with qualitative visualisations from Sintel and Slow Flow, while noting dataset-specific limits on training and quantitative evaluation.
- Additional Qualitative Results: Additional visualisations are provided for the Sintel Albedo training dataset, although the model was not trained on that dataset.
- Additional Qualitative Results: Sintel Clean and Final test-set visualisations are shown, but per-image average EPE cannot be reported because test-set ground truth is unavailable.
- Additional Qualitative Results: Qualitative results on the real-world Slow Flow dataset demonstrate the approach’s benefits on real-world data.
- Screenshots of Sintel Server Results: Figure 7 shows GMA ranking first on Sintel Clean and Final for ‘EPE all’ and ‘EPE unmatched’ as of March 17th, 2021.The unmatched-pixel result has a large margin over previous approaches.