Source-linked AI summary
GMFlow: Learning Optical Flow via Global Matching
Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, Dacheng Tao
TL;DR
Local cost-volume regression struggles with large displacements, while iterative refinement improves accuracy at linearly increasing inference cost. GMFlow instead formulates optical flow as global matching with Transformer-enhanced features, differentiable matching, propagation, and refinement; with one refinement, it outperforms 31-refinements RAFT on Sintel and runs faster. The framework nevertheless has room for improvement in occluded regions and may generalize poorly across large train-test domain gaps.
Problem
Local cost-volume regression restricts optical-flow search to local correlations, while RAFT's many refinements increase inference time linearly.
Method
GMFlow reformulates optical flow as global matching using Transformer feature enhancement, correlation-and-softmax matching, self-attention propagation, and higher-resolution residual refinement.
Results
With one refinement, GMFlow outperforms 31-refinements RAFT on Sintel while running faster.
Takeaways & Limitations
Global matching offers a new paradigm for accurate and efficient optical-flow estimation with fewer sequential refinements.
Takeaways & Limitations
GMFlow still has room for improvement in occluded regions and may not generalize well when training and test data have significantly different domains.
Abstract
from arXiv · showhide
Learning-based optical flow estimation has been dominated with the pipeline of cost volume with convolutions for flow regression, which is inherently limited to local correlations and thus is hard to address the long-standing challenge of large displacements. To alleviate this, the state-of-the-art framework RAFT gradually improves its prediction quality by using a large number of iterative refinements, achieving remarkable performance but introducing linearly increasing inference time. To enable both high accuracy and efficiency, we completely revamp the dominant flow regression pipeline by reformulating optical flow as a global matching problem, which identifies the correspondences by directly comparing feature similarities. Specifically, we propose a GMFlow framework, which consists of three main components: a customized Transformer for feature enhancement, a correlation and softmax layer for global feature matching, and a self-attention layer for flow propagation. We further introduce a refinement step that reuses GMFlow at higher feature resolution for residual flow prediction. Our new framework outperforms 31-refinements RAFT on the challenging Sintel benchmark, while using only one refinement and running faster, suggesting a new paradigm for accurate and efficient optical flow estimation. Code is available at https://github.com/haofeixu/gmflow.
1. Introduction
Existing optical-flow regression methods are constrained by local cost volumes, while iterative refinements improve accuracy at increasing inference cost. GMFlow reformulates flow estimation as global matching and reports higher accuracy with fewer refinements and faster inference.
- Predefined cost-volume search spaces restrict regression methods to local ranges, making large displacements difficult to handle.
- RAFT addresses large displacements through many sequential refinements, but inference time increases linearly with the refinement count.
- GMFlow reformulates optical flow as global matching by comparing feature similarities with a Transformer and differentiable correlation-and-softmax matching layer.
- GMFlow combines Transformer feature enhancement, global correlation-and-softmax matching, self-attention flow propagation, and higher-resolution residual refinement.
- With one refinement, GMFlow outperforms 31-refinements RAFT on Sintel while running faster, supporting a new accurate and efficient flow-estimation paradigm.
2. Related Work
Related optical-flow frameworks commonly regress flow from local cost volumes through multi-stage refinement, whereas GMFlow uses global matching to address large displacements. Transformer-based correspondence methods motivate this direction, but dense flow additionally requires handling occlusions and out-of-boundary pixels.
- Flow estimation approach: PWC-Net and RAFT use multi-stage refinements, but both conceptually regress flow from local cost volumes with convolutions.
- Large displacements: Coarse-to-fine methods estimate large motion incrementally but may miss fast-moving small objects when resolution is too coarse.
- Transformer for correspondences: Sparse matching methods use Transformers to reason about descriptor relationships and explicit matching layers to extract correspondences.
- Transformer for correspondences: GMFlow uses a Transformer, all-pair feature correlation, softmax matching, and self-attention propagation to produce dense flow predictions.
- Transformer for correspondences: Unlike sparse or stereo correspondence methods, GMFlow addresses dense optical flow and propagates predictions to occluded or unmatched pixels.
3. Methodology
GMFlow formulates optical flow as global feature matching, using Transformer-enhanced features, differentiable softmax correspondence estimation, and self-attention-based propagation. A higher-resolution refinement stage predicts residual flow while reusing GMFlow components.
- 3.1. Formulation: Optical flow is formulated as matching pixels by comparing feature similarities across all locations in the two frames.Dense features are extracted from consecutive frames using a weight-sharing convolutional network before global correlation.
- 3.1. Formulation: The correlation matrix compares every feature in the first frame with every feature in the second frame using matrix multiplication.Each correlation entry represents a pairwise correspondence score, with feature dimension normalization preventing excessively large dot products.
- 3.1. Formulation: Softmax converts correlations into a differentiable matching distribution, whose weighted coordinate average yields correspondences and optical flow.The softmax formulation supports end-to-end training and sub-pixel accuracy.
- 3.2. Feature Enhancement: A Transformer enhances features through symmetric self-attention, cross-attention, and feed-forward processing with positional encoding.The implementation uses shifted local windows and cross-window connections to reduce the quadratic cost of standard pairwise attention.
- 3.3. Flow Propagation: Self-attention propagates reliable matched-pixel flow to occluded and out-of-boundary pixels using feature self-similarity.Backward flow can also be obtained by transposing the global correlation matrix, without forwarding the network twice.
- 3.4. Refinement: The refinement stage upsamples 1/8-resolution flow to 1/4 resolution and uses GMFlow in local windows to predict residual flow.Transformer and self-attention weights are shared with the global matching stage, while backbone features are also shared across resolutions.
4. Experiments
Experiments compare GMFlow with RAFT and analyze its components, training, efficiency, benchmark performance, and generalization. GMFlow performs strongly on Sintel with fewer refinements and faster inference, but remains weaker on KITTI under synthetic-to-real transfer.
- Experimental Setup: GMFlow uses Chairs and Things for training, then evaluates on Things validation, Sintel, and KITTI, with EPE and KITTI F1-all as metrics.Motion-specific EPE is also reported for 0–10, 10–40, and more than 40 pixels.
- Ablations: Global matching outperforms local matching, especially for large motion, while matrix multiplication makes global matching efficient.Larger local matching ranges are slower because they require excessive sampling.
- Ablations: Flow propagation significantly improves performance in unmatched regions, including occluded and out-of-boundary pixels.The authors attribute this gain to structural correlation between features and flow.
- Ablations: Sharing the Transformer and multi-scale features improves performance and reduces parameters; longer training also benefits GMFlow.The final comparison model uses 800K training iterations on Things.
- Comparison with RAFT: On Sintel, GMFlow beats RAFT with 11 refinements without refinement and beats RAFT with 31 refinements using one refinement, while running faster.On A100, reported speedups are 2.29× for GMFlow and 1.87× for RAFT, with times of 151 →66 and 170 →91.
- Generalization and Limitations: GMFlow does not outperform RAFT on KITTI after synthetic training, but the gap becomes smaller after fine-tuning with Virtual KITTI 2.The authors connect the limitation to Transformers requiring relatively more training data for cross-dataset generalization.
5. Conclusion
The paper presents global matching for optical flow and reports strong performance, while cautioning that predictions may be unreliable in occluded regions.
- GMFlow introduces a global matching formulation for accurate and efficient optical flow estimation.
- Predictions may be unreliable in occluded regions, requiring care in safety-critical applications such as self-driving cars.
A. More Comparisons
Additional comparisons find that the Transformer and softmax-based flow estimation method consistently outperforms other tested combinations while using fewer parameters.
- The Transformer and softmax-based method is consistently better and has fewer parameters than other flow-estimation variants.
B. Computational Complexity
The framework offers several implementation choices for managing computational cost, including sequential or local operations and smaller attention windows, while preserving flexibility for different needs.
- Global Matching: Global matching constructs an H × W × H × W correlation matrix for all pair-wise similarities between two features.
- Global Matching: Sequential computation can split global matching into smaller K × K correlation matrices when high-resolution memory becomes a bottleneck.
- Transformer: The Transformer uses shifted local window attention and can switch to smaller windows if computational cost becomes a bottleneck.
- Flow Propagation: Flow propagation defaults to global self-attention but can use local window self-attention to reduce memory consumption at some accuracy cost for large motion.
- Refinement: Higher-resolution refinement is not a significant bottleneck because it uses smaller attention windows and local-window matching.
- GMFlow is general and flexible, allowing concrete implementations to meet specific needs.
C. More Visual Results
Visual results indicate that self-attention flow propagation handles occluded and out-of-boundary pixels effectively, and the model is also evaluated on diverse DAVIS scenes.
- Self-attention flow propagation is effective for handling occluded and out-of-boundary pixels.
- The pre-trained Sintel model is tested on the DAVIS dataset across diverse scenes.
D. More Implementation Details
GMFlow combines Transformer-based feature enhancement, softmax matching, and flow propagation, with refinement using localized attention. Its propagation scheme improves performance on occluded and out-of-boundary pixels, while softmax matching remains parameter-free.
- Network Architectures: GMFlow uses a 128-dimensional Transformer with a 4× feed-forward expansion and single-head attention; refinement reuses it within smaller local windows.The single-head design was chosen because multihead attention slowed inference without obvious performance gains.
- Comparisons: The implementation evaluates flow-estimation variants across Sintel clean and final settings, including matched and unmatched regions.The supplied table text identifies these comparison categories but does not provide their numeric values.
- Comparisons: The softmax-based method is parameter-free and uses 4.20M parameters versus 7.79M for the cost-plus-convolution variant.The cost-plus-convolution approach requires a deep convolutional regressor to catch up in performance.
- Results: Flow propagation significantly improves performance for occluded and out-of-boundary pixels.Visual results are also reported on the DAVIS dataset.