PaperlayerView exact revision on arXiv

Source-linked AI summary

Unifying Flow, Stereo and Depth Estimation

Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, Fisher Yu, Dacheng Tao, Andreas Geiger

arXiv:2211.05783v3cs.CV

TL;DR

The paper addresses the proliferation of task-specific models for optical flow, stereo matching, and depth estimation. It uses shared Transformer features with cross-attention and parameter-free matching layers to solve all three as dense correspondence problems, achieving competitive performance across 10 datasets while enabling cross-task transfer.

  • Problem

    Existing approaches largely design separate architectures for related flow, stereo, and depth tasks, limiting model reuse and increasing deployment complexity.

  • Method

    The method formulates all three tasks as dense correspondence matching using Transformer cross-attention features and parameter-free task-specific matching layers.

  • Results

    The final model outperforms or compares favorably to recent state-of-the-art methods on 10 popular flow, stereo, and depth datasets.

  • Takeaways & Limitations

    Shared learnable parameters enable cross-task transfer, including direct use of a pretrained optical flow model for stereo and depth without finetuning.

  • Takeaways & Limitations

    The method still has larger errors in unmatched regions, relies on RAFT-style iterative refinement for post-processing, and does not yet achieve real-time inference.

Abstract

from arXiv · show

We present a unified formulation and model for three motion and 3D perception tasks: optical flow, rectified stereo matching and unrectified stereo depth estimation from posed images. Unlike previous specialized architectures for each specific task, we formulate all three tasks as a unified dense correspondence matching problem, which can be solved with a single model by directly comparing feature similarities. Such a formulation calls for discriminative feature representations, which we achieve using a Transformer, in particular the cross-attention mechanism. We demonstrate that cross-attention enables integration of knowledge from another image via cross-view interactions, which greatly improves the quality of the extracted features. Our unified model naturally enables cross-task transfer since the model architecture and parameters are shared across tasks. We outperform RAFT with our unified model on the challenging Sintel dataset, and our final model that uses a few additional task-specific refinement steps outperforms or compares favorably to recent state-of-the-art methods on 10 popular flow, stereo and depth datasets, while being simpler and more efficient in terms of model design and inference speed.

1 INTRODUCTION

The paper unifies optical flow, rectified stereo matching, and unrectified stereo depth estimation as dense correspondence problems solved by shared features and matching layers. This design supports cross-task transfer and achieves competitive results across flow, stereo, and depth benchmarks.

  • Unified formulation: 3 tasks are recast as dense correspondence matching by directly comparing feature similarities.The tasks are optical flow, rectified stereo matching, and unrectified stereo depth estimation from posed images.
  • Feature representation: Cross-attention integrates information from the other image, producing more discriminative matching features than independently processed views.The paper contrasts this cross-view interaction with classical convolutional backbones that operate on each view independently.
  • Cross-task transfer: The same learnable feature-extraction parameters support direct transfer from pretrained optical flow to stereo and depth without finetuning.Finetuning from the pretrained flow model also gives faster stereo and depth training and better performance in the reported experiments.
  • Results: The unified model with one task-agnostic hierarchical matching refinement outperforms RAFT on the challenging Sintel dataset while running faster.The result is reported for the unified model using only one task-agnostic hierarchical matching refinement.
  • Results: The final model outperforms or compares favorably to recent state-of-the-art methods on 10 popular flow, stereo, and depth datasets.The model remains simpler and more efficient in model design and inference speed.
  • Contributions: The work extends GMFlow from optical flow to rectified stereo matching and unrectified stereo depth estimation and studies cross-task transfer.The authors describe the paper as a substantial extension of their CVPR 2022 conference paper.

2 RELATED WORK

Prior work largely uses task-specific architectures for optical flow, stereo, and depth, despite their shared correspondence structure. The paper positions unified matching as an alternative to specialized cost volumes, iterative refinements, and task-specific network components.

  • Optical flow: Optical-flow methods trade off large-displacement estimation against cost-volume size or use coarse-to-fine and iterative refinements with distinct failure modes.Coarse-to-fine methods can miss fast-moving small objects, whereas iterative methods such as RAFT increase processing time linearly with sequential refinements.
  • Optical flow: The paper reformulates optical flow as global matching, directly comparing pair-wise feature similarities to improve large-displacement estimation.This contrasts with local correlation followed by convolutional regression.
  • Stereo: Stereo methods commonly introduce task-specific components such as 1D correlation, whereas the proposed perspective uses identical learnable parameters for flow and stereo.This comparison motivates sharing the same model across correspondence tasks.
  • Stereo: STTR also uses Transformer matching, but relies on complex optimal transport and does not predict occluded pixels; the proposed method uses softmax matching and propagation.The paper distinguishes its goal of covering three tasks from STTR’s stereo-only design.
  • Unified models: Unified models differ from multi-task joint training because they target one network applicable to several tasks without task-specific learnable parameters.The paper contrasts this with jointly trained systems that retain task-specific architectures.

3 METHODOLOGY

The methodology represents three correspondence tasks with shared Transformer-enhanced features and parameter-free task-specific matching layers. A final self-attention propagation step extends predictions to unmatched regions.

  • Feature extraction: Cross-attention integrates information from another image, strengthening features for dense correspondence matching.This cross-view interaction is presented as unavailable to convolutions that process each view independently.
  • Unified formulation: Dense correspondence matching is the shared formulation underlying optical flow, rectified stereo matching, and unrectified stereo depth estimation.The formulation identifies solutions through direct feature-similarity comparisons.
  • Unified model: The model extracts dense features from two images, applies a parameter-free matching layer, and uses self-attention to propagate predictions to unmatched regions.Propagation relies on feature self-similarity.
  • Matching layers: Task-specific matching layers encode different geometric constraints while remaining parameter-free because they only compare feature similarities.All three tasks therefore use exactly the same learnable parameters, enabling cross-task transfer.

3.1 Formulation

The formulation treats optical flow, rectified stereo matching, and unrectified stereo depth estimation as parameter-free dense correspondence problems over image features. Each task converts a task-specific matching distribution into flow, disparity, or depth predictions.

  • 3.1 Formulation: The model accepts two images and 8× downsampled dense features, then applies parameter-free matching layers for the three tasks.For unrectified depth, the images additionally require known intrinsic and extrinsic camera parameters.
  • 3.1.1 Flow Matching: Optical flow globally compares every feature in the first image with every feature in the second using correlations computed by matrix multiplication.The correlation uses dot products with a normalization factor based on feature dimension D.
  • 3.1.1 Flow Matching: A softmax over flow correlations produces a matching distribution whose coordinate-weighted average gives correspondences and whose coordinate difference gives optical flow.The softmax matching layer is differentiable and supports sub-pixel accuracy.
  • 3.1.2 Stereo Matching: Rectified stereo restricts correspondence search to the horizontal scanline, making it a one-dimensional special case of two-dimensional optical-flow matching.Its matching distribution is normalized over the disparity correlation dimension, and implausible upper-triangle matches are masked.
  • 3.1.2 Stereo Matching: Stereo disparity is obtained by averaging all potential horizontal locations with the matching distribution and subtracting the resulting coordinate from the reference horizontal grid.This produces a positive disparity under the stated leftward-correspondence convention.
  • 3.1.3 Depth Matching: For posed unrectified stereo, the method discretizes a depth range, projects image-grid coordinates for each candidate depth, and samples corresponding features.Known camera intrinsics and extrinsics define the projection between the two images.
  • 3.1.3 Depth Matching: Depth candidates produce correlations that are concatenated, normalized into a matching distribution, and averaged with the candidate depths to estimate depth.The resulting depth prediction has spatial dimensions H×W.
  • 3.1 Formulation: All three matching layers are differentiable and parameter-free, so GMFlow, GMStereo, and GMDepth share the same unified global-matching formulation.The design supports end-to-end training without task-specific learnable parameters in the matching layers.

3.2 Feature Extraction

Feature extraction combines shared convolutional features with Transformer processing, using cross-attention to enhance representations through cross-view interactions. Adaptive local-window attention reduces computational cost while preserving connections useful for large displacements.

  • Feature extraction: A weight-sharing ResNet extracts 8× downsampled features before Transformer-based feature enhancement.The downsampling keeps computation tractable, while the CNN initially processes the two images independently.
  • Feature enhancement: The final Transformer stacks six blocks containing self-attention, cross-attention, and feed-forward networks.The blocks progressively improve performance according to the cited ablation.
  • Feature enhancement: Fixed sine-cosine positional encodings add spatial information so matching uses both feature similarity and spatial distance.The paper reports that positional information helps resolve ambiguities and improves performance.
  • Feature enhancement: Cross-attention lets each image’s features incorporate information from the other image through cross-view interactions.The Transformer uses the other image’s features as key and value while the first image supplies the query.
  • Efficient attention: Local-window self- and cross-attention replaces quadratic global pairwise attention and uses shifted partitions to create cross-window connections.The windows are adaptive to feature-map size, supporting long-range dependencies on high-resolution maps.
  • Task-specific attention: Rectified stereo uses one-dimensional horizontal cross-attention because full two-dimensional attention is redundant for horizontal correspondences.The one-dimensional operation is faster, while learnable parameters remain identical across the three tasks.

3.3 Propagation

Matching assumes corresponding pixels are visible in both images, so unmatched occluded and out-of-boundary regions require a separate propagation mechanism. A self-attention layer propagates reliable predictions using feature self-similarity.

  • Propagation: The matching formulation can produce unreliable predictions where pixels are occluded or outside the other image’s boundary.These regions violate the visibility assumption required for direct similarity matching.
  • Propagation: The method propagates high-quality flow, disparity, or depth predictions to unmatched regions by measuring feature self-similarity.The proposed operation uses a single self-attention layer and exploits structural similarity between images’ features and predicted fields.
  • Propagation: The propagation layer receives the softmax matching-layer prediction as V and learns how to correct unmatched regions under ground-truth flow supervision.The model does not explicitly label pixels as matched or unmatched.
  • Output resolution: Predictions are upsampled from 1/8 feature resolution to full image resolution using learned weighted combinations of 3×3 coarse-resolution neighborhoods.A small two-layer convolutional network produces the weights for the 8× upsampling.

3.4 Refinement

The unified model can be refined with higher-resolution matching and optional local regression, combining global matching’s strength on large motion with local refinement’s strength on small or fine-grained structure. Some of these additional refinements introduce task-specific components.

  • 3.4 Refinement: Additional refinement steps improve the base 1/8-resolution model and expose different speed-accuracy trade-offs.The paper evaluates hierarchical matching with 1/4-resolution features and convolutional local regression.
  • Hierarchical matching refinement: Hierarchical matching upsamples the coarse prediction to 1/4 resolution, warps the second feature, and performs local-range matching for flow and rectified stereo.The same matching model is reused at higher resolution without task-specific learnable parameters; depth omits this stage because its improvement was less significant.
  • Hierarchical matching refinement: Transformer and self-attention weights are shared between the 1/8 global and 1/4 local matching stages.The paper reports that sharing reduces parameters and improves generalization.
  • Hierarchical matching refinement: A task-agnostic hierarchical refinement outperforms RAFT with 31 refinements while running faster.The comparison is presented as evidence for the effectiveness and efficiency of global matching.
  • Local regression refinement: Global matching is particularly strong for large motion, whereas local regression is advantageous for small motions.The paper combines them by applying local regression as post-processing to improve fine-grained details and hard-to-match regions.
  • Local regression refinement: Optional iterative refinements regress updates from local correlations, using task-specific correlation constructions for flow, stereo, and depth.The number of refinements is selected empirically and differs by task.

3.5 Training Loss

The model supervises intermediate and final predictions using a weighted sum of task-specific losses, with later predictions receiving greater weight.

  • All intermediate network outputs and final predictions are supervised against the ground truth.
  • The training objective weights prediction losses exponentially with γ = 0.9, emphasizing later predictions as in RAFT.
  • Optical flow uses L1 loss, rectified stereo matching uses smooth L1 loss, and unrectified stereo depth uses inverse-depth L1 loss.
  • The depth objective combines inverse-depth and gradient losses with both combination weights set to 20.

4 EXPERIMENTS

Experiments evaluate GMFlow’s optical-flow formulation, Transformer components, matching space, propagation strategy, efficiency, and transfer to stereo. The unified model generally improves accuracy and speed, especially for large motions, while KITTI generalization exposes a domain gap.

  • Methodology Comparison: The methodology comparison evaluates convolutional residual blocks and Transformer blocks on Things validation and Sintel cross-dataset generalization after Chairs-and-Things training.Performance is reported across motion ranges, including s40+ large motions.
  • Ablations: Cross-attention contributes most among Transformer components by modeling cross-view interactions, while positional information, self-attention, and the feed-forward network provide additional gains.Removing the feed-forward network reduces parameters but causes a moderate performance drop.
  • Ablations: Global matching causes a significant performance drop when replaced by local matching, especially for large motions, while matrix multiplication keeps global matching efficient.The local formulation reduces all-pairs matching to a K × K window and requires more sampling as the window grows.
  • Ablations: Flow propagation significantly improves unmatched regions, including occluded and out-of-boundary pixels, using structural correlation between features and flow.The reported gains appear in Table 2d and Fig. 3.
  • Comparison with RAFT: One hierarchical matching refinement outperforms RAFT with 31 refinements while running faster; on A100, GMFlow reaches 2.29× speedup versus RAFT’s 1.87×.The reported A100 timings are GMFlow 151 →66 and RAFT 170 →91 at Sintel resolution.
  • Comparison with RAFT: On KITTI without additional real-world training, GMFlow does not outperform RAFT, attributed to the synthetic-to-real gap and convolutional inductive biases.The passage states that Transformers require relatively more training data to generalize to a new dataset.
  • Stereo Ablations: For rectified stereo, 1D horizontal cross-attention is faster and better than 2D attention because it avoids unnecessary matching information while preserving shared learnable parameters.The parameter-free attention operation does not change the Transformer’s learnable projection layers, enabling pretrained-model sharing.
  • Stereo Comparison: GMStereo with random initialization significantly outperforms RAFT-Stereo with fewer parameters and faster inference, while flow initialization provides a further performance boost.The larger margin from flow initialization demonstrates cross-task transfer in the stereo comparison.

4.3 Depth Prediction

The depth experiments evaluate cross-task transfer, component contributions, and comparisons with RAFT and related depth models. The unified model supports direct flow-to-depth transfer, while depth transfer remains more challenging than stereo transfer.

  • Cross-attention contributes most to depth performance gains, while the other model components also improve results.
  • Flow to depth transfer: A pretrained optical-flow model can directly predict ScanNet depth without finetuning, with further improvement after ScanNet finetuning.
  • Comparison with depth models: The model achieves similar ScanNet performance to DeFiNe with 4× fewer parameters and 2× faster inference.
  • Comparison with depth models: The proposed Transformer features and parameter-free matching outperform the compared Transformer-enhanced cost-volume approach.
  • Flow to depth transfer: Compared with RAFT, the method performs better on RMSE log but worse on Abs Rel, Sq Rel, and RMSE for flow-to-depth transfer.
  • Flow to depth transfer: Depth transfer may contain large outliers, while finetuning the pretrained flow model improves adaptation and eventually outperforms the triangulation-based approach.

4.5 Benchmark Results

Benchmark experiments apply task-specific refinements to the unified model across optical flow, stereo, and depth datasets. The model reports highly competitive results together with efficiency advantages on several benchmarks.

  • Optical Flow: The model achieves state-of-the-art results on Sintel clean and outperforms all previous methods on KITTI optical flow.
  • Stereo Matching: The final stereo model uses three local regression refinements after one hierarchical refinement, with 2D correlation outperforming 1D correlation.
  • Stereo Matching: On Middlebury, GMStereo ranks first by RMS disparity error and runs 5× faster than CREStereo and 15× faster than RAFT-Stereo.
  • Stereo Matching: GMStereo achieves first place in the Argoverse Stereo Challenge and clearly outperforms the other submissions.
  • Depth Estimation: For unrectified depth, the final model omits hierarchical refinement and adds local regression refinement at 1/8 resolution.
  • Depth Estimation: Compared with IIB, the method has similar performance while being more lightweight and faster.

5 LIMITATION AND DISCUSSION

The paper identifies limitations in unmatched-region accuracy, reliance on RAFT-style post-processing, real-time speed, and extension beyond two-frame tasks. Joint training of all three tasks is also left for future work.

  • Unmatched regions remain substantially less accurate than matched regions, despite Sintel matched-region errors of 0.34 pixels on clean and 1.10 pixels on final.
  • The method still relies on RAFT’s iterative refinement as post-processing, although a lighter alternative may be possible.
  • The full model does not yet achieve real-time inference at 20 FPS or more.
  • The demonstrated approach covers multiple two-frame tasks, while extension to multi-view settings remains future work.
  • The paper does not show experiments training one model on all three tasks simultaneously, and balancing joint training remains challenging.

6 CONCLUSION

The paper unifies optical flow, rectified stereo, and unrectified stereo depth as dense correspondence matching. Transformer cross-attention strengthens cross-view features, enabling shared parameters, transfer, and competitive performance with simpler, more efficient models.

  • Optical flow, rectified stereo matching, and unrectified stereo depth estimation are formulated as one dense correspondence matching problem.
  • Transformer cross-attention integrates information from the other image to produce stronger, more discriminative correspondence features.
  • Shared learnable parameters naturally enable transfer across the three tasks.
  • The final model achieves state-of-the-art or highly competitive performance on 10 flow, stereo, and depth datasets while remaining simpler and more efficient.
Loading 2211.05783v3…