PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Frame Self-Supervised Depth with Transformers

Vitor Guizilini, Rares Ambrus, Dian Chen, Sergey Zakharov, Adrien Gaidon

arXiv:2204.07616v2cs.CV

TL;DR

Multi-frame depth estimation adds geometric cues through feature matching, but correspondences can be noisy and ambiguous. DepthFormer uses epipolar sampling and attention to refine matching probabilities, achieving state-of-the-art self-supervised results and transferable representations.

  • Problem

    Multi-frame methods use geometric information at inference, but feature correspondences can be noisy and inaccurate because of ambiguities and local minima.

    Source: §1 · p. 1

  • Method

    DepthFormer builds a cost volume from depth-discretized epipolar candidates and refines per-pixel matching probabilities with self- and cross-attention.

    Source: §1 · p. 2

  • Results

    DepthFormer establishes state-of-the-art results on KITTI and DDAD, outperforming self-supervised methods and comparing favorably with state-of-the-art supervised single-frame architectures.

    Source: §5 · pp. 8–9

  • Takeaways & Limitations

    The learned cross-attention module transfers across datasets without fine-tuning to speed convergence and decrease training memory requirements.

    Source: §5 · p. 9

    The method does not explicitly model dynamic objects or static frames, and monocular self-supervision retains scale ambiguity that prevents metrically accurate predictions.

    Source: F. Limitations · p. 11

Abstract

from arXiv · show

Multi-frame depth estimation improves over single-frame approaches by also leveraging geometric relationships between images via feature matching, in addition to learning appearance-based features. In this paper we revisit feature matching for self-supervised monocular depth estimation, and propose a novel transformer architecture for cost volume generation. We use depth-discretized epipolar sampling to select matching candidates, and refine predictions through a series of self- and cross-attention layers. These layers sharpen the matching probability between pixel features, improving over standard similarity metrics prone to ambiguities and local minima. The refined cost volume is decoded into depth estimates, and the whole pipeline is trained end-to-end from videos using only a photometric objective. Experiments on the KITTI and DDAD datasets show that our DepthFormer architecture establishes a new state of the art in self-supervised monocular depth estimation, and is even competitive with highly specialized supervised single-frame architectures. We also show that our learned cross-attention network yields representations transferable across datasets, increasing the effectiveness of pre-training strategies. Project page: https://sites.google.com/tri.global/depthformer

1. Introduction

Multi-frame methods use geometric information at inference, but their feature matches can be noisy and ambiguous. DepthFormer refines matching probabilities with attention and achieves strong depth-estimation results, with learned matching representations that transfer across datasets.

1. Introduction

  • Multi-frame methods use multi-view information at inference and learn geometric as well as appearance-based features, but matching remains noisy and often inaccurate.Ambiguities and local minima can arise from texture, repetition, luminosity changes, and dynamic objects.Source: §1 · p. 1
  • The approach targets self-supervised monocular depth estimation by improving feature matching between images.Feature matching establishes cross-frame correspondences and supports depth estimation [5] [20].Source: §1 · p. 1
  • DepthFormer combines depth-discretized epipolar sampling with self- and cross-attention to refine per-pixel matching probabilities.The refined probabilities are reported to be sharper and more representative of underlying 3D structure than traditional similarity metrics.Source: §1 · p. 2
  • DepthFormer achieves state-of-the-art depth estimation, outperforming self-supervised multi-frame methods and even surpassing supervised single-frame architectures.The paper also reports strong results across KITTI and DDAD.Source: §1 · p. 2
  • The learned attention-based matching function transfers across datasets and can improve convergence speed while decreasing memory.The conclusion reports using the cross-attention module without fine-tuning across datasets to speed convergence and reduce training memory requirements.Source: §1 · p. 2

2. Related Work

Prior work spans self-supervised view-synthesis depth estimation and multi-frame methods that add geometric cues at inference. This section situates attention-based matching among these approaches and contrasts their settings and limitations.

2.1. Self-Supervised Depth Estimation

  • Monocular self-supervision estimates depth from view synthesis and raw videos, but faces scale ambiguity and difficulty modeling dynamic objects.Learning from raw videos makes this approach scalable to data from different sources [20], despite these limitations [20].Source: §2.1 · p. 2

2.2. Multi-Frame Depth Estimation

  • Multi-frame inference adds geometric cues through cross-frame matching, while existing cost-volume methods still rely on hand-crafted metrics prone to ambiguity and local minima.Single-image depth is ill-posed; implicit multi-view methods may focus on static scenes and high-overlap views, limiting large-scale generalization [20].Source: §2.2 · p. 2

2.3. Attention for Depth Estimation

  • Prior attention approaches include supervised stereo cost-volume decoding [48] and self-supervised spatio-temporal attention [60]; Figure 2 depicts epipolar candidates refined by self- and cross-attention into matching probabilities.The latter approach focuses on 3D points rather than the epipolar constraints used here [60].Source: §2.3 · pp. 2–3, Figure 2 · p. 3

3. Self-Supervised Depth with Transformers

The method builds a multi-frame cost volume by sampling context-image features along epipolar lines at discretized depths, then refines matching with attention. It decodes the refined matches into depth estimates, supplements them with single-frame features, and trains the system end-to-end using photometric reprojection loss.

3.1. Monocular Depth Estimation

  • The monocular depth and pose networks are trained jointly by minimizing photometric reprojection error between the target image and a synthesized view.The synthesized image is formed by projecting context-image pixels using predicted depth and pose.Source: §3.1 · p. 3

3.2.1 Monocular Epipolar Sampling

  • Context features are sampled along each target feature’s epipolar line at depth-discretized candidates, forming a feature volume.Depth values are uniformly sampled in log space using spatial-increasing discretization; camera motion comes from the pose network and intrinsics are assumed known and constant.Source: §3.2.1 · p. 3

3.2.2 Cross-Attention Matching

  • Cross-attention refines target-to-context matching probabilities, producing sharper distributions than SSIM-based similarity and reducing ambiguities and local minima.The final attention values populate a cost volume whose bins encode similarities between target features and matching candidates.Source: §3.2.2 · p. 4, Figure 3 · p. 4

3.2.3 Self-Attention Refinement

  • Self-attention among epipolar-sampled context features alternates with cross-attention, with refinement omitted after the final cross-attention layer.The final layer’s attention weights populate the cost volume, so its output values are not needed for another self-attention update.Source: §3.2.3 · p. 4

3.3.1 High-Response Depth Decoding

  • A localized window around the most probable attention bin yields continuous depth estimates from discretized candidates, improving robustness to multi-modal distributions.Maximum attention also serves as matching confidence; pixels below a threshold are masked from high-response loss calculation and decoded features.Source: §3.3.1 · pp. 4–5

3.3.2 Context-Adjusted Depth Decoding

  • Because epipolar cost-volume regression lacks surrounding context, a residual context-adjustment layer conditions estimated depths on the input images.Its output is added to the normalized high-response depth map before restoration using the same statistics.Source: §3.3.2 · p. 5

3.3.3 Multi-Scale Depth Decoding

  • The multi-scale decoder concatenates masked cross-attention cost-volume features with single-frame features and produces depth maps at increasing resolutions.Joint teacher-student training uses the single-frame network to supervise areas where cost-volume generation fails; that network is discarded at evaluation.Source: §3.3.3 · p. 5, Figure 5 · p. 5

3.4. Training Loss

  • End-to-end training uses a photometric reprojection objective combining SSIM and absolute error, plus depth regularization for smoothness in low-textured regions.The loss is aggregated across high-response, context-adjusted, and multi-scale depth outputs.Source: §3.4 · pp. 5–6

4. Experiments

Experiments evaluate DepthFormer on KITTI and DDAD, examine its components, and test cross-dataset reuse of its cross-attention cost volume. Results include state-of-the-art depth estimates and faster, lower-memory training with a frozen pretrained matching network.

4.1. Datasets

  • DepthFormer is evaluated on KITTI, DDAD, Cityscapes, VKITTI2, Parallel Domain, and TartanAir, with dataset-specific splits and training data.TartanAir context frames are selected when average optical flow is between 8 and 96 pixels.Source: §4.1 · p. 6

4.2. Implementation Details

  • Training uses 50 epochs, 128 epipolar depth bins, six attention layers, and four output scales, with pose and teacher networks frozen for the final five epochs.Training uses eight Titan V100 GPUs and frame t−1 as context for cost-volume calculation.Source: §4.2 · p. 6

4.3. Depth Evaluation

  • DepthFormer outperforms the considered KITTI methods, including supervised single-frame approaches, and achieves state-of-the-art results on DDAD.The KITTI comparison includes methods using semantic labels or synthetic data; DDAD evaluation covers longer depth ranges and more dynamic objects.Source: §4.3 · pp. 6–7, Table 1 · p. 7, Table 3 · p. 7
  • Replacing SAD or SSIM volumes with the cross-attention cost volume improves Abs.Rel. from 0.647 and 0.632 to 0.264, then context adjustment reaches 0.167 and joint decoding reaches 0.090.The reported progression adds high-response depth supervision, context adjustment, and finally single-frame features for joint decoding.Source: §4.3 · p. 7, Table 2 · p. 7

4.4. Ablation Analysis

  • Ablations identify cross-attention, high-response and context-adjusted depth self-supervision, and self-attention on context features as important components.The high-response and context-adjusted weights produce similar results across λH = λC = [0.1, 1.0].Source: §4.4 · pp. 7–8, Table 4 · p. 8
  • Increasing cross-attention network complexity improves results, although simpler configurations remain competitive.The tested variations modify attention layers, feature channels, and depth bins.Source: §4.4 · p. 8

4.5. Cost Volume Generalization

  • Swapping in a cross-attention network pretrained on another dataset causes around 5% performance degradation, while frozen-network training is faster and uses less memory.Frozen pretraining yields around 100% faster training, from 7.3 to 14.2 FPS, and around 20% less memory, from 15.3 to 12.4 GB; fine-tuning all networks for five epochs then reproduces the reported state-of-the-art results.Source: §4.5 · p. 8, Table 5 · p. 8

5. Conclusion

The paper proposes an attention-based cost-volume procedure for self-supervised multi-frame monocular depth estimation. It reports state-of-the-art results on KITTI and DDAD and transferable cross-attention matching across datasets.

5. Conclusion

  • The proposed attention-based cost-volume procedure targets self-supervised multi-frame monocular depth estimation.Cross-attention refines feature matching between images beyond traditional appearance-based similarity metrics.Source: §5 · p. 8
  • DepthFormer establishes state-of-the-art results on KITTI and DDAD, outperforming other single- and multi-frame self-supervised methods and matching state-of-the-art supervised single-frame architectures.The conclusion also reports that the learned cross-attention module can be used across datasets without fine-tuning to speed convergence and reduce training memory.Source: §5 · p. 9

A. Network Details

The appendix describes modular network components for DepthFormer, while noting that its contributions are not tied to a particular network architecture.

A. Network Details

  • The contributions can be incorporated into different network architectures rather than requiring one specific design.The appendix points to recent developments as potential additions.Source: A. Network Details · p. 9

A.1. Cross-Attention Network

  • The cross-attention encoder uses residual connections and spatial pyramid pooling, and its decoder produces full-resolution features before downsampling them for the cost volume.The encoder is hourglass-shaped; the decoder combines transposed convolutions, dense blocks, and a final convolution.Source: §A.1 · p. 9

A.2. Single-Frame Depth Network

  • The single-frame network uses a ResNet18 encoder and predicts depth at four resolutions, combining its H/4 features with the multi-frame cost volume.A bottleneck convolution combines the feature sources before further encoding and decoding.Source: §A.2 · p. 9

A.3. Context Adjustment Network

  • The context-adjustment network takes the normalized high-response depth map and target image, then produces a context-adjusted depth estimate.Residual blocks refine the depth map, which is un-normalized using the original statistics.Source: §A.3 · p. 9

A.4. Pose Network

  • The pose network predicts relative translation and rotation between frames using a ResNet18 backbone adapted for two input images.Its final six-dimensional vector represents motion in Euler angles.Source: §A.4 · p. 9

B. Comparison to Supervised Methods

The supervised-method comparison emphasizes DepthFormer’s performance despite its self-supervised training, while identifying resolution as a relevant condition.

B. Comparison to Supervised Methods

  • At half resolution, DepthFormer achieves performance comparable to BTS when training and evaluating at 640 × 192.The comparison concerns a self-supervised architecture evaluated against a supervised single-frame method.Source: B. Comparison to Supervised Methods · p. 9
  • At full resolution, 1216 × 352, DepthFormer surpasses BTS in almost all metrics.The passage suggests high-resolution self-supervised architectures as a direction for further improvement, while leaving that exploration for future work.Source: B. Comparison to Supervised Methods · p. 9
  • DepthFormer is trained without explicit ground-truth depth supervision, despite limitations of the photometric objective with dynamic objects, static frames, and luminosity changes.The comparison passage frames these as challenges of the self-supervised setting.Source: B. Comparison to Supervised Methods · p. 9

C. Qualitative Examples

The examples show how low-confidence depth regions are masked or context-adjusted, while joint decoding adds single-frame features to address difficult matching cases. The accompanying network descriptions outline how target and context images feed cost-volume generation.

C. Qualitative Examples

  • Low-confidence regions are masked in high-response depth maps and commonly include distant objects near the vanishing point, sky, occlusions, and dynamic objects.Context-adjusted maps use target-image information to reason over these regions, but can still fail with inaccurate matching or ill-posed cases.Source: C. Qualitative Examples · pp. 9, 11
  • The single-frame depth network takes the target image and cross-attention cost volume as inputs, while the attention network processes target and context images independently to generate that volume.The architecture table also defines its listed operations and symbols, including batch normalization and feature concatenation.Source: C. Qualitative Examples · p. 10, Table 6 · p. 10
  • Joint decoding combines single-frame features to reason over cases where matching is inaccurate or ill-posed, including scenes without camera motion or with dynamic objects.The authors report state-of-the-art results with this addition.Source: C. Qualitative Examples · p. 11

D. Reconstructed Pointclouds

The paper illustrates reconstructions from predicted depth and relative motion, produced using videos without ground-truth at training or inference. The examples include varied scene structures, with accuracy also assessed quantitatively.

D. Reconstructed Pointclouds

  • The KITTI and DDAD pointclouds are reconstructed by unprojecting pixel colors with camera intrinsics, predicted depth maps, and predicted relative motion.Examples of reconstructions over entire sequences are provided in a supplementary video.Source: D. Reconstructed Pointclouds · p. 11
  • Training and inference use videos only, with no ground-truth.Source: D. Reconstructed Pointclouds · p. 11
  • The reconstructions depict low-texture regions, object boundaries, and dynamic objects, and their accuracy is evaluated quantitatively in Table 7.Source: D. Reconstructed Pointclouds · p. 11

E. Negative Impact

The monocular self-supervised method can process arbitrarily large amounts of unlabeled visual data without human intervention. The authors nevertheless identify data imbalance and privacy as concerns requiring care.

E. Negative Impact

  • The method can process arbitrarily large amounts of unlabeled visual data without human intervention.Source: E. Negative Impact · p. 11
  • More data is not necessarily better; data curation remains desirable to avoid bias from data imbalance.Source: E. Negative Impact · p. 11
  • Processing large quantities of data without supervision raises privacy concerns, and procedures should preserve individual anonymity.Source: E. Negative Impact · p. 11

F. Limitations

DepthFormer improves robustness to some common challenges but does not explicitly model dynamic objects or static scenes. Monocular self-supervision also leaves scale ambiguous, although scale-aware methods could be combined with it.

F. Limitations

  • The method improves robustness to dynamic objects and static frames through better feature matching, but does not explicitly address them; doing so would require 3D motion modeling such as scene flow or tracking.Source: F. Limitations · p. 11
  • Purely image-based monocular training has scale ambiguity, so predictions are not metrically accurate; scale-aware estimates matter for downstream uses such as 3D object detection.Source: F. Limitations · p. 11
  • The method does not resolve scale ambiguity, but can benefit from approaches adding weak velocity supervision or geometric information such as camera height or multi-camera extrinsics.Source: F. Limitations · p. 11
Loading 2204.07616v2…