Source-linked AI summary
CrossDepth: Geometry-Constrained Attention for Generalizable Multi-View Surround Depth Estimation
Samer Abualhanud, Max Mehltretter
TL;DR
Surround-view depth estimation must handle minimal overlap and camera-dependent monocular cues, which can produce inconsistent predictions across images. CrossDepth combines camera-ray-conditioned features with geometrically constrained cross-image attention in a fully self-supervised framework, achieving stronger accuracy and cross-view consistency across in-domain and cross-domain evaluations. Its scalability to substantially larger and more diverse datasets remains unevaluated.
Problem
Minimal overlap forces most surround-view pixels to rely on monocular cues that vary with camera intrinsics and image context, creating cross-image depth inconsistency.
Method
CrossDepth uses per-pixel camera-ray embeddings and cross-image attention restricted to geometrically plausible neighborhoods, trained with photometric consistency.
Results
The method achieves higher overall, overlapping-region, and cross-view depth accuracy than state-of-the-art approaches on most metrics under in-domain and cross-domain evaluation.
Takeaways & Limitations
Geometrically constrained attention positively affects both depth accuracy and cross-image consistency compared with unconstrained attention.
Takeaways & Limitations
The method’s scalability to substantially larger training scales and more diverse datasets remains unevaluated because of computational constraints.
Abstract
from arXiv · showhide
Reliable 3D understanding of the surrounding environment is a core requirement for autonomous driving. Multi-view surround camera rigs provide broad scene coverage, but the spatially adjacent images typically overlap only minimally. Consequently, the depth of most pixels must be inferred from monocular appearance cues. These cues can appear differently across images and may therefore be interpreted differently by the depth estimation model. We target two main sources of cross-image inconsistency: differences in camera intrinsics and the limited receptive field of each image. We address the former by conditioning the features on per-pixel camera-aware ray embeddings, enabling the network to account for camera-dependent variations in monocular cues. We address the latter by extending each pixel's context beyond its own image through cross-image attention constrained to geometrically plausible regions, derived from the calibrated rig setup. The model is trained in a fully self-supervised manner based on photometric consistency. Evaluations on DDAD and nuScenes show improved overall depth accuracy and cross-image depth consistency over state-of-the-art self-supervised methods under in-domain and cross-domain evaluation. Code is available at https://abualhanud.github.io/CrossDepthPage/.
1. Introduction
Surround-view cameras provide broad coverage and metric-scale cues, but limited overlap leaves most depth to monocular inference. Cross-image inconsistencies arise from camera-dependent appearance and differing image context, motivating ray conditioning and geometry-constrained attention.
- Motivation: Surround-view rigs provide 360° coverage and metric scale through calibrated baselines, enabling self-supervised metric depth without ground-truth labels.Their limited overlap nevertheless makes stereo-based depth feasible for only a few pixels.
- Motivation: Limited overlap means most pixels require monocular depth cues, which can appear differently across cameras because of varying intrinsics.The same 3D object may have different perspectives and apparent sizes across images.
- Camera-aware features: Per-pixel sinusoidal camera-ray embeddings condition features on camera geometry and model the relationship between appearance, intrinsics, and metric depth.This replaces focal-length normalization and global depth scaling as the stated strategy for handling camera-dependent cues.
- Geometry-constrained attention: Cross-image attention exchanges context across views while restricting each token to geometrically plausible neighborhoods on a shared cylindrical representation.Discretizing the cylinder reduces the cost of neighborhood construction and attention.
- Training and evaluation: The framework is trained fully self-supervised with photometric consistency and evaluated for in-domain and cross-domain generalization on DDAD and nuScenes.The stated contributions include efficient cylindrical attention and per-pixel camera-ray conditioning.
2. Related Work
Prior work estimates depth from monocular or overlapping multi-view imagery, but surround rigs combine wide coverage with limited overlap. Cross-image methods use shared geometric representations, including cylindrical projections, while the proposed framework is illustrated as a two-pass attention-based pipeline.
- Monocular depth estimation: Supervised monocular methods require costly, often sparse ground-truth depth, whereas self-supervised methods enforce photometric consistency using adjacent images or frames.Most existing approaches operate on perspective images with limited fields of view.
- Multi-view depth estimation: Multi-view depth methods either warp features across candidate depths into cost volumes or use other reconstruction paradigms, but quality generally depends on sufficient image overlap.The supplied passage introduces these as two broad learning-based paradigms.
- Surround depth estimation: Surround-depth methods exploit wide coverage and known baselines for metric depth, but limited adjacent-view overlap requires most pixels to be inferred monocularly.Perspective surround images also exhibit less distortion than omnidirectional imagery.
- Proposed pipeline: The proposed pipeline independently encodes target images, predicts an initial depth map, restricts cross-image attention through cylindrical projection, and produces the final depth prediction.A pose network additionally predicts relative transformations between target and source images.
- Surround depth estimation: CylinderDepth projects views onto a shared cylinder and weights cross-image attention by cylindrical proximity to promote cross-view consistency.The proposed method’s overview instead describes an initial depth pass followed by cylindrical projection and restricted attention in a second pass.
3. Methodology
The method combines shared cylindrical cross-image feature aggregation with camera-aware ray conditioning in a multi-view encoder-decoder. Geometry constrains cross-image attention to plausible neighborhoods, while ray embeddings help interpret monocular cues across differing camera intrinsics.
- Network architecture: The network independently encodes each surround-view image with a shared ViT encoder, then decodes fused features into a depth map for every input image.Selected encoder features are processed by cross-image aggregation, fused with camera-aware features, and passed through a CNN-based DPT decoder.
- Cross-image feature aggregation: Cross-image attention expands each image’s receptive field while restricting token interactions to geometrically plausible regions.The restriction is constructed from a preliminary forward pass and calibrated camera geometry before attention is applied at each extracted encoder layer.
- Cross-image feature aggregation: Tokens from all images are projected onto a shared vertical cylinder, where overlapping regions become nearby locations for neighborhood retrieval across views.Each token is mapped to 3D using preliminary depth, camera intrinsics, and camera-to-ego transformations before cylindrical projection.
- Cross-image feature aggregation: Each query token attends only to candidate keys in a local cylindrical window, reducing interactions from Z^2 to ZM with M ≪ Z.Angular windows wrap around cylinder boundaries, while invalid vertical cells are ignored and multiple tokens may occupy one cell.
- Camera-aware ray embedding: Camera-aware ray embeddings condition decoder features on per-pixel ray directions determined by camera intrinsics.Ray directions are sinusoidally encoded and concatenated with encoder features; relative camera poses are handled by geometric attention constraints instead.
4. Experiments
Experiments evaluate CrossDepth on DDAD and nuScenes across in-domain and cross-domain settings, measuring overall and overlapping-region depth accuracy, cross-image consistency, and memory use. Results show stronger accuracy and consistency than state-of-the-art methods, while ablations support geometry-constrained attention and ray embeddings.
- Experimental setup: Experiments compare CrossDepth with five self-supervised methods on DDAD and nuScenes under in-domain and cross-domain evaluation.Results are reported for full images and overlapping regions.
- Experimental results: CrossDepth achieves higher overall accuracy, overlapping-region accuracy, and cross-view consistency than state-of-the-art approaches on most metrics in both evaluation settings.The improvements are particularly pronounced for cross-image consistency, including comparisons with CylinderDepth.
- Ablation studies: Constrained cross-image attention improves overall accuracy and especially cross-image consistency, while relaxing the geometric neighborhood constraint worsens consistency without improving accuracy.Unconstrained attention can aggregate features from unrelated image regions, including non-overlapping views.
- Ablation studies: Per-token ray embeddings further improve depth accuracy by modeling spatially varying relationships between monocular cues, camera intrinsics, and metric depth.They provide each token with its viewing direction instead of using only focal-length normalization and a global scale.
- Ablation studies: LoRA fine-tuning modestly improves over a frozen backbone, while the additional architecture retains a modest memory increase relative to its complexity.The small frozen-versus-adapted gap suggests pretrained DINO features already encode much of the visual information needed for monocular depth estimation.
5. Conclusion
CrossDepth provides a unified self-supervised framework for metric depth estimation from surround-view camera rigs, addressing cross-view inconsistency through geometry-constrained attention and camera-ray embeddings. Its scalability to substantially larger and more diverse datasets remains unevaluated due to computational constraints.
- The framework addresses visual-context inconsistency with geometry-constrained cross-image attention and camera-intrinsic variation with per-token camera-ray embeddings.These components target two identified sources of cross-view depth inconsistency.
- Geometrically constraining cross-image attention improves depth accuracy and cross-image consistency compared with unconstrained attention.
- Due to computational constraints, the method was not evaluated at substantially larger training scales.Its scalability to larger and more diverse datasets remains for future investigation.