Source-linked AI summary
CylinderDepth: Cylindrical Spatial Attention for Multi-View Consistent Self-Supervised Surround Depth Estimation
Samer Abualhanud, Christian Grannemann, Max Mehltretter
TL;DR
Self-supervised surround-view depth estimation must produce consistent metric depth from minimally overlapping camera images without relying on dense labels. CylinderDepth uses shared cylindrical geometry and non-learned distance-based cross-view attention to address this challenge, improving multi-view consistency and depth accuracy on DDAD and nuScenes while remaining limited by coarse-scale attention and time-synchronization assumptions.
Problem
Existing surround-view depth methods can produce inconsistent 3D coordinates for the same object point across overlapping images because camera geometry is not explicitly considered at inference.
Method
The method projects features from calibrated, time-synchronized cameras onto a shared cylinder and applies non-learned attention weighted by cylindrical distances to aggregate nearby cross-view features.
Results
CylinderDepth improves multi-view depth consistency over state-of-the-art methods and achieves slightly higher depth accuracy on overlapping regions and full images across DDAD and nuScenes.
Takeaways & Limitations
Shared cylindrical geometry enables cross-image feature sharing that improves multi-view consistency and overall depth accuracy within the evaluated surround-camera setting.
Takeaways & Limitations
Attention is applied only at the lowest feature resolution because of its computational cost, while time differences between cameras can degrade results in dynamic scenes and under rig motion.
Abstract
from arXiv · showhide
Self-supervised surround-view depth estimation enables dense, low-cost 3D perception with a 360° field of view from multiple minimally overlapping images. Yet, most existing methods suffer from depth estimates that are inconsistent across overlapping images. To address this limitation, we propose a novel geometry-guided method for calibrated, time-synchronized multi-camera rigs that predicts dense metric depth. Our approach targets two main sources of inconsistency: the limited receptive field in border regions of single-image depth estimation, and the difficulty of correspondence matching. We mitigate these two issues by extending the receptive field across views and restricting cross-view attention to a small neighborhood. To this end, we establish the neighborhood relationships between images by mapping the image-specific feature positions onto a shared cylinder. Based on the cylindrical positions, we apply an explicit spatial attention mechanism, with non-learned weighting, that aggregates features across images according to their distances on the cylinder. The modulated features are then decoded into a depth map for each view. Evaluated on the DDAD and nuScenes datasets, our method improves both cross-view depth consistency and overall depth accuracy compared with state-of-the-art approaches. Code is available at https://abualhanud.github.io/CylinderDepthPage.
1. Introduction
Self-supervised surround-view depth estimation avoids sparse sensor labels but remains vulnerable to inconsistent reconstructions across minimally overlapping cameras. CylinderDepth addresses this by using shared cylindrical geometry and distance-based cross-view attention to expand context and align correspondences.
- Self-supervised depth estimation uses photometric consistency between target images and source-image renderings instead of ground-truth depth labels.
- Surround cameras provide 360° coverage and metric-scale depth when their relative orientations and baselines are known, but typically have minimal spatial overlap.
- Minimal overlap and image-specific processing can cause the same 3D object point to receive different reconstructed coordinates across views.
- Prior methods enforce multi-view consistency implicitly through motion constraints, consistency losses, or learned attention without guaranteeing geometric consistency at inference.
- CylinderDepth maps reconstructed points from all images onto a shared unit cylinder, aligning overlapping regions through consistent cylindrical neighborhood relations.
- The method applies non-learned spatial attention weighted by geodesic distances on the cylinder, expanding border-region context while restricting correspondence matching to nearby features.
2. Related Work
Prior depth-estimation research includes monocular, omnidirectional, multi-view stereo, surround-camera, and attention-based approaches. CylinderDepth differs by using cross-view attention with explicit geometric relationships rather than relying only on learned feature aggregation.
- Monocular Depth Estimation: Monocular depth estimation predicts dense per-pixel depth from one RGB image, while supervised methods rely on often-sparse depth-sensor labels.
- Monocular Depth Estimation: Omnidirectional images capture broader scenes, but setups without camera baselines do not support scale-aware self-supervised depth estimation.
- Multi-View Depth Estimation: Learning-based multi-view stereo methods either identify correspondences and triangulate 3D points or directly regress point maps, often with image orientations.
- Multi-View Depth Estimation: Surround-camera depth estimation combines multiple central-projection cameras for 360° coverage, with minimal overlap requiring monoscopic depth estimation for most pixels.
- Attention-Based Depth Estimation: Attention mechanisms have been used to extend receptive fields and promote consistency in monocular and multi-view depth estimation.
- Attention-Based Depth Estimation: Unlike prior spatial or temporal attention approaches, CylinderDepth fuses features across images using non-learned spatial weighting derived from geometric relations between views.
3. Methodology
The method builds a shared cylindrical representation of multi-camera features and uses geometry-guided spatial attention to improve cross-view information exchange and depth consistency. It combines this mechanism with spatial, temporal, and spatio-temporal photometric supervision to predict depth for every view.
- 3. Methodology: A shared encoder first produces multi-scale features and a preliminary depth map for each synchronized camera image.The encoded features are reused in a second forward pass for cylindrical projection and feature aggregation.
- 3.1. Multi-View Consistency: The attention-modulated features are decoded with skip-connected features to produce a per-pixel depth map for every image.Attention is applied only at the lowest feature scale for efficiency while skip connections preserve high-frequency information.
- 3.1. Multi-View Consistency: Feature positions are back-projected to 3D using camera intrinsics, rig poses, and preliminary depth, then projected onto a common unit cylinder.The resulting cylindrical position map stores each feature pixel by azimuth angle and height.
- 3.1. Multi-View Consistency: Cylindrical coordinates place corresponding pixels from different images nearby even when preliminary depth is inaccurate, enabling geometry-aware cross-view feature exchange.This representation is used because direct 3D distances can separate corresponding pixels when initial depth estimates are wrong.
- 3.1. Multi-View Consistency: Non-learned spatial attention weights features by geodesic distance on the cylinder using a truncated 2D Gaussian neighborhood.The covariance matrix controls kernel shape and size, while truncation excludes distant, irrelevant pixels and accommodates minor projection errors.
- 3.2. Self-Supervision: Training combines spatial, temporal, and spatio-temporal photometric consistency with auxiliary smoothing and depth-consistency losses.Spatial supervision uses metric relative poses and overlap, while temporal supervision addresses limited same-frame overlap; the overall objective is a weighted sum.
4. Experiments
Experiments on DDAD and nuScenes evaluate the method’s depth accuracy, multi-view consistency, efficiency, and ablations. The results show stronger consistency and slightly higher accuracy, while attention placement and explicit feature sharing are important design factors.
- Experimental Setup: DDAD and nuScenes provide six-camera, 360° surround imagery with LiDAR-derived reference depth for evaluation.Images are resized to 384×640 on DDAD and 352×640 on nuScenes; evaluation ranges are 200 m and 80 m, respectively.
- Experimental Setup: The evaluation uses Abs Rel, Sq Rel, RMSE, threshold accuracy, and Depth Cons, which measures Euclidean-distance disagreement for corresponding pixels in overlaps.Depth Cons converts each overlapping pixel’s depth into a Euclidean distance from a common reference coordinate system before computing RMSE between correspondences.
- Experimental Results: Compared with FSM, SurroundDepth, VFDepth, and CVCDepth, the method substantially improves multi-view depth consistency across the evaluated comparisons.The comparison includes both 2D-based and 3D-based depth estimation methods.
- Experimental Results: The method achieves slightly higher depth accuracy in overlapping regions and full-image evaluations on both DDAD and nuScenes.Qualitative DDAD comparisons show better preserved details and more well-defined object boundaries.
- Experimental Results: Deviations from the time-synchronization assumption degrade result quality in dynamic scenes and under rig motion, affecting methods that rely on shared camera motion and spatial supervision.The nuScenes cameras can have time differences of up to 40 ms relative to one another.
- Ablation Studies: Identity-attention ablations show that cross-image feature sharing, particularly during inference, benefits multi-view consistency.The study isolates the contribution of the proposed spatial attention mechanism.
- Ablation Studies: Applying attention only at the coarsest scale preserves fine-scale structures better than applying it at all scales, whose predictions become over-smoothed without significant consistency gains.The low-resolution attention acts as a global consistency mechanism while higher-resolution features retain fine-scale detail.
- Ablation Studies: With both ResNet-18 and MambaVision-T, depth consistency improves only when features are explicitly shared through the proposed attention mechanism.This comparison indicates that the observed consistency issue is not primarily tied to the encoder architecture or capacity.
5. Conclusion
The method enforces multi-view consistency by projecting pixels into a shared cylindrical representation and applying distance-based attention. This enables cross-image feature sharing and improves consistency and depth accuracy, but coarse-resolution attention limits fine-grained consistency.
- The method projects pixels from all input images into a shared cylindrical representation for cross-view processing.Attention is applied according to distances on the cylinder.
- Distance-based attention enables effective cross-image feature sharing.
- The approach improves multi-view consistency and overall depth accuracy.
- Attention is applied only at the lowest feature resolution because of its high computational cost.This enforces global consistency but aggregates large regions and restricts fine-grained detail.
- The authors propose adapting distance computations to improve pixel-level consistency and modeling rig trajectories continuously for asynchronously captured images.