Source-linked AI summary
DA$^{2}$: Depth Anything in Any Direction
Haodong Li, Wangguangdong Zheng, Jing He, Yuhao Liu, Xin Lin, Xin Yang, Ying-Cong Chen, Chunchao Guo
TL;DR
Panoramic depth estimation is limited by scarce data and spherical distortions that constrain generalization and efficiency. DA2 addresses these issues with curated panoramic training data and SphereViT, achieving leading zero-shot performance while remaining end-to-end; its limitations include missed fine details and boundary seams at lower resolution.
Problem
Panoramic depth estimation has limited training data and spherical distortions that restrict zero-shot generalization and encourage inefficient perspective-splitting approaches.
Method
DA2 combines a data curation engine that generates panoramic data from perspective data with SphereViT, which explicitly uses spherical coordinates.
Results
DA2 achieves SoTA zero-shot performance, with an average 38% AbsRel improvement over the strongest zero-shot baseline, and surpasses prior in-domain methods.
Takeaways & Limitations
Scaling panoramic data and explicitly modeling spherical geometry yields high-quality, robust 360°×180° geometric estimation in an efficient end-to-end system.
Takeaways & Limitations
At 1024×512 resolution with partially available spherical ground truth, DA2 may miss fine details and produce seams at panorama boundaries.
Abstract
from arXiv · showhide
Panorama has a full FoV (360$^\circ\times$180$^\circ$), offering a more complete visual description than perspective images. Thanks to this characteristic, panoramic depth estimation is gaining increasing traction in 3D vision. However, due to the scarcity of panoramic data, previous methods are often restricted to in-domain settings, leading to poor zero-shot generalization. Furthermore, due to the spherical distortions inherent in panoramas, many approaches rely on perspective splitting (e.g., cubemaps), which leads to suboptimal efficiency. To address these challenges, we propose $\textbf{DA}$$^{\textbf{2}}$: $\textbf{D}$epth $\textbf{A}$nything in $\textbf{A}$ny $\textbf{D}$irection, an accurate, zero-shot generalizable, and fully end-to-end panoramic depth estimator. Specifically, for scaling up panoramic data, we introduce a data curation engine for generating high-quality panoramic depth data from perspective, and create $\sim$543K panoramic RGB-depth pairs, bringing the total to $\sim$607K. To further mitigate the spherical distortions, we present SphereViT, which explicitly leverages spherical coordinates to enforce the spherical geometric consistency in panoramic image features, yielding improved performance. A comprehensive benchmark on multiple datasets clearly demonstrates DA$^{2}$'s SoTA performance, with an average 38% improvement on AbsRel over the strongest zero-shot baseline. Surprisingly, DA$^{2}$ even outperforms prior in-domain methods, highlighting its superior zero-shot generalization. Moreover, as an end-to-end solution, DA$^{2}$ exhibits much higher efficiency over fusion-based approaches. Both the code and the curated panoramic data has be released. Project page: https://depth-any-in-any-dir.github.io/.
DA2: DEPTH ANYTHING IN ANY DIRECTION
DA2 is presented as a panoramic depth-estimation system whose teaser emphasizes dense distance prediction and reconstructed 3D structure quality.
- The paper lists Haodong Li and collaborators as its authors.The author list identifies corresponding authors with the ‡ symbol.
- DA2 predicts dense distance from a single 360° panorama.The teaser highlights geometric fidelity in the predicted depth.
- Its reconstructed 3D structures show sharp geometric details across diverse scenes.
1 INTRODUCTION
The introduction motivates panoramic depth estimation through panoramas’ broad visual coverage, limited panoramic data, and spherical distortion, then presents DA2’s data and architecture to address these constraints.
- Panoramas provide 360°×180° coverage useful for spatially comprehensive 3D applications.The introduction connects panoramic depth to 3D reconstruction, 3D scene generation, and physical simulation.
- Limited panoramic data has restricted earlier methods largely to in-domain evaluation and weakened zero-shot generalization.
- DA2 converts perspective RGB–depth data into panoramic training data through a proposed data curation engine.The engine is motivated by the larger supply of high-quality perspective depth data.
- SphereViT uses spherical coordinates to produce distortion-aware image representations for panoramas.Its attention mechanism uses spherical-angle information to encode panoramic geometry.
- DA2’s benchmark reports a 38% average AbsRel improvement over the strongest zero-shot baseline.The introduction also states that DA2 achieves SoTA performance and compares zero-shot panoramic and perspective estimators.
- The contributions include panoramic data curation, SphereViT, a comprehensive benchmark, and SoTA performance.The benchmark covers zero-shot and in-domain methods across panoramic and perspective settings.
2 RELATED WORKS
Related work spans rapidly improving perspective depth estimation, in-domain panoramic methods that address spherical distortion, and newer zero-shot panoramic estimators.
- Perspective depth estimation has advanced through large-scale training across metric, scale-invariant, and relative-depth models.Examples include UniDepth, Metric3D, DepthPro, MoGe, and DepthAnything.
- 2 RELATED WORKS: In-domain panoramic methods evolved from CNNs to ViTs and commonly mitigate distortion by combining ERP and cubemap features.
- 2 RELATED WORKS: Zero-shot panoramic approaches include tangent-perspective blending, Möbius-transformation augmentation, and separate camera-ray prediction.The related work notes that 360MonoDepth suffers from multi-view inconsistencies.
3 METHODOLOGY
DA2 combines a perspective-to-panorama data curation engine with SphereViT, which injects spherical awareness into image features for end-to-end panoramic depth estimation.
- 3.1 PANORAMIC DATA CURATION ENGINE: Perspective-to-panorama curation projects perspective pixels into spherical coordinates, maps them to ERP positions, and uses panoramic out-painting to complete uncovered regions.Limited perspective FoVs cover only part of the sphere, motivating the generation of full panoramas from partial ones.
- 3.1 PANORAMIC DATA CURATION ENGINE: ∼543K curated panoramic samples expand the training set from ∼63K to ∼607K, addressing data scarcity linked to poor generalization.The engine transforms perspective data into panoramic samples and scales the dataset by approximately ten times.
- 3.2 SPHEREVIT & TRAINING LOSSES: SphereViT computes spherical coordinates for ERP pixels and encodes them into spherical embeddings that inject explicit spherical awareness into ViT features.The angle field is resized, flattened, and expanded with sine-cosine embeddings to match the image-feature dimension.
- 3.2 SPHEREVIT & TRAINING LOSSES: SphereViT replaces ordinary self-attention with cross-attention in which image features query fixed spherical embeddings as keys and values.This design produces distortion-aware representations and is reported to improve geometrical fidelity.
- 3.2 SPHEREVIT & TRAINING LOSSES: The model is trained end-to-end with distance and surface-normal losses, combining globally accurate distances with locally smooth and sharp geometry.Predicted distances are median-aligned for scale-invariant training, and the total loss is a weighted sum of the two L1 terms.
4 EXPERIMENTS
DA2 is evaluated on multiple panoramic benchmarks and consistently outperforms prior methods, including in zero-shot settings. Ablations further show benefits from scaled panoramic data, spherical embedding, and normal supervision.
- 4.2 QUANTITATIVE & QUALITATIVE COMPARISONS: 38% lower AbsRel and 22% lower RMSE than the second-best zero-shot method, with average δ1 of 95.73% and δ2 of 98.51%.DA2 also surpasses earlier in-domain methods despite being evaluated zero-shot.
- 4.2 QUANTITATIVE & QUALITATIVE COMPARISONS: DA2 produces more accurate geometry than UniK3D and MoGev2 while achieving higher inference efficiency than fusion-based approaches.Its training uses about 21× more panoramic data than UniK3D; MoGev2 is limited by multi-view inconsistencies during fusion.
- 4.3 ABLATION STUDIES: Training-data scaling steadily improves DA2’s performance, with gains becoming more gradual as the curated data volume approaches convergence.The scaling study reports average results across multiple datasets and uses perspective data converted into panoramas.
- 4.3 ABLATION STUDIES: Spherical embedding improves panoramic geometry and mitigates distortions such as curved walls.Removing Esphere produces less accurate geometric understanding and curved walls.
- 4.3 ABLATION STUDIES: Auxiliary normal supervision improves performance and yields flatter, smoother, more coherent geometry with fewer artifacts.The effect is most relevant in ambiguous regions such as corners, edges, and the upper or lower poles.
5 LIMITATION & CONCLUSION
DA2 achieves strong zero-shot panoramic distance estimation through large-scale data and distortion-aware modeling, while remaining limited by resolution and incomplete spherical ground truth. The paper concludes that it outperforms prior methods and supports high-fidelity 3D applications.
- Limitation: Lower 1024×512 training resolution and partially available spherical ground truth can cause missed fine details and visible seams at panorama boundaries.The examples include an incorrectly aligned lamp distance and seams along the left–right boundaries.
- Conclusion: DA2 delivers state-of-the-art zero-shot performance while outperforming prior zero-shot and in-domain methods as an efficient end-to-end estimator.The model is trained on over 600K samples combining curated perspective data and native panoramas.
7 LLMS IN PAPER WRITING
The paper states that large language models were used only to polish grammar and sentence structure. The scientific and technical content was written entirely by humans.
- 7 LLMS IN PAPER WRITING: LLMs were used solely for grammar, sentence-structure, readability, clarity, and fluency improvements, not for original research content.The paper attributes all scientific and technical writing to human authors.
SUPPLEMENTARY MATERIALS OF DA2: DEPTH ANYTHING IN ANY DIRECTION
DA2 supports multi-panorama 3D reconstruction and related applications by producing geometrically consistent point clouds. Demonstrations include aligned indoor layouts, layered home renovation, and robot-manipulation simulation.
- Applications: The applications leverage DA2’s zero-shot generalizable panoramic depth estimation for 3D reconstruction-related tasks.The supplementary demonstrations connect the estimator to indoor-layout reconstruction and simulation scenarios.
- Pano3R: Panoramic Multi-view Reconstruction: DA2 reconstructs globally aligned 3D point clouds from panoramas of different rooms, with rooms alignable by translation without scaling.The reconstruction preserves spatial coherence across the indoor layout.
- Layered Home Renovation: Across empty, simple, and full foreground-complexity levels, reconstructed point clouds remain consistently aligned and preserve sharp text and wall boundaries.The fused point clouds are described as free of distortions in the zoom-in regions.
- Robotics Simulation: DA2’s reconstructed point clouds can provide a practical 3D simulation platform for evaluating robot manipulation performance.The application is presented as an example of 3D reconstruction enabled by panoramic distance estimation.
B PANORAMIC DATA CURATION ENGINE (MORE DETAILS)
The panoramic data curation engine transforms perspective datasets into panoramic data while deriving vertical field of view from horizontal field of view.
- 6 perspective datasets are transformed into panoramic data by the proposed curation engine.
- Sampling probabilities are normalized across datasets primarily according to dataset size.
- YFoV is derived from XFoV using the panorama dimensions.
C EVALUATION METRICS
The evaluation aligns predicted distance with ground truth before computing standard depth metrics over valid pixels.
- Median alignment is applied to predicted distance before metric computation.
- AbsRel and RMSE are computed following prior evaluation protocols.
- The metrics operate over the set of valid pixels.
- δ1 and δ2 measure the proportions of pixels satisfying thresholds 1.25 and 1.25^2, respectively.
D DIFFERENCE AMONG: METRIC & SCALE-INVARIANT (BIASED) & AFFINE-INVARIANT (RELATIVE)
The paper distinguishes metric, scale-invariant, and affine-invariant depth by the geometric information they preserve and the alignment used during evaluation.
- Metric depth represents absolute physical distance, while scale-invariant depth permits a global bias or shift and uses median alignment.
- DA2 focuses on scale-invariant distance because it preserves 3D geometry while avoiding optimization challenges from absolute scale differences across domains.
- Affine-invariant depth preserves only depth ordering and uses scale-and-shift-invariant alignment.
- Metric and scale-invariant depth preserve full 3D geometry, whereas affine-invariant depth cannot reconstruct accurate 3D structure.
- The benchmark compares DA2 with in-domain, fusion-based zero-shot, and end-to-end zero-shot baselines under differing depth definitions.