Source-linked AI summary
Depth Any Panoramas: A Foundation Model for Panoramic Depth Estimation
Xin Lin, Meixi Song, Dizhe Zhang, Wenxuan Lu, Haodong Li, Bo Du, Ming-Hsuan Yang, Truong Nguyen, Lu Qi
TL;DR
Panoramic depth estimation needs metric predictions that generalize across scene distances and diverse domains, despite limited and costly training data. DAP combines a large cross-domain data engine, three-stage pseudo-label curation, and geometry-aware modeling, achieving strong zero-shot generalization and state-of-the-art indoor–outdoor performance with robust metric predictions.
Problem
Panoramic depth methods struggle to generalize across diverse real-world scenes, particularly outdoors, partly because existing data lack sufficient scale and diversity.
Method
DAP combines large-scale synthetic and real panorama data, three-stage pseudo-label curation, DINOv3-Large, a range mask head, and geometry- and sharpness-aware optimization.
Results
DAP achieves state-of-the-art performance and strong zero-shot generalization across indoor–outdoor benchmarks, with robust and stable metric predictions in diverse real-world outdoor environments.
Takeaways & Limitations
DAP unifies indoor and outdoor panoramic metric-depth estimation within a single foundation model without fine-tuning.
Abstract
from arXiv · showhide
In this work, we present a panoramic metric depth foundation model that generalizes across diverse scene distances. We explore a data-in-the-loop paradigm from the view of both data construction and framework design. We collect a large-scale dataset by combining public datasets, high-quality synthetic data from our UE5 simulator and text-to-image models, and real panoramic images from the web. To reduce domain gaps between indoor/outdoor and synthetic/real data, we introduce a three-stage pseudo-label curation pipeline to generate reliable ground truth for unlabeled images. For the model, we adopt DINOv3-Large as the backbone for its strong pre-trained generalization, and introduce a plug-and-play range mask head, sharpness-centric optimization, and geometry-centric optimization to improve robustness to varying distances and enforce geometric consistency across views. Experiments on multiple benchmarks (e.g., Stanford2D3D, Matterport3D, and Deep360) demonstrate strong performance and zero-shot generalization, with particularly robust and stable metric predictions in diverse real-world scenes. The project page can be found at: \href{https://insta360-research-team.github.io/DAP_website/} {https://insta360-research-team.github.io/DAP\_website/}
1. Introduction
Panoramic depth estimation seeks full-surrounding metric understanding but remains difficult to generalize across diverse real-world scenes. DAP addresses data scale, pseudo-label quality, and geometry-consistent modeling through a unified foundation-model design.
- 1. Introduction: Limited data scale and diversity constrain panoramic depth methods, which struggle to generalize across diverse real-world scenes, particularly outdoors.The passage attributes this limitation partly to the high cost of data collection and annotation.
- 1. Introduction: DAP explores a data-in-the-loop paradigm centered on scalable datasets with reliable ground truth and models adapted to that scaling.The design targets a geometry-consistent and generalizable panoramic foundation model.
- 1. Introduction: The data engine combines Structured3D, 90K UE5-based outdoor samples, about 1.7M web images, and 200K DiT360 indoor panoramas.A pseudo-label curation pipeline is used to reduce indoor–outdoor and synthetic–real domain gaps.
- 1. Introduction: DAP uses metric-depth estimation with DINOv3-Large, a conventional dense depth head, a plug-and-play range mask head, and complementary geometric and sharpness losses.The range mask filters regions using distance thresholds, while the losses target distance robustness and cross-view geometric consistency.
- 1. Introduction: Experiments on Stanford2D3D, Matterport3D, and Deep360 show strong generalization, visual consistency, and scale-aware predictions in challenging real-world scenes.The reported scenarios include complex geometry, distant regions, and sky areas across synthetic and real environments.
2. Related work
Panoramic depth research has progressed from in-domain distortion handling toward zero-shot and universal metric estimation. DAP is positioned against this transition with a substantially larger, cross-domain training foundation.
- 2. Related work: DAP scales panoramic training to about 2M samples spanning indoor/outdoor and synthetic/real-world domains, exceeding the scale and coverage of prior panoramic datasets.The data composition comparison identifies this broader foundation as a distinguishing characteristic.
- 2. Related work: Perspective depth estimation has advanced through metric, scale-invariant, and pretrained-model approaches that achieve strong performance and zero-shot generalization.Examples include UniDepth, Metric3D, DepthPro, MoGe, DepthAnything, and generative-model fine-tuning methods.
- 2. Related work: In-domain panoramic methods address equirectangular distortion through distortion-aware designs or projection-driven strategies, but often have limited cross-domain generalization.The related-work passage links this limitation to over-reliance on in-domain training.
- 2. Related work: Zero-shot panoramic methods use pretrained perspective models, pseudo-label distillation, or universal metric-capable camera modeling for broader cross-domain applicability.Depth Anywhere and PanDA exemplify pseudo-label transfer, while DAC and UniK3D model diverse imaging geometries.
3. Method
DAP combines a large, diverse panorama data engine with progressive pseudo-label curation and a geometry-consistent depth network. Its design targets scalable metric estimation across indoor/outdoor, synthetic/real, and varying-distance scenes.
- 3.1. Data Engine: DAP unifies synthetic and real panoramic data, labeled and unlabeled sources, into a scalable foundation-model training framework.The data engine is designed to span diverse domains and scene types.
- 3.2. Three-Stage Pipeline: The three-stage pipeline trains scene-invariant and realism-invariant labelers before final DAP training on labeled and pseudo-labeled data.A PatchGAN-based discriminator selects 300K indoor and 300K outdoor high-confidence pseudo-labeled samples in Stage 2.
- 3.3. Model Design: DAP uses DINOv3-Large with distortion-aware decoding, a plug-and-play range mask head, and a metric depth head for adaptive distance control.The range-mask design supports thresholds of 10 m, 20 m, 50 m, and 100 m.
- 3.3. Model Design: Sharpness-centric optimization combines perspective-patch dense fidelity with ERP gradient refinement to preserve fine details and object boundaries.The dense fidelity loss averages Gram-based supervision across N = 12 perspective views, while the gradient loss focuses on high-frequency edge regions.
- 3.3. Model Design: Qualitative comparisons report sharper boundaries, smoother global geometry, and stronger robustness in distant and sky regions than DAC and Unik3D.These comparisons cover diverse real-world indoor and outdoor scenes.
- 3.3. Model Design: Geometry-centric optimization applies normal and point-cloud losses, while distortion-aware weighting balances gradient contributions across the equirectangular sphere.Depth maps are converted into surface normals and spherical-coordinate point clouds for geometric supervision.
4. Experiment
DAP is evaluated on zero-shot indoor and outdoor benchmarks, a proposed outdoor benchmark, qualitative comparisons, and component ablations. Across these evaluations, it shows strong generalization, metric consistency, and improved visual structure, with the range mask head providing useful scale adaptation.
- Experiment Setup: The evaluation uses diverse labeled and unlabeled indoor and outdoor data, including 1.9M unlabeled real images for pseudo-label learning.Test sets include Stanford2D3D, Matterport3D, Deep360, and the proposed DAP-Test benchmark.
- Zero-shot Performance: DAP consistently achieves the best performance across Stanford2D3D, Matterport3D, and Deep360 without fine-tuning.It lowers AbsRel and raises δ1 on the indoor benchmarks, while achieving the lowest AbsRel and RMSE and highest δ1 on Deep360.
- DAP-Test Performance: DAP outperforms DAC and Unik3D across all metrics on DAP-Test, reducing AbsRel from 0.2517 to 0.0781 and RMSE from 10.563 to 6.804.δ1 increases from 0.6086 to 0.9307 on the proposed outdoor benchmark.
- Qualitative Comparison: Qualitative comparisons show sharper boundaries, more coherent global geometry, and stronger preservation of distant structures and fine details than existing methods.The Stanford2D3D comparison emphasizes scale-awareness and retained geometric detail in distant wall and ceiling regions.
5. Conclusion
DAP is presented as a panoramic metric-depth foundation model built through large-scale data scaling and a unified training pipeline. It achieves strong zero-shot generalization and state-of-the-art performance across indoor–outdoor benchmarks, with robust metric predictions in diverse outdoor scenes.
- 5. Conclusion: DAP combines reliable pseudo-labeling, geometry-aware design, and a plug-and-play range mask head to achieve strong zero-shot generalization across indoor–outdoor benchmarks.The model is described as particularly robust and stable in diverse real-world outdoor environments.