Source-linked AI summary
OmniDepth: Dense Depth Estimation for Indoors Spherical Panoramas
Nikolaos Zioulis, Antonis Karakottas, Dimitrios Zarpalas, Petros Daras
TL;DR
Existing depth-learning methods largely target pinhole images, while paired 360° color-depth data are difficult to acquire. OmniDepth renders existing synthetic and scanned 3D datasets into 360° training data and learns dense depth directly from equirectangular images, with promising results on synthesized and unseen realistic scenes. The authors also report that traditional monocular methods perform poorly on equirectangular inputs.
Problem
Learning-based depth estimation has focused on pinhole-camera images, while high-quality paired 360° color-depth datasets are difficult to acquire.
Method
The paper reuses synthetic and scanned 3D datasets, renders diverse 360° views with depth, and trains CNNs directly on equirectangular images.
Results
The models show promising results on synthesized data and unseen realistic images, while traditional monocular methods produce low-quality or inferior equirectangular predictions.
Takeaways & Limitations
Direct learning on 360° data supports dense depth estimation from omnidirectional images and provides a public dataset for further research.
Takeaways & Limitations
The datasets cover only indoor scenes and use aligned cameras, constant lighting, baked scan lighting, and no stitching artifacts, limiting real-world robustness.
Abstract
from arXiv · showhide
Recent work on depth estimation up to now has only focused on projective images ignoring 360 content which is now increasingly and more easily produced. We show that monocular depth estimation models trained on traditional images produce sub-optimal results on omnidirectional images, showcasing the need for training directly on 360 datasets, which however, are hard to acquire. In this work, we circumvent the challenges associated with acquiring high quality 360 datasets with ground truth depth annotations, by re-using recently released large scale 3D datasets and re-purposing them to 360 via rendering. This dataset, which is considerably larger than similar projective datasets, is publicly offered to the community to enable future research in this direction. We use this dataset to learn in an end-to-end fashion the task of depth estimation from 360 images. We show promising results in our synthesized data as well as in unseen realistic images.
1 Introduction
Depth estimation supports broad 3D-vision applications, but learning-based methods have largely targeted pinhole-camera imagery despite growing 360° content. OmniDepth addresses this gap by rendering existing 3D datasets into paired omnidirectional color and depth data and training directly on equirectangular images.
- Depth estimation supports 3D mapping, navigation, stereoscopic rendering, novel-view synthesis, compositing, and scene understanding.
- Learning-based depth methods have focused on traditional 2D imagery captured with pinhole-projection cameras.
- Growing adoption of spherical cameras has expanded 360° content across entertainment, robotics, vehicles, events, journalism, and VR.
- Acquiring high-quality paired 360° color and depth data is difficult because spherical sensors and depth scanners introduce resolution, visibility, baseline, and artifact problems.
- The paper trains a CNN to estimate depth from an equirectangular image by rendering diverse 360° views from existing 3D datasets.
- The contributions include a direct dense 360° depth method, a paired 360° color-depth dataset, and a CNN architecture designed for equirectangular images.
2 Related Work
Prior work estimates spherical geometry analytically, with specialized learning approaches emerging alongside extensive monocular depth-estimation research for conventional images. Existing spherical methods address layouts, distances, distortion, or viewpoint variation, but the reviewed work motivates direct dense learning on 360° inputs.
- Spherical stereo reformulates multi-view geometry by accounting for spherical projection and representing disparity as angular displacement.
- Single-image panoramic methods estimate room layouts or CAD geometry, but their outputs are coarse and cannot recover finer scene details.
- Early machine-learning work predicted one range value per image column for robotic navigation rather than dense per-pixel depth.
- CNNs for spherical input commonly operate directly on equirectangular images or on cubemap faces, while newer methods model distortion with graphs or transferred filters.
- Monocular depth estimation research includes supervised CNNs, CRF refinement, classification formulations, and unsupervised view-synthesis approaches.
3 Synthesizing Data
OmniDepth constructs a large indoor 360° dataset by rendering views from complementary synthetic and scanned 3D datasets. The pipeline preserves accurate depth where available while masking missing or unreliable regions in rendered outputs.
- Depth sensors and laser scanners are difficult to use for spherical datasets because sensors can appear in one another’s views and 360° cameras vary in resolution.
- The dataset combines synthetic CAD scenes with perfect per-pixel depth and realistic scanned scenes containing holes, coarse geometry, or baked lighting.
- Camera placement uses central poses for CG houses and scanning-derived poses for scanned buildings, producing multiple 360° samples per building.
- Rotating each pose by 90° yields four viewpoints, and the pipeline produces 94,098 renders from 23,524 unique viewpoints.
- The generated dataset mixes synthetic and realistic indoor 360° color-depth data across houses, offices, educational spaces, and varied room layouts, and is publicly available.
4 Omnidirectional Depth Estimation
The paper trains fully convolutional encoder-decoder networks for supervised dense depth prediction from equirectangular color images. RectNet adapts its architecture to spherical distortion and global context, while training masks unreliable rendered regions.
- Both networks take only an equirectangular 360° color image and predict an equirectangular depth map using fully convolutional encoder-decoder architectures.
- UResNet: UResNet uses an asymmetric encoder-decoder with skip connections to support deeper architectures and larger receptive fields.
- RectNet: RectNet addresses latitude-dependent equirectangular distortion with rectangle and square filters while using progressively dilated convolutions for broader global context.
- Training Loss: Training is completely supervised because the synthesized dataset provides ground-truth depth annotations.
- Training Loss: The loss combines multi-scale depth regression and smoothness terms, with scale-specific weights α_s and β_s.
- Training Loss: Missing rendered regions are excluded during backpropagation, allowing training with incomplete or slightly inaccurate 3D models.
5 Results
The experiments evaluate two 360° depth networks on synthetic, realistic, and unseen data, comparing them with monocular baselines and cubemap-based processing. RectNet generally outperforms UResNet and traditional monocular approaches, while unseen Sun360 results show plausible qualitative predictions.
- Model Performance: RectNet outperforms the standard UResNet despite using approximately 8.8M rather than 51.2M parameters.The comparison is reported on the authors’ test set and unseen synthetic SceneNet data.
- Model Performance: RectNet continues to outperform UResNet in leave-one-out training, although both models perform slightly worse with less training data.The leave-one-out setting trains on SunCG before fine-tuning on Matterport3D and evaluates on Stanford2D3D and SceneNet.
- Model Performance: RectNet’s higher performance is attributed to its larger receptive field, measuring 266 × 276 versus UResNet’s 190 × 190.RectNet also reduces spatial dimensions by only a factor of four, preserving denser feature information.
- Comparison against other methods: Cubemap processing improves monocular performance over direct equirectangular inference but remains quantitatively inferior to the proposed models.It also requires multiple face-wise inferences and produces inconsistent depth scales between cube faces.
- Qualitative Results: On unseen realistic Sun360 scenes without ground-truth depth, both proposed models estimate plausible scene depth, while UResNet produces smoother predictions than RectNet.The evaluation uses the “Room” and “Indoors” splits.
6 Conclusions
The paper presents a supervised framework for estimating depth from single 360° images using a rendered dataset built from existing indoor 3D scenes. Its scope is limited by indoor-only data and synthetic acquisition conditions, motivating future work on unsupervised learning and realism.
- Conclusions: The framework estimates scene depth from a single 360° image using supervised learning with ground-truth depth.The approach addresses dense depth estimation directly from omnidirectional input.
- Conclusions: The dataset is synthesized by rendering 360° views from synthetic and real-world scanned indoor 3D scenes.This reuses existing 3D datasets to overcome the scarcity of paired 360° color and depth data.
- Conclusions: The datasets cover only indoor cases, limiting applicability to outdoor settings.The limitation is specific to the environments represented in the training data.
- Conclusions: Perfect camera vertical alignment, constant lighting, absent stitching artifacts, and baked-in scan lighting may hamper robustness in real-world conditions.Real scenes can contain substantially higher luminosity dynamic range than the scanned data.
- Conclusions: Future work proposes unsupervised view-synthesis supervision and improved real-world robustness through realistic content generation or discrimination.The paper specifically mentions GAN-based generation and discriminators for plausible or real images.