Source-linked AI summary
Learning to Recover 3D Scene Shape from a Single Image
Wei Yin, Jianming Zhang, Oliver Wang, Simon Niklaus, Long Mai, Simon Chen, Chunhua Shen
TL;DR
The paper addresses why strong monocular depth models cannot directly recover accurate 3D scene shape when depth shift and camera focal length are unknown. It predicts affine-invariant depth, then uses point cloud encoders to recover shift and focal length, while adding geometry-aware depth losses; experiments report accurate reconstruction and state-of-the-art zero-shot transfer to 9 unseen datasets.
Problem
Unknown depth shift from scale-and-shift-invariant training losses, together with unavailable focal length, prevents accurate 3D scene-shape recovery from monocular depth predictions.
Method
A two-stage framework predicts depth up to scale and shift, then point cloud encoders estimate depth shift and focal-length adjustment; normalized regression and surface-normal losses improve mixed-data depth training.
Results
Accurate 3D shape is recovered from a single image, and the depth model achieves state-of-the-art zero-shot dataset transfer to 9 unseen datasets.
Takeaways & Limitations
The fully data-driven framework reconstructs 3D scene shape from a monocular image and uses 3D point cloud networks to improve depth-derived point-cloud structure.
Takeaways & Limitations
The method is less accurate for scenes with insufficient geometric cues, uncommon viewpoints, extreme focal lengths, or severe radial distortion, and assumes a pinhole camera model.
Abstract
from arXiv · showhide
Despite significant progress in monocular depth estimation in the wild, recent state-of-the-art methods cannot be used to recover accurate 3D scene shape due to an unknown depth shift induced by shift-invariant reconstruction losses used in mixed-data depth prediction training, and possible unknown camera focal length. We investigate this problem in detail, and propose a two-stage framework that first predicts depth up to an unknown scale and shift from a single monocular image, and then use 3D point cloud encoders to predict the missing depth shift and focal length that allow us to recover a realistic 3D scene shape. In addition, we propose an image-level normalized regression loss and a normal-based geometry loss to enhance depth prediction models trained on mixed datasets. We test our depth model on nine unseen datasets and achieve state-of-the-art performance on zero-shot dataset generalization. Code is available at: https://git.io/Depth
1. Introduction
The paper targets dense 3D scene-shape reconstruction from a single in-the-wild image, where unknown depth shifts and focal length can distort point clouds. It proposes a two-module framework that recovers these quantities and improves mixed-dataset depth prediction.
- Single-image scene reconstruction is pursued as a dense alternative to SLAM or SfM, which rely on multiple views or consecutive frames.
- Unknown depth shifts distort reconstructed scene shape, while unknown scale only uniformly rescales the 3D scene; inaccessible focal length adds further distortion.
- The framework predicts depth up to scale and shift, then uses point cloud encoders to estimate the missing shift and focal-length adjustment from an initial reconstruction.
- Point cloud processing enables learning 3D scene-shape priors from synthetic or laser-scanned data, with less domain-gap impact than image-based learning.
- The depth module adds image-level normalized regression and pairwise surface-normal losses for robust mixed-source training and improved depth geometry.
- State-of-the-art zero-shot transfer is reported across 9 unseen datasets, while point cloud reconstruction recovers accurate 3D shape from a single image.
2. Related Work
Prior work addresses monocular depth and single-image reconstruction using relative-depth objectives, object-specific representations, or hand-designed scene priors. This paper instead targets fully data-driven 3D scene-shape reconstruction while estimating focal length in 3D.
- Monocular depth estimation in the wild: In-the-wild monocular depth methods use ranking or scale-and-shift-invariant losses because diverse internet data lacks metric depth supervision.
- Monocular depth estimation in the wild: These objectives yield robust depth predictions, but unknown camera models and depth shifts prevent direct 3D-shape reconstruction from predicted depth maps.
- 3D reconstruction from a single image: Single-image object reconstruction methods recover shapes for categories such as humans, cars, planes, and tables using meshes or implicit functions.
- 3D reconstruction from a single image: Earlier single-image scene methods use plane segmentation, shading, or contour cues, but rely on hand-designed priors and restrictive geometry assumptions.
- Camera intrinsic parameter estimation: Traditional focal-length estimation uses calibration grids or vanishing points, while learned approaches regress focal length from images; this paper estimates it directly in 3D.
3. Method
The method combines a monocular depth predictor with point-cloud networks that recover unknown depth shift and focal-length scaling for 3D scene reconstruction. It also improves mixed-source depth training with normalized regression, pairwise normal, and multi-scale gradient losses.
- Pipeline: The two-stage pipeline trains a depth prediction module and point cloud module separately, then combines them during inference.The depth module predicts affine-invariant depth, while the point cloud module estimates corrections for reconstruction.
- Point Cloud Module: The point cloud module receives a distorted point cloud built from predicted depth and an initial focal length, then predicts depth-shift and focal-length adjustments.Separate point-cloud networks estimate the shift and focal-length scaling factor; inference uses an initial 60° field of view.
- Point Cloud Module: A predicted focal length changes x and y coordinates, while a depth shift changes x, y, and z non-uniformly, distorting reconstructed shape.These distortions motivate estimating both parameters from the reconstructed point cloud rather than relying on the depth map alone.
- Point Cloud Module: Training synthesizes perturbed point clouds by sampling depth shifts from U(−0.25, 0.8) and focal-length scales from U(0.6, 1.25).The focal-length range keeps the focal length positive and non-zero.
- Monocular Depth Prediction Module: The depth predictor is trained on mixed LiDAR, synthetic, calibrated stereo, and web-stereo data using image-level normalized regression and pairwise surface-normal losses.ILNR combines tanh normalization with a trimmed Z-score and pixel-wise MAE; trimming removes the nearest and farthest 10% of pixels.
- Monocular Depth Prediction Module: The pairwise normal loss samples points across edges and within planar regions to constrain local and global geometric relations.Surface normals are computed from locally fitted 3D point clouds after scale-and-shift alignment; the method also uses a multi-scale gradient loss.
4. Experiments
Experiments evaluate training, zero-shot transfer, shift and focal-length recovery, reconstructed 3D shape, and depth quality. The method generalizes across unseen datasets and improves geometric reconstruction, especially under a pinhole camera model.
- Datasets and implementation details: The depth prediction module is trained on RGBD and stereo-derived data from five sources, while the point cloud module uses ScanNet, Taskonomy, and synthetic depth maps.The depth training set includes Taskonomy, DIML, 3D Ken Burns, Holopix50K, and HRWSI.
- Evaluation: The evaluation uses nine datasets unseen during training, with depth accuracy, geometric quality, and shape reconstruction measured using dataset-specific metrics.The metrics include WHDR, AbsRel, δ1, depth boundary error, and plane-quality measures.
- Shift recovery: Across all shift-recovery test sets, recovering the depth shift lowers AbsRel compared with the baseline, whereas an image-based 2D shift predictor does not generalize well to unseen datasets.The shift is recovered from unprojected point clouds using the point cloud module.
- Focal length recovery: On unseen 2D-3D-S data, the method generalizes better than Hold-Geoffroy et al. for focal-length recovery and remains relatively insensitive to initial focal lengths.The initial FOV is varied from 20° to 70°; quality degrades slightly past 25° and 65°.
- 3D shape reconstruction: Using a pinhole camera model, the combined method significantly outperforms existing approaches for reconstructed 3D shape, and its point cloud module also transfers to MiDaS depth predictions.With orthographic projection, performance is roughly comparable to existing state-of-the-art methods.
- Qualitative and depth comparisons: Qualitative results show flatter planes, more realistic surface angles, less car distortion, and improved accuracy at edges, planar regions, far locations, and complex details.Pair-wise normal loss further improves reconstruction of edges and planes.
- Depth prediction: The depth method outperforms prior works on the reported comparisons, while a larger ResNeXt101 backbone further improves results.The evaluations use scale-and-shift alignment before comparing AbsRel.
5. Discussion
The method has identifiable geometric and camera-setting limitations. Recovery becomes less accurate when geometric cues are insufficient, viewpoints or focal lengths are extreme, or radial distortion is severe.
- Limitations: The point cloud module cannot accurately recover focal length or depth shift in scenes dominated by walls or sky, and accuracy decreases for uncommon view angles or extreme focal lengths.The authors suggest more diverse 3D training data may address these cases.
- Limitations: Because radial distortion is not modeled, severe radial distortion can distort the reconstructed scene shape.Recovering radial-distortion parameters is identified as future work.
6. Conclusion
The paper presents a fully data-driven monocular method that recovers scene shape by estimating depth shift and focal length with point cloud networks, with strong generalization to unseen data.
- Conclusion: The method uses point cloud networks trained on datasets with known global depth shifts and focal lengths to recover those quantities for 3D reconstruction.The approach showed strong generalization capabilities in extensive experiments.
- Conclusion: Extensive experiments demonstrated effective scene shape reconstruction and superior generalization to unseen data.The conclusion describes the method as the first fully data-driven approach of its kind, to the authors’ knowledge.
A.1. Datasets for Training
The training setup combines heterogeneous depth sources spanning sensor-captured, calibrated stereo, synthetic, and uncalibrated web-stereo data.
- Training data sources: Taskonomy provides LiDAR-captured RGBD data, with around 114K RGBD pairs sampled for training.
- Training data sources: DIML provides calibrated stereo images converted to disparity supervision, with around 121K RGBD pairs sampled.
- Training data sources: 3D Ken Burns contributes synthetic ground-truth depth, with around 51K RGBD pairs sampled for training.
- Training data sources: Holopix50K contributes diverse uncalibrated web stereo, using FlowNet to compute relative inverse-depth supervision.
- Training data sources: HRWSI contributes 20K RGBD images from diverse uncalibrated web stereo.
A.2. Datasets Used in Testing
The evaluation uses nine unseen datasets spanning indoor, outdoor, synthetic, stereo, RGB-D, LiDAR, and human-annotated in-the-wild imagery. Dataset-specific test subsets and metrics assess generalization across varied scene types and depth supervision.
- Datasets: The evaluation covers nine unseen datasets, including indoor, outdoor, synthetic, and in-the-wild scenes.The datasets are RedWeb, iBims-1, NYU, KITTI, ScanNet, DIODE, ETH3D, Sintel, OASIS, and YouTube3D.
- Datasets: Testing uses diverse depth sources, including optical-flow-derived disparity, LiDAR, Kinect, structure-from-motion, and human annotations.This mix spans relative, sparse metric, and high-quality depth supervision.
- Evaluation metrics: RedWeb evaluates WHDR error on 3.6K images using 5K randomly sampled point pairs per image, while iBims-1 evaluates edge and plane quality on 100 images.These tests target different aspects of depth quality rather than a single common metric.
B. Details for Depth Prediction Model and Training.
The depth prediction model is trained on mixed-source data to predict depth up to scale and shift, with losses adapted to each source’s calibration and noise. An auxiliary inverse-depth branch adds ranking and normalized regression supervision across the data.
- Source-specific losses: Web-stereo data receives only ranking loss because unknown inverse-depth scale and shift prevent affine-invariant depth and geometry losses.This training choice applies to Holopix50K and HRWSI.
- Source-specific losses: DIML uses normalized regression, multi-scale gradient, and ranking losses, while pairwise normal regression is restricted to planar regions because noisy edges create artifacts.The restriction reflects DIML’s noisier local disparity estimates.
- Source-specific losses: Taskonomy and synthetic 3D Ken Burns use pairwise normal regression, ranking, and multi-scale gradient losses because their edges and planes are accurately located.The loss combination exploits the higher-quality geometry available in these datasets.
- Auxiliary path: A lightweight auxiliary decoder path predicts inverse depth and applies ranking plus image-level normalized regression loss across all data.The main decoder outputs depth, while the auxiliary branch outputs inverse depth.
C. Sampling Strategy for Pairwise Normal Loss
Pairwise normal supervision samples points across meaningful edges and within planar regions, while balancing edge detections with negative pairs. Unseen-scene examples show realistic reconstructed shapes and reasonably accurate depth maps across diverse data.
- Edge sampling: Normal-loss sampling uses point pairs on both sides of detected image edges to supervise predicted and ground-truth normals.The edge-based pairs are denoted P and the corresponding normal sets are N and N* in the described formulation.
- Edge sampling: Edges are selected when paired normals differ by more than arccos(0.3), while negative samples use differences smaller than arccos(0.95).Negative samples are included to balance the sampled pairs.
- Plane sampling: For DIML, roads are treated as planar regions, whereas Taskonomy planes are found from regions sharing the same normal.The method samples 5000 paired points on each detected plane before combining the pair sets.
- Qualitative results: On unseen training examples, the approach recovers realistic 3D shapes across a wide range of in-the-wild scenes.The reconstructed point-cloud examples are presented as evidence of generalizability.
- Qualitative results: On unseen DIW and OASIS images, the method predicts reasonably accurate depth maps in global structure and local details.These examples are randomly sampled from the two datasets.