Source-linked AI summary
P3Depth: Monocular Depth Estimation with a Piecewise Planarity Prior
Vaishakh Patil, Christos Sakaridis, Alexander Liniger, Luc Van Gool
TL;DR
Supervised monocular depth estimation must infer dense depth from a single image despite scale ambiguity, while standard pixel-wise supervision overlooks piecewise-smooth scene structure. P3Depth introduces plane-coefficient representations, learned seed-pixel offsets, and confidence-based fusion to exploit local planarity. It achieves state-of-the-art supervised results across major benchmarks, while remaining limited by disconnected coplanar regions and erroneous ground-truth depth.
Problem
Pixel-level supervised depth prediction overlooks real scenes’ piecewise-smooth geometry, while monocular inputs additionally suffer from scale ambiguity.
Method
P3Depth predicts plane coefficients and a dense offset-confidence field to transfer information from coplanar seed pixels and adaptively fuse the resulting depth with an initial prediction.
Results
P3Depth sets a new state of the art among supervised approaches on NYU Depth-v2 and KITTI, and outperforms prior state of the art in zero-shot transfer on four additional datasets.
Takeaways & Limitations
The results support the potential of well-selected geometric priors for predicting consistent depth across smooth scene regions and producing plausible 3D reconstructions.
Takeaways & Limitations
Offset learning is not directly supervised, and performance is limited when coplanar regions are disconnected or when depth measurements contain erroneous ground truth.
Abstract
from arXiv · showhide
Monocular depth estimation is vital for scene understanding and downstream tasks. We focus on the supervised setup, in which ground-truth depth is available only at training time. Based on knowledge about the high regularity of real 3D scenes, we propose a method that learns to selectively leverage information from coplanar pixels to improve the predicted depth. In particular, we introduce a piecewise planarity prior which states that for each pixel, there is a seed pixel which shares the same planar 3D surface with the former. Motivated by this prior, we design a network with two heads. The first head outputs pixel-level plane coefficients, while the second one outputs a dense offset vector field that identifies the positions of seed pixels. The plane coefficients of seed pixels are then used to predict depth at each position. The resulting prediction is adaptively fused with the initial prediction from the first head via a learned confidence to account for potential deviations from precise local planarity. The entire architecture is trained end-to-end thanks to the differentiability of the proposed modules and it learns to predict regular depth maps, with sharp edges at occlusion boundaries. An extensive evaluation of our method shows that we set the new state of the art in supervised monocular depth estimation, surpassing prior methods on NYU Depth-v2 and on the Garg split of KITTI. Our method delivers depth maps that yield plausible 3D reconstructions of the input scenes. Code is available at: https://github.com/SysCV/P3Depth
1. Introduction
Supervised monocular depth estimation typically predicts pixels independently, overlooking piecewise-smooth scene geometry. P3Depth uses plane-based pixel interactions and learned seed selection to improve depth while preserving occlusion boundaries.
- Motivation: Monocular depth estimation predicts each pixel’s perpendicular 3D coordinate but suffers from scale ambiguity because one image can correspond to infinitely many 3D scenes.Depth estimation supports applications including robotics and autonomous cars.
- Motivation: Pixel-level supervision treats predictions independently and ignores the regularity of real scenes, which generally produces piecewise-smooth depth maps.
- Method: P3Depth predicts dense plane coefficients as an intermediate representation, exploiting the fact that coplanar pixels share coefficients even when their depths differ.Plane coefficients provide an output space for defining interactions between pixels based on planarity priors.
- Method: A second head predicts offsets to nearby seed pixels and confidence values, then adaptively fuses seed-based and initial depth predictions where planarity may or may not hold.The offset field resamples plane coefficients from the first head, while confidence down-weights the offset-based prediction near non-planar regions.
- Results: P3Depth sets a new supervised monocular-depth state of the art on NYU Depth-v2 and the Garg split of KITTI, with qualitative evidence of high-quality 3D reconstructions.The evaluation covers six datasets and also reports superior zero-shot transfer on ScanNet, SUN-RGBD, DIODE Indoor, and ETH-3D.
2. Related Work
Prior work spans supervised monocular, unsupervised and semi-supervised monocular, stereo-based, and geometric-prior approaches. Piecewise planarity has long supported multi-view stereo and reconstruction, while newer deep methods incorporate geometric information in learned depth systems.
- Monocular depth estimation: Supervised monocular depth estimation learns an image-to-depth mapping from images paired with ground-truth depth during training.Early work used handcrafted piecewise-planar structures, followed by end-to-end deep CNN mappings.
- Alternative setups: Unsupervised and semi-supervised monocular methods use alternatives such as stereo pairs, temporal information, multi-view collections, or diverse supervision formats.
- Stereo depth estimation: Stereo depth estimation progressed from handcrafted matching costs to learned matching functions and fully convolutional architectures.
- Geometric priors: Piecewise planarity has traditionally been used in multi-view stereo and 3D reconstruction, with explicit planes fitted to superpixels or point sets.Related applications also include depth denoising and completion.
3. Method
P3Depth represents depth through plane coefficients, discovers nearby coplanar seed pixels with learned offsets, and fuses seed-based and initial predictions using confidence. This design supports differentiable depth estimation while accounting for cases where the planarity prior is unreliable.
- Plane Coefficient Representation for Depth: Plane coefficients provide a shared representation for pixel depth and planar structure, enabling depth computation from coefficients and pixel position.The model predicts dense coefficients and converts them into an initial depth map through the plane-based parameterization.
- Learning to Identify Seed Pixels: The piecewise planarity prior assumes that a nearby seed pixel can share the same 3D plane as each pixel when such a seed exists.The method lets the network discover both the seed pixel and the region in which its plane coefficients should be reused.
- Learning to Identify Seed Pixels: A second network head predicts dense offsets that resample plane coefficients from seed locations, producing a second depth prediction.The two heads share an encoder and use separate decoders; bilinear interpolation handles fractional offsets during resampling.
- Confidence-Based Fusion: A confidence map adaptively fuses seed-based depth Ds with initial depth Di because the planarity prior is not always valid.The final prediction is Df(u, v) = F(u, v)Ds(u, v) + (1 - F(u, v))Di(u, v).
- Training: The model supervises the fused, seed-based, and initial depth predictions, while offsets are learned indirectly through the depth loss.Without direct offset supervision, boundary smoothing provides a signal favoring offsets away from depth boundaries and helps produce non-trivial offset fields.
4. Experiments
The experiments evaluate P3Depth on supervised monocular depth benchmarks, zero-shot transfer datasets, qualitative reconstructions, and component ablations. Results report state-of-the-art performance on NYU Depth-v2 and the Garg split of KITTI, strong zero-shot generalization, and benefits from the proposed planar representation and offset refinement.
- Experimental Setup: The evaluation uses NYU Depth-v2 and KITTI for primary training and testing, plus four unseen datasets for zero-shot transfer.The experiments use standard depth metrics across all six datasets.
- Comparison with State of the Art: 9.18% relative RMSE gain, 3.7% A.Rel gain, and 1.1% δ1 improvement establish new state-of-the-art results on NYU Depth-v2.The gains are reported against prior state-of-the-art methods across all six standard metrics.
- Comparison with State of the Art: Qualitative NYU results show sharp, consistent predictions on planar indoor structures and 3D reconstructions that better preserve scene structure.The reported examples include tables, cupboards, computer screens, and photo frames.
- Comparison with State of the Art: 14.2% RMSE improvement and 1.0% δ1 improvement surpass prior state of the art on the Garg split of KITTI.Performance is comparable to state of the art on the Eigen split, while distant regions are harder because smooth depth regions become smaller.
- Comparison with State of the Art: A qualitative KITTI example exposes a limitation: disconnected road regions beside a traffic-sign pole receive significantly different depth values.The regions do not interact in terms of plane information.
- Comparison with State of the Art: Zero-shot testing achieves best performance on ScanNet and SUN-RGBD in all metrics, while leading RMSE and δ1 on DIODE Indoor and ETH-3D.All models are trained on NYU Depth-v2 and tested on other datasets without fine-tuning.
- Ablation Studies: Ablations show that plane coefficients become beneficial when combined with offset vectors, while guidance, cascaded offset refinement, and mean plane loss further improve results.The full configuration with mean plane loss yields state-of-the-art NYU Depth-v2 results.
5. Conclusion
P3Depth leverages local planar information through plane coefficients, learned seed offsets, and adaptive depth fusion to produce consistent depth across smooth scene regions. Empirical validation across major indoor and outdoor benchmarks establishes new state-of-the-art performance among supervised approaches.
- P3Depth shares plane information from learned seed locations to improve depth consistency across smooth scene regions.The method predicts offsets to seed locations and adaptively weights their depth according to accuracy.
- The method was empirically validated on major indoor and outdoor monocular depth-estimation benchmarks.
- P3Depth establishes new state-of-the-art performance among supervised monocular depth-estimation approaches.
A. Network Architecture
The architecture uses a ResNet101 encoder and a decoder built from feature-fusion modules that transform, combine, and upsample encoder features. Additional architecture visualizations include the plane coefficient guidance module.
- Encoder: A ResNet101 encoder extracts increasingly contextual features while reducing feature-map resolution to approximately 1/32 of the input.
- Decoder: The decoder replaces ReLU with ELU nonlinearities and uses feature-fusion modules to transform and combine encoder skip features.
- Decoder: Feature maps are upsampled within the decoder to match the resolution required by the final decoder layer.
- Plane coefficient guidance: Figure 7 presents the plane coefficient guidance module.
B. Additional Results
Additional experiments report KITTI benchmark-server results and qualitative visualizations on KITTI and NYU Depth-v2. The visualizations include offset fields, failure cases, predictions, and reconstructed scene views.
- KITTI benchmark-server evaluation: KITTI benchmark-server results are on par with state-of-the-art methods and outperform the baseline, although one cited method performs better.The method’s performance degrades at distances beyond the ranges where its planar-offset design is most effective.
- KITTI benchmark-server evaluation: The KITTI benchmark-server results are reported in Table 6.
- Qualitative results: Qualitative results cover KITTI and NYU Depth-v2, including predicted depth maps, offset vector fields, failure cases, and 3D reconstructions.
- Qualitative results: The visualizations include left-side, right-side, top, and top-left or top-right views.