Source-linked AI summary
Enforcing geometric constraints of virtual normal for depth prediction
Wei Yin, Yifan Liu, Chunhua Shen, Youliang Yan
TL;DR
Monocular depth prediction remains ill-posed, while many methods neglect global 3D geometric constraints. This paper adds virtual-normal supervision by sampling distant triplets in reconstructed point clouds, achieving state-of-the-art results on NYUD-v2 and KITTI while recovering 3D geometry directly from depth.
Problem
Monocular depth prediction is ill-posed, and existing methods largely neglect geometric constraints in 3D space.
Method
The method combines pixel-wise depth supervision with a virtual-normal loss computed from long-range triplets of points in reconstructed 3D point clouds.
Results
The method achieves state-of-the-art performance on NYUD-v2 and KITTI, with improvements of up to 29% over real-time systems using comparable network parameters.
Takeaways & Limitations
Accurate depth can directly yield high-quality point clouds and surface normals, without training a new sub-model for surface-normal estimation.
Abstract
from arXiv · showhide
Monocular depth prediction plays a crucial role in understanding 3D scene geometry. Although recent methods have achieved impressive progress in evaluation metrics such as the pixel-wise relative error, most methods neglect the geometric constraints in the 3D space. In this work, we show the importance of the high-order 3D geometric constraints for depth prediction. By designing a loss term that enforces one simple type of geometric constraints, namely, virtual normal directions determined by randomly sampled three points in the reconstructed 3D space, we can considerably improve the depth prediction accuracy. Significantly, the byproduct of this predicted depth being sufficiently accurate is that we are now able to recover good 3D structures of the scene such as the point cloud and surface normal directly from the depth, eliminating the necessity of training new sub-models as was previously done. Experiments on two benchmarks: NYU Depth-V2 and KITTI demonstrate the effectiveness of our method and state-of-the-art performance.
1. Introduction
Monocular depth prediction is ill-posed, and existing methods largely optimize pixel-wise or local geometric constraints. The paper introduces virtual normals as a global 3D constraint, enabling accurate depth alongside high-quality point clouds and surface normals, with state-of-the-art benchmark results.
- Monocular depth prediction is ill-posed because multiple 3D scenes can project to the same 2D image.
- Prior methods commonly use pixel-wise supervision, while alternative geometric constraints generally capture local or neighborhood relationships.Examples include pair-wise CRF information, gravity constraints, and depth–surface-normal transformations.
- Virtual normals impose a global constraint by sampling three distant non-collinear 3D points to form a virtual plane and comparing its normal direction with ground truth.The points are obtained after lifting image pixels into 3D using the estimated depth map.
- Combining virtual-normal and pixel-wise supervision produces accurate depth maps and high-quality 3D point clouds from which surface normals are computed directly.The method does not introduce a separate model or network branch for surface-normal estimation.
- The method achieves state-of-the-art performance on NYUD-v2 and KITTI, improving over real-time systems by up to 29% with comparable network parameters.Recovered surface-normal precision is reported as on par with dedicated DCNN-based surface-normal estimation methods.
2. Our Method
The method combines pixel-wise depth supervision with a high-order virtual normal constraint in reconstructed 3D space. It uses sampled point triplets to improve geometric robustness and directly recover 3D features from predicted depth.
- Pipeline: An encoder-decoder predicts depth, reconstructs a 3D point cloud, and applies both pixel-wise and virtual-normal supervision during training.The predicted and ground-truth depth maps are lifted into corresponding point clouds before geometric alignment.
- Inference: After training, surface normals and other 3D features are computed directly from reconstructed point clouds without additional models or network branches.The pipeline therefore produces depth and point-cloud geometry from the same trained network.
- Motivation: Surface normals are sensitive to noise and local sampling size, with a 22° Mean Difference Error between 3×3 and 11×11 neighborhoods.This instability limits surface normals as geometric supervision for depth prediction.
- Robustness: The virtual normal is designed to be more robust to depth and data noise than local surface normal estimation.The paper evaluates robustness using noisy point clouds and Mean Difference Error across noise levels.
- Virtual Normal: Virtual normal establishes long-range, triplet-based geometric relations by computing plane normals from randomly sampled non-collinear 3D point groups.Sampling also applies a long-range restriction to select points with sufficiently large distances.
- Training Objective: The overall training objective combines weighted cross-entropy depth loss and virtual normal loss as ℓ = ℓ_WCE + λℓ_VN, with λ set to 5.The two terms are made roughly comparable in scale, and the resulting loss is differentiable.
3. Experiments
Experiments on NYUD-V2 and KITTI evaluate the method against state-of-the-art systems and isolate the effects of pixel-wise and 3D geometric supervision. Virtual-normal supervision improves depth prediction, especially in ambiguous regions, while also supporting accurate surface-normal recovery from reconstructed point clouds.
- Datasets and metrics: The method is evaluated on NYUD-V2 and KITTI using standard depth metrics, including rel, log10, rms, rms (log), and threshold accuracy.NYUD-V2 includes NYUD-Large and NYUD-Small settings, while KITTI uses the Eigen split and driving-scene imagery.
- Comparison with state of the art: On NYUD-V2, the method outperforms state-of-the-art methods across all reported evaluation metrics and improves over DORN by 0.2% to 18%.Visual comparisons also show smoother planes and fewer errors in walls, shelves, and desks.
- Comparison with state of the art: On KITTI, the method leads on every evaluation metric except rms, where it is only slightly behind DORN; it performs better on rms (log).The authors identify rms (log) as the more relevant outdoor-scene metric.
- Ablation studies: Adding two pixel-wise losses does not improve performance, whereas combining WCEL with VNL improves the baseline by up to 8%.The ablation compares CEL, L1, surface-normal loss, pair-wise loss, and virtual-normal loss.
- Ablation studies: VNL encodes stronger global geometry than pair-wise supervision, and its benefits appear in ambiguous KITTI regions containing signs, pedestrians, and traffic lights.Performance saturates after 20K sampled point groups, while the lightweight MobileNetV2 setting improves by 1%–8% and by around 29% over prior real-time systems.
- Recovered geometry: Surface normals computed directly from reconstructed point clouds outperform many depth-based alternatives and approach methods trained specifically to predict normals.The comparison includes DORN, GeoNet, and Eigen, with strong results on both planar and curved regions.
4. Conclusion
The paper concludes that virtual-normal loss supplies a long-range 3D constraint for monocular depth prediction. This improves depth maps and enables high-quality point-cloud and surface-normal recovery without training an additional sub-model.
- Conclusion: VNL constructs long-range geometric constraints in 3D, complementing the pixel-wise depth supervision used by previous approaches.The conclusion contrasts the method with approaches relying only on 2D pixel-wise supervision.
- Conclusion: The method recovers high-quality point clouds and surface normals directly from predicted depth, eliminating the need to optimize a new sub-model.The directly derived normals outperform many recent depth-estimation methods and approach methods trained to output normals.
- Conclusion: Experiments on NYUD-V2 and KITTI support the method’s effectiveness and state-of-the-art performance.The conclusion also characterizes VNL as more robust to noise and stronger than other 3D constraints.
5. Appendix
The appendix describes the encoder-decoder architecture and additional visualizations of predicted depth, recovered normals, and reconstructed point clouds on NYUD-V2 and KITTI.
- Architecture: The model uses an encoder-decoder architecture with adaptive merging and dilated residual blocks to reconstruct the depth map.The encoder establishes multilevel features, while the decoder fuses and transforms them.
- Architecture: The prediction module transfers features from 64 channels to 60 depth bins.This implementation detail is stated separately from the broader architecture description.
- Architecture: Figure 11 depicts four flip connections that merge low-level features in the encoder-decoder network.The figure provides an architectural overview rather than an experimental comparison.
- Qualitative results: Additional figures show predicted depth maps and recovered surface normals for KITTI and NYUD-V2.The normals are presented as recovered geometry derived from the predicted depth.
- Qualitative results: Reconstructed point clouds are visualized from three randomly selected scenes per dataset, using three views for each scene.The visualizations cover both NYUD-V2 and KITTI test scenes.
Image GT Ours
The appendix presents qualitative outputs for recovered surface normals, predicted KITTI depth, and reconstructed point clouds on NYUD-V2 and KITTI.
- Recovered surface normals: Figure 13 shows high-quality surface normals recovered from reconstructed NYUD-V2 point clouds.The figure contains samples of the recovered normals.
- Predicted depth: Figure 14 presents high-quality predicted depth maps for KITTI scenes.The caption uses these results to illustrate the effectiveness of the method.
- Reconstructed point clouds: Figures 15 and 16 visualize reconstructed point clouds from three NYUD-V2 and three KITTI scenes, respectively.Each scene is shown from three selected views.