Source-linked AI summary
Unsupervised Learning of Geometry with Edge-aware Depth-Normal Consistency
Zhenheng Yang, Peng Wang, Wei Xu, Liang Zhao, Ramakant Nevatia
TL;DR
Unsupervised monocular-video depth learning is challenged by limited depth labels and ambiguous photometric matching. The paper introduces edge-aware depth-normal consistency through coupled depth and normal representations, reporting improved depth and normal estimation with faster convergence.
Problem
Depth labels are difficult to collect for random videos, while photometric matching and weak depth smoothness can produce inconsistent scene geometry.
Method
The method uses edge-aware depth-normal consistency inside an unsupervised framework, combining predicted depths and normals with photometric and smoothness training losses.
Results
The method reports 20% relative improvement in depth estimation, 10% improvement in predicted normals, and approximately 3× faster training convergence than the state-of-the-art comparison.
Takeaways & Limitations
Depth-normal regularization improves geometry evaluation and training speed, with the paper reporting state-of-the-art performance using only monocular videos.
Takeaways & Limitations
The authors identify extending consistency to supervised joint normal-and-depth learning as future work.
Abstract
from arXiv · showhide
Learning to reconstruct depths in a single image by watching unlabeled videos via deep convolutional network (DCN) is attracting significant attention in recent years. In this paper, we introduce a surface normal representation for unsupervised depth estimation framework. Our estimated depths are constrained to be compatible with predicted normals, yielding more robust geometry results. Specifically, we formulate an edge-aware depth-normal consistency term, and solve it by constructing a depth-to-normal layer and a normal-to-depth layer inside of the DCN. The depth-to-normal layer takes estimated depths as input, and computes normal directions using cross production based on neighboring pixels. Then given the estimated normals, the normal-to-depth layer outputs a regularized depth map through local planar smoothness. Both layers are computed with awareness of edges inside the image to help address the issue of depth/normal discontinuity and preserve sharp edges. Finally, to train the network, we apply the photometric error and gradient smoothness for both depth and normal predictions. We conducted experiments on both outdoor (KITTI) and indoor (NYUv2) datasets, and show that our algorithm vastly outperforms state of the art, which demonstrates the benefits from our approach.
1 Introduction
The paper targets unsupervised single-image geometry learning from monocular videos, where photometric ambiguity and weak depth smoothness can produce inconsistent scene structure. It introduces edge-aware depth-normal consistency to regularize geometry and reports improved accuracy and faster convergence.
- Ground-truth depth is difficult to collect for random videos, limiting supervised models’ generalization to different scenes.
- Photometric matching is ambiguous, and weak neighboring-pixel smoothness can yield inconsistent normal estimates.
- The framework explicitly represents normals, adds higher-order interactions among estimated depths, and permits further operations such as Manhattan assumptions.
- The method incorporates an edge-aware depth-normal consistency constraint to regularize depths while handling depth and normal discontinuities at image edges.
- 20% relative improvement over the state of the art is reported for depth estimation, alongside 10% improvement in predicted normals and approximately 3× faster training convergence.
2 Related Work
Prior work includes geometry-based reconstruction, supervised CNN estimation, and video-based unsupervised learning. These approaches differ in their reliance on matching, labeled depth, or learned cross-view reconstruction.
- Structure from motion and single view geometry: Structure-from-motion, ORB-SLAM, and DTAM rely on feature matching and can fail in low-texture regions or under drastic viewpoint changes.
- Supervised single view geometry via CNN: Supervised CNNs estimate pixel-wise depth and normal maps from single images, improving over hand-crafted features and sometimes using CRFs for refinement.
- Supervised single view geometry via CNN: Most supervised methods treat depth and normal prediction as independent tasks, although dense-CRF methods exploit their correlations over large planar regions.
- Unsupervised single view geometry: Unsupervised methods learn single-view geometry from videos or stereo supervision, using view synthesis and differentiable depth-related operations instead of dense depth labels.
3 Preliminaries
The unsupervised pipeline uses predicted geometry and camera motion to inverse-warp source views, synthesize the target, and train with photometric, smoothness, and masking objectives.
- Perspective projection between multiple views: Inverse warping uses target-view depth, camera intrinsics, and relative pose to localize corresponding pixels in a source view.
- Perspective projection between multiple views: Dehomogenized source coordinates form target-source matching pairs whose image similarity evaluates the predicted scene structure.
- Photometric error from view synthesis: Photometric reconstruction synthesizes the target view from warped source images through bilinear interpolation under Lambertian and static-scene assumptions.
- Photometric error from view synthesis: The depth smoothness term penalizes second-order spatial gradients, encouraging planar depth where image gradients are absent.
- A multiscale joint objective combines the preceding terms, and spatial-transform backpropagation supplies learning signals to depth, pose, and mask networks.
4 Geometry estimation with edge-aware depth-normal consistency
The method enforces depth–normal consistency inside the network through edge-aware depth-to-normal and normal-to-depth layers, using the resulting representations to regularize geometry during photometric training.
- Consistency formulation: The network predicts depth and embeds predicted normals as a regularizer for depth estimation, enforcing consistency across the full image.The approach uses two layers to connect depth and normal predictions directly during learning.
- Depth-to-normal layer: The depth-to-normal layer derives normals from an estimated depth map using neighboring 3D points and cross products.Eight-neighbor computation is used, with color-based weights reducing the influence of pixels across discontinuities.
- Normal-to-depth layer: The normal-to-depth layer recovers a regularized depth map by solving local orthogonality constraints from predicted normals and an initial depth map.The initial depth resolves the otherwise non-unique depth solution, while local equations are solved using minimum-square estimation.
- Edge awareness: Image gradients guide both conversion layers so incompatible pixels near depth or normal discontinuities receive reduced influence.This edge awareness is intended to preserve sharp image boundaries during depth–normal conversion.
- Training objective: Training combines view-synthesis supervision with photometric and gradient matching losses, plus smoothness losses for depth and normals.Normal smoothness introduces higher-order interactions between pixels, while the converted depth participates in reconstruction supervision.
- Training objective: Using the normal-to-depth output for supervision already outperforms the previous state of the art by around 10% in depth estimation with the same architecture.The reported comparison concerns the training strategy described in this section.
5 Experiments
The experiments evaluate the framework on KITTI depth benchmarks and use ablations to measure the contributions of depth-normal consistency, edge-aware smoothness, and normal smoothness. Comparisons also examine predicted normals, qualitative results, and preliminary indoor NYUv2 performance.
- Datasets and metrics: The framework is evaluated on KITTI using Eigen and KITTI test splits, with depth capped at 80 meters and standard depth and normal metrics.The Eigen split has 697 test images, while the KITTI split has 200 disparity images.
- Ablation study: The ablation study removes individual components from the full model and evaluates their effects on the KITTI split.The tested components include depth-normal consistency, image gradients in smoothness terms, and normal smoothness.
- Ablation study: Removing depth-normal consistency substantially worsens KITTI performance and delays convergence from 5 epochs to 15 epochs.The comparison is between the full model and the variant without the normal-to-depth layer.
- Ablation study: Removing image gradients from the smoothness terms worsens results, while removing normal smoothness has a smaller effect on depth but a larger effect on normal performance.The image-gradient ablations test edge awareness in depth and normal-depth consistency.
- Comparison with other methods: On KITTI, the method outperforms several supervised and unsupervised methods, while remaining slightly worse than methods using stereo inputs or depth ground truth on some comparisons.It outperforms Godard et al. on the KITTI test split for the Sq Rel metric, which penalizes large depth errors.
- Comparison with other methods: The method outperforms the Zhou et al. baseline under all reported normal metrics, while preliminary NYUv2 results work on scenes with intersecting planes but fail on cluttered-object scenes.The NYUv2 experiments use a subset centered on study-room scenes.
6 Conclusion
The paper proposes unsupervised depth and normal estimation through edge-aware depth-normal consistency, reporting improved evaluation performance and training speed. Results include ablations, scene comparisons, and state-of-the-art performance using only monocular videos, while supervised cross-supervision remains future work.
- Conclusion: The framework jointly estimates depth and normals using edge-aware depth-normal consistency to improve geometry consistency.The regularization connects different projections of the 3D scene.
- Conclusion: The method improves evaluation performance and training speed over the reported baselines.
- Conclusion: Ablation experiments examine each framework component and performance across different image scenes.
- Conclusion: The method achieves state-of-the-art performance using only monocular videos and exceeds some supervised methods.
- Conclusion: Supervised cross-supervision of normal and depth predictions is identified as future work.The authors expect it could provide additional improvements.