Source-linked AI summary
Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction
Huangying Zhan, Ravi Garg, Chamara Saroj Weerasekera, Kejie Li, Harsh Agarwal, Ian Reid
TL;DR
The paper targets the costly supervision and scale ambiguity affecting learned monocular depth and visual odometry. It trains jointly on stereo sequences with spatial and temporal reconstruction constraints and adds deep feature warping, reporting improved depth and odometry results on KITTI without scale ambiguity. Its scope is limited by rigid-scene and no-occlusion assumptions, and it does not compare favorably with state-of-the-art SLAM systems.
Problem
Existing depth and visual-odometry methods often require expensive ground-truth annotations, while monocular self-supervision leaves camera translation scale ambiguous.
Method
The framework jointly learns single-view depth and monocular odometry from stereo video using spatial and temporal photometric constraints plus deep feature reconstruction.
Results
The authors report improved depth and odometry accuracy, state-of-the-art unsupervised results without scale ambiguity, and strong performance on KITTI.
Takeaways & Limitations
Stereo training supplies a common real-world scale, allowing single-camera frame-to-frame odometry without mapping while jointly learned constraints improve depth estimation.
Takeaways & Limitations
The framework assumes rigid scenes without occlusions, and its odometry does not compare favorably with state-of-the-art SLAM systems.
Abstract
from arXiv · showhide
Despite learning based methods showing promising results in single view depth estimation and visual odometry, most existing approaches treat the tasks in a supervised manner. Recent approaches to single view depth estimation explore the possibility of learning without full supervision via minimizing photometric error. In this paper, we explore the use of stereo sequences for learning depth and visual odometry. The use of stereo sequences enables the use of both spatial (between left-right pairs) and temporal (forward backward) photometric warp error, and constrains the scene depth and camera motion to be in a common, real-world scale. At test time our framework is able to estimate single view depth and two-view odometry from a monocular sequence. We also show how we can improve on a standard photometric warp loss by considering a warp of deep features. We show through extensive experiments that: (i) jointly training for single view depth and visual odometry improves depth prediction because of the additional constraint imposed on depths and achieves competitive results for visual odometry; (ii) deep feature-based warping loss improves upon simple photometric warp loss for both single view depth estimation and visual odometry. Our method outperforms existing learning based methods on the KITTI driving dataset in both tasks. The source code is available at https://github.com/Huangying-Zhan/Depth-VO-Feat
1. Introduction
The paper addresses unsupervised joint learning of single-view depth and visual odometry using stereo video, which supplies spatial and temporal constraints and resolves scale ambiguity. It also introduces deep feature reconstruction to complement color-based photometric warping.
- Motivation: Ground-truth annotations for depth and odometry are expensive, motivating photometric-error supervision for learning these tasks.Prior methods use photometric warp error to reduce reliance on depth ground truth.
- Motivation: Monocular joint depth-and-odometry methods suffer per-frame scale ambiguity because camera-translation magnitude is missing.Scale consistency is important for accurate tracking in SLAM systems.
- Proposed framework: Stereo video enables spatial and temporal photometric constraints while placing predicted depth and camera motion in a shared real-world scale.The scale is set by the known stereo baseline.
- Feature reconstruction: Deep feature reconstruction adds contextual information beyond per-pixel color matching in photometric warp supervision.Color warping assumes brightness and color consistency and can be ambiguous in uniformly colored regions.
- Reported contributions: The authors report improved depth and odometry accuracy from feature reconstruction, alongside strong depth and frame-to-frame odometry results without scale ambiguity.The contributions include improvements over prior single-view depth work and odometry results on par with geometric methods.
2. Related Work
Related work moves from supervised depth and odometry toward photometric self-supervision, but monocular approaches retain scale ambiguity and photo-consistency limitations. This paper combines stereo and temporal constraints and adds learned feature matching to address those issues.
- Supervised methods: Early deep-learning approaches estimated monocular depth or odometry with supervised objectives and ground-truth data.Supervised structure-from-motion can require depth, camera motion, surface normals, and optical flow ground truth.
- Unsupervised or semi-supervised methods: Stereo photometric warp losses replaced depth ground truth by reconstructing one view from another using known inter-camera geometry.This established an unsupervised route to stereo depth estimation.
- Unsupervised or semi-supervised methods: Monocular depth-and-ego-motion systems use photometric supervision but inherit scale ambiguity in monocular reconstruction.The ambiguity concerns the metric scale of motion and depth.
- Unsupervised or semi-supervised methods: The paper trains with stereo pairs to avoid depth-speed ambiguity and jointly estimates ego-motion, enforcing both stereo and temporal constraints.These constraints are used together to improve depth estimation.
- Warping losses: Photometric consistency is often violated, so prior work uses robust norms or handcrafted features such as SSIM, SIFT, HOG, and ORB.Learned matching features are presented as another direction for robust warping losses.
3. Method
The framework jointly learns depth and visual odometry from stereo sequences, using spatial and temporal reconstruction constraints plus differentiable geometry and feature-based supervision.
- Image reconstruction as supervision: Stereo sequences provide left-right and forward-backward consistency checks for jointly learning the depth and visual odometry ConvNets.The known stereo pose constrains depth and temporal pose in a common real-world scale, avoiding scale ambiguity at test time.
- Image reconstruction as supervision: Image reconstruction trains both networks by comparing real views with views synthesized from predicted depth, camera pose, camera intrinsics, and differentiable warping.The reference depth comes from CNND, while temporal pose comes from CNNV O.
- Image reconstruction as supervision: Each training instance uses a temporal pair and a stereo pair, treating the left image at time t2 as the reference view and synthesizing it from two live views.The two live views are the earlier left frame and the right stereo frame at t2.
- Feature reconstruction as supervision: Feature reconstruction supplements image reconstruction to improve robustness when the Lambertian brightness-constancy assumption is violated.The method synthesizes dense feature views and uses their reconstruction loss as auxiliary supervision.
- Training loss: The total training loss combines image reconstruction, feature reconstruction, and edge-aware depth smoothness terms with separate loss weightings.The smoothness regularizer uses inverse depth and penalizes discontinuities less where image edges indicate depth changes.
- Network architecture: The depth network uses a reduced ResNet50 encoder and decoder, while the odometry network predicts a 6D relative pose vector from two concatenated views.The reduced encoder has fewer than 7 million parameters; the odometry network uses six stride-2 convolutions followed by three fully connected layers.
4. Experiments
Experiments on KITTI evaluate visual odometry, single-view depth, and loss components through benchmark comparisons, qualitative examples, and ablations. The results examine stereo and temporal training, deep feature reconstruction, and trajectory behavior across translation magnitudes.
- Evaluation setup: The experiments evaluate depth and visual odometry on KITTI using separate Eigen and Odometry splits.Odometry uses sequences 09 and 10; depth evaluation follows the Eigen Split protocol.
- Visual odometry results: Visual odometry is compared with monocular learning and ORB-SLAM using translational and rotational drift errors on sequences 09 and 10.ORB-SLAM results use trajectory alignment, whereas the proposed method integrates frame-to-frame poses without post-processing.
- Visual odometry results: The stereo-based method outperforms monocular learning by a large margin and approaches full ORB-SLAM on sequence 09 without translation-scale post-processing.Sequence 10 shows larger gradual drift associated with greater frame-to-frame rotation error.
- Visual odometry results: As translation magnitude increases, frame-to-frame odometry drifts gradually, while both learned methods outperform ORB-SLAM for small translations.The comparison is reported for sequence 09 across different translation bins.
- Depth estimation results: Stereo-trained depth methods substantially outperform monocular training, and the proposed method improves on prior stereo methods through feature reconstruction and additional odometry warping.The evaluation uses the Eigen Split with 50m and 80m maximum-depth thresholds and standard error measures.
- Ablation studies: Ablations show incremental depth improvements from joint pose training, ImageNet features, task-specific features, and the combined Stereo + Temporal + NYUv2 Feat. system.The combined system performs best in the top part of Table 3.
- Ablation studies: Self-embedded depth features improve depth predictions without explicit matching supervision, while deep feature errors provide more distinctive minima than photometric errors.Photometric matching is especially ambiguous in texture-less or otherwise ambiguous regions.
5. Conclusion
The framework jointly learns single-view depth and monocular visual odometry from stereo data, while feature reconstruction supports state-of-the-art performance without scale ambiguity. Its main limitations are rigid-scene and no-occlusion assumptions, and weaker comparison with state-of-the-art SLAM systems.
- Conclusion: Stereo data enables joint learning of single-view depth and monocular visual odometry, with odometry prediction in metric scale from two frames.The stereo baseline provides the shared real-world scale.
- Conclusion: Temporal image alignment complements stereo alignment for single-view depth prediction.
- Conclusion: The framework assumes no occlusion and rigid scenes, limiting its direct applicability to dynamic or occluded environments.
- Conclusion: Its odometry is comparable to strong two-frame estimates but does not compare favorably with state-of-the-art SLAM systems.