Source-linked AI summary
Recurrent Neural Network for (Un-)supervised Learning of Monocular VideoVisual Odometry and Depth
Rui Wang, Stephen M. Pizer, Jan-Michael Frahm
TL;DR
Single-view depth methods overlook motion, despite its importance for depth perception. The paper proposes an RNN-based monocular-video framework that jointly estimates depth and visual odometry using multi-view reprojection and flow consistency, achieving superior KITTI results in supervised and unsupervised settings.
Problem
Single-view depth estimation methods ignore temporal motion information that is important for determining depth.
Method
An RNN-based monocular-video framework jointly estimates depth and visual odometry using ConvLSTM units, multi-view reprojection, and forward-backward flow-consistency constraints.
Results
The method produces superior results to state-of-the-art learning-based depth and visual odometry approaches on KITTI under both supervised and unsupervised training.
Takeaways & Limitations
The system uses temporal information from previous frames, supports supervised or unsupervised training, and runs on arbitrary-length sequences with temporally coherent results.
Takeaways & Limitations
Flow consistency is computed only on consecutive frame pairs because larger separations increase flow magnitude and make pseudo-inverses inaccurate through interpolation.
Abstract
from arXiv · showhide
Deep learning-based, single-view depth estimation methods have recently shown highly promising results. However, such methods ignore one of the most important features for determining depth in the human vision system, which is motion. We propose a learning-based, multi-view dense depth map and odometry estimation method that uses Recurrent Neural Networks (RNN) and trains utilizing multi-view image reprojection and forward-backward flow-consistency losses. Our model can be trained in a supervised or even unsupervised mode. It is designed for depth and visual odometry estimation from video where the input frames are temporally correlated. However, it also generalizes to single-view depth estimation. Our method produces superior results to the state-of-the-art approaches for single-view and multi-view learning-based depth estimation on the KITTI driving dataset.
1. Introduction
The paper addresses the limitation of single-view depth estimation by incorporating temporal information from monocular video. Its recurrent multi-view framework jointly estimates depth and visual odometry, supports supervised and unsupervised training, and reports superior KITTI results.
- Depth and odometry estimation provide valuable information for autonomous driving, AR/VR applications, and virtual tourism.
- CNN depth methods can match traditional geometric methods in measurable areas and produce more complete results in ambiguous areas through learned priors.
- Most CNN depth methods treat estimation as a single-view task, ignoring temporal information in monocular or stereo videos.
- The proposed framework uses ConvLSTM units to carry information from previous views into current-frame depth and visual odometry estimates.
- Multi-view image reprojection and forward-backward flow-consistency constraints provide supervision for ambiguous image areas and enable satisfactory training without groundtruth.
- The RNN design supports supervised and unsupervised training and continuous operation on arbitrary-length sequences with a consistent scene scale.
- The method produces superior results to state-of-the-art approaches for supervised and unsupervised training on KITTI.
2. Related work
Related work spans geometric reconstruction, CNN-based monocular depth, and learned stereo or unsupervised methods. The paper positions its RNN approach as using multiple views to improve depth and pose estimation while combining temporal and geometric constraints.
- SfM primarily performs offline reconstruction from unordered image collections, whereas visual SLAM targets real-time reconstruction with a single camera.
- CNNs have increasingly been applied to dense monocular depth reconstruction, whose structure can be adapted from segmentation networks.
- Supervised methods: Supervised work progressed from end-to-end single-view depth networks to deeper residual, joint depth-normal, ordinal-regression, and multi-view approaches.
- Supervised methods: The proposed work uses an RNN architecture to incorporate multiple views for more accurate depth and pose estimation.
- Two-view or multi-view methods: Learned two-view stereo methods explicitly incorporate dense optical-flow correspondences so networks use stereo information beyond single-view priors.
- Unsupervised methods: Unsupervised monocular depth methods transform depth estimation into image reconstruction, using stereo or temporal constraints and, in some cases, jointly learned optical flow.
3. Method
The method jointly estimates monocular-video depth and visual odometry with recurrent networks that retain temporal information across frames. It trains these estimates using multi-view reprojection, forward-backward flow consistency, smoothness, and optionally ground-truth depth constraints.
- Network architecture: The framework uses separate recurrent depth and visual odometry networks to estimate depth maps and relative camera poses from monocular video.The depth network interleaves ConvLSTM units with an encoder, while the odometry network uses recurrent units and outputs relative 6DoF poses.
- Network architecture: ConvLSTM units carry temporal information from previous views into current depth estimation, combining spatial-temporal encoder features for depth reconstruction.The depth network receives the current RGB frame and internally transmits hidden states between time steps.
- Network architecture: The odometry network uses the current image and estimated depth, while previous-frame knowledge is retained in hidden states rather than an input frame stack.Using current depth as input maintains a consistent scene scale between depth and camera pose, which is important when unsupervised depth scale is ambiguous.
- Multi-view reprojection: A differentiable geometric module warps previous views into the current view using estimated depth, camera poses, camera intrinsics, and differentiable 2D-to-3D and 3D-to-2D projections.The multi-view reprojection loss explicitly constrains the current view against every previous view in the sequence, including a reversed-sequence backward pass.
- Flow consistency: Forward-backward flow consistency compares dense forward flow with a pseudo-inverse derived from backward flow, adding constraints for depth and pose estimation.The loss is applied only to consecutive frame pairs because larger flow magnitudes make pseudo-inverses less accurate through interpolation.
- Training objectives: Combining forward and backward reprojection, flow consistency, and smoothness enables unsupervised training without ground-truth depth, but produces depth only at relative scale.When depth ground truth is available, including sparse ground truth can support supervised training or replace local smoothness with gradient similarity.
4. Experiments
Experiments on KITTI evaluate supervised and unsupervised depth, visual odometry, and architectural components of the recurrent multi-view framework. The method outperforms competing depth methods, maintains coherent long-sequence pose estimates, and benefits from encoder recurrence and multi-view consistency constraints.
- Experimental setup: The KITTI experiments evaluate supervised and unsupervised versions of the RNN-based depth and visual odometry framework.The supervised and unsupervised models are named ours-sup and ours-unsup, respectively.
- Depth estimation: Visual comparisons show that the method captures more detail in thin structures, including motorcycles and columns.The comparison uses interpolated ground-truth depth for visualization.
- Depth estimation: Ours-sup (multi-view) performs significantly better than all compared supervised and unsupervised depth-estimation methods on KITTI.The framework also reports single-view results to separate bootstrapping performance from the benefits of previous views.
- Pose estimation: Learning-based full trajectories are formed by integrating frame-to-frame relative poses over whole sequences without drift correction, and the predictions are compared with ground truth.Table 4 reports translational and rotational errors using the KITTI visual-odometry criterion.
- Pose estimation: The recurrent model supports continuous pose estimation over arbitrarily long video sequences while maintaining a consistent scale within each full sequence.Competing methods using independent five-frame subsequences have different scales between subsequences.
- Ablation study: Ablations select encoder LSTM placement because it performs significantly better than full-LSTM and decoder-LSTM alternatives.The architecture experiments use the same evaluation data and protocol as the depth comparison.
- Ablation study: Multi-view reprojection and forward-backward consistency provide stronger supervision, especially in unsupervised training and regions lacking ground-truth depth.The ablation compares consecutive reprojection alone, consecutive reprojection with consistency, and multi-view reprojection with consistency.
- Ablation study: Depth-estimation performance does not increase after 10 frames, although the network trained on 10-frame subsequences can operate on sequences of arbitrary length.This result concerns the time-window ablation and continuous-sequence capability.