PaperlayerView exact revision on arXiv

Source-linked AI summary

Don't Forget The Past: Recurrent Depth Estimation from Monocular Video

Vaishakh Patil, Wouter Van Gansbeke, Dengxin Dai, Luc Van Gool

arXiv:2001.02613v2cs.CVcs.LGcs.ROeess.IV

TL;DR

The paper addresses depth estimation that processes video inputs but typically predicts frames independently, limiting use of temporal structure for robotic perception. It integrates three depth-estimation settings with ConvLSTM recurrence and reports consistent, significant gains over image-based and prior methods, including a 0.005 improvement for depth completion over an image-based baseline.

  • Problem

    Existing depth methods largely estimate frames independently, although robotic platforms perceive correlated video streams and different depth settings each have important cost, data, or accuracy limitations.

  • Method

    The method combines supervised prediction, self-supervised monocular prediction, and self-supervised depth completion with ConvLSTM recurrence and a training strategy for dense video inputs.

  • Results

    The recurrent method consistently and significantly outperforms image-based and state-of-the-art methods across the considered self-supervised scenarios; depth completion improves by 0.005 over the image-based baseline.

  • Takeaways & Limitations

    The framework supports recurrent time-series depth estimation from monocular video, optionally incorporating sparse depth, across supervised and self-supervised settings.

Abstract

from arXiv · show

Autonomous cars need continuously updated depth information. Thus far, depth is mostly estimated independently for a single frame at a time, even if the method starts from video input. Our method produces a time series of depth maps, which makes it an ideal candidate for online learning approaches. In particular, we put three different types of depth estimation (supervised depth prediction, self-supervised depth prediction, and self-supervised depth completion) into a common framework. We integrate the corresponding networks with a ConvLSTM such that the spatiotemporal structures of depth across frames can be exploited to yield a more accurate depth estimation. Our method is flexible. It can be applied to monocular videos only or be combined with different types of sparse depth patterns. We carefully study the architecture of the recurrent network and its training strategy. We are first to successfully exploit recurrent networks for real-time self-supervised monocular depth estimation and completion. Extensive experiments show that our recurrent method outperforms its image-based counterpart consistently and significantly in both self-supervised scenarios. It also outperforms previous depth estimation methods of the three popular groups. Please refer to https://www.trace.ethz.ch/publications/2020/rec_depth_estimation/ for details.

I. INTRODUCTION

The paper targets depth estimation for robotic platforms that perceive the world as video, rather than isolated images, by learning from temporal sequences. It unifies three depth-estimation settings with a ConvLSTM and introduces a training strategy for dense video prediction.

  • The proposed method trains and tests on time series to exploit correlations between perceptual inputs and scene geometry across a robot’s trajectory.
  • Depth estimation methods are not optimal individually: RGB prediction needs accurate ground truth, self-supervision has weaker results, and LiDAR-assisted completion is costly.
  • It places supervised prediction, self-supervised monocular prediction, and self-supervised depth completion in one framework with ConvLSTM-based temporal integration.
  • The paper addresses ConvLSTM training challenges caused by large dense-prediction feature maps, memory-limited sequences, and hidden-state learning requirements.
  • Experiments and ablations report that the recurrent method consistently and significantly outperforms image-based and current state-of-the-art methods across considered scenarios.

II. RELATED WORK

Prior work covers supervised RGB depth prediction, self-supervised monocular video estimation, sparse-depth completion, and video-based estimation. These approaches provide useful capabilities but generally do not integrate information across video sequences in the way targeted here.

  • Supervised depth estimation has progressed from geometric and probabilistic methods to deep networks that predict depth from single RGB images.
  • Self-supervised monocular methods use view synthesis to reduce dependence on ground-truth depth but require tuning and face scale ambiguity.
  • Depth-completion methods propagate sparse LiDAR, SLAM, or structure-from-motion measurements under image guidance.
  • Video-depth methods use ConvLSTM in supervised settings or motion cues and optical flow for temporally consistent offline estimation.

III. APPROACH

The approach formulates three depth-estimation settings within a unified framework for learning depth-map sequences. It uses synchronized RGB and optional sparse-depth inputs, with recurrent information passed through an encoder representation.

  • The framework unifies three existing depth-estimation methods and defines a recurrent model for learning time series of depth maps.
  • At each timestamp, synchronized RGB images, sparse depth inputs, and ground-truth depth maps share dimensions H × W, while sparse and invalid values are represented explicitly.
  • Supervised RGB prediction uses an encoder-decoder network that maps an image I to a dense predicted depth map ˆD.
  • The encoder produces a compact representation X that passes information across video frames for recurrent depth estimation.
  • The supervised loss uses a masked berHu objective so invalid ground-truth pixels do not contribute to training.

B. Self-supervised Depth Prediction with Monocular Video

For self-supervised monocular depth prediction, the model replaces ground-truth depth supervision with view synthesis from neighboring frames. It jointly estimates depth and camera motion and combines photometric, masking, and smoothing components.

  • Self-supervised monocular depth prediction uses an encoder-decoder network but replaces ground-truth depth loss with view-synthesis supervision.
  • The warped neighboring image is generated from camera intrinsics, relative pose, and predicted depth using bilinear sampling.
  • The model jointly estimates camera transformations with a pose network using the same frame sequences as input.
  • For each target frame, the loss uses the previous and next frames and selects the minimum photometric error across sources.
  • The training objective combines view-synthesis and edge-aware smoothing losses, with α set to 0.85.
  • Pixel-wise masking excludes objects such as cars moving at the camera’s speed, which would otherwise be predicted at infinite depth.

C. Self-supervised Depth Completion with Monocular Video and Sparse Depth Maps

The paper extends self-supervised depth completion to monocular videos with synchronized sparse depth maps, combining view synthesis, smoothing, and sparsity supervision. Separate processing and fusion of the two modalities supports depth completion under multiple sparse-input patterns.

  • The method takes monocular video and synchronized sparse depth maps as inputs because dense depth is difficult to obtain while sparse measurements are relatively cheap.Sparse measurements can be acquired with sensors such as 2D LiDAR.
  • The encoder processes RGB images and sparse depth inputs separately before fusing their representations.This design is reported to outperform simpler fusion methods based on input concatenation or distance-transformation maps.
  • Self-supervised depth completion combines view-synthesis loss, edge-aware smoothing loss, and berHu loss on sparse depth measurements.The berHu term is applied to the sparse depth input and its binary validity mask.
  • Sparsity supervision addresses scale ambiguities and textureless regions, stabilizes training, and can accelerate convergence, while dense view synthesis reduces sensitivity to sparse-depth noise.
  • The evaluation uses randomly sampled pixels, regularly spaced scanning lines, and a dense depth map as three sparse-depth patterns.

D. Learning Time Series of Depth Maps

The framework converts sequences of multimodal inputs into sequences of dense depth maps by inserting ConvLSTM temporal modeling between encoder and decoder stages. Encoder features feed the recurrent module, whose hidden state is decoded into depth in an end-to-end pipeline.

  • The framework formulates depth recovery as translation from spatiotemporal image and sparse-depth sequences to spatiotemporal dense-depth sequences.ConvLSTM models dependencies across frames in the input sequence.
  • The encoder output X is used as the ConvLSTM input because it is compact and information-dense, making recurrent optimization more efficient.
  • At each frame, the encoder produces Xt, ConvLSTM applies temporal convolutions, and the decoder converts the hidden state Ht into predicted depth.The complete network is trained end to end.
  • The training procedure for the ConvLSTM hidden state is illustrated in Fig. 3.

A. Network Architecture

The architecture combines depth and pose networks with an encoder-decoder depth pathway and a recurrent ConvLSTM module. Its training strategy learns an initial hidden state before optimizing ConvLSTM weights, while qualitative results show improved detail on small objects.

  • Network architecture: The depth and pose networks use separate ResNet-18 encoder modules, while the depth decoder contains four upconvolutional blocks inspired by DispNet.
  • Network architecture: The depth encoder connects to ConvLSTM, which receives hidden and cell states from the preceding frame and produces multiscale disparity outputs through the decoder.The paper refers to the combined hidden and cell representation as the hidden state.
  • ConvLSTM training strategy: The initial ConvLSTM hidden state is learned in a first stage by backpropagating through it from zero initialization.This addresses the dominant effect of initialization caused by large feature maps and short training sequences.
  • Qualitative results: The self-supervised recurrent method produces more accurate KITTI Eigen depth maps than competing methods, especially for poles, tree trunks, and traffic signs.
  • ConvLSTM training strategy: The second training stage adjusts ConvLSTM weights using the pretrained hidden state, and testing uses both the learned initial state and updated weights.The authors report that this lets ConvLSTM adapt to sequences faster.

V. EXPERIMENTS

Experiments evaluate the recurrent framework across supervised depth prediction, self-supervised depth prediction, and self-supervised depth completion on KITTI. The recurrent method consistently improves over image-based counterparts, including across sparse-input patterns.

  • Evaluation setup: The experiments evaluate supervised depth prediction, self-supervised depth prediction, and self-supervised depth completion on KITTI.Monodepth2 is used as the baseline for the self-supervised depth estimation experiments.
  • Self-supervised depth prediction: The recurrent method improves self-supervised depth prediction over its baseline by -0.133m RMSE.The method uses longer-range temporal information than the baseline, while reflective surfaces and dynamic objects remain problematic.
  • Self-supervised depth completion: The recurrent framework outperforms its image-based counterpart for all three evaluated sparse depth patterns in self-supervised depth completion.It also outperforms the state-of-the-art self-supervised depth completion method.
  • Supervised depth prediction: The recurrent supervised model improves RMSE by -0.172m and δ1 by +1% relative to its baseline, but remains behind.Only marginal improvement is observed when training on corrected dense KITTI ground truth.
  • Sparse-input analysis: Performance improves with denser sparse-depth inputs, while the recurrent method consistently outperforms its frame-based counterpart.The RMSE improvement decreases as the number of input points increases, indicating reduced reliance on other sources when LiDAR supervision becomes stronger.

B. Ablation Study

The ablation study examines hidden-state initialization, sparse-input density, temporal consistency, and sequence length. Results favor a pretrained initial hidden state and recurrent processing, with benefits varying by input sparsity and sequence length.

  • Pretrained initial hidden state: A pretrained initial hidden state improves early-sequence adaptation and generalization over zero initialization.The benefit is especially visible at the beginning of sequences, although sequence 101 provides a counterexample before later improvement.
  • Sparse-input density: The recurrent method consistently outperforms its frame-based counterpart as sparse points or scanning lines vary.The improvement in RMSE decreases as the number of input points increases.
  • Temporal consistency: The recurrent method demonstrates better temporal consistency than the image-based method on KITTI video sequences.The comparison is reported in the temporal-consistency evaluation associated with Fig. 7.
  • Temporal consistency: For self-supervised depth estimation, recurrent processing reduces ARTE from 0.1401 to 0.1297 relative to the image-based baseline.The improvement is smaller for depth completion, where it is 0.005, consistent with greater benefit when fewer LiDAR points are available.
  • Sequence length: Training on sequences of 100 or more frames achieves similar results, while shorter sequences perform worse.This indicates that the ConvLSTM captures temporal information by propagating its hidden state through the sequence.

VI. CONCLUSION

The work estimates time series of depth maps from monocular video, optionally using sparse depth, by exploiting spatiotemporal structure for three depth-estimation tasks. In both self-supervised settings, the recurrent framework significantly outperforms current state-of-the-art methods.

  • The method estimates time series of depth maps from monocular video with optional sparse depth.
  • A recurrent framework exploits spatiotemporal structures during training and testing for supervised depth prediction, self-supervised depth prediction, and self-supervised depth completion.
  • In both self-supervised scenarios, the recurrent method significantly outperforms current state-of-the-art methods.
Loading 2001.02613v2…