PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploiting temporal consistency for real-time video depth estimation

Haokui Zhang, Chunhua Shen, Ying Li, Yuanzhouhan Cao, Yu Liu, Youliang Yan

arXiv:1908.03706v1cs.CV

TL;DR

Static-image depth methods can be computationally costly, while video contains temporal information that existing methods do not use. This paper proposes a video-depth framework combining ST-CLSTM with an adversarial temporal loss, and reports real-time operation and strong results on indoor and outdoor benchmarks.

  • Problem

    Static-image depth methods can be computationally costly, while existing methods do not use the temporal information available in video sequences.

    Source: §1 · p. 1

  • Method

    The framework uses ST-CLSTM to capture spatial features and temporal correlations, and combines spatial loss with an adversarial temporal consistency loss.

    Source: §1 · p. 1, §2.2.2 · p. 5

  • Results

    The authors report that the framework outperforms most state-of-the-art methods and achieves outstanding performance on indoor and outdoor benchmarks while running in real time.

    Source: §3.7 · p. 8, §4 · p. 8

  • Takeaways & Limitations

    The authors conclude that the framework runs in real time and can be generalized to most existing depth-estimation frameworks.

    Source: §4 · p. 8

    The evaluation uses a shallow ResNet18 backbone, and the authors note that deeper backbones could improve performance.

    Source: §3.7 · p. 8

Abstract

from arXiv · show

Accuracy of depth estimation from static images has been significantly improved recently, by exploiting hierarchical features from deep convolutional neural networks (CNNs). Compared with static images, vast information exists among video frames and can be exploited to improve the depth estimation performance. In this work, we focus on exploring temporal information from monocular videos for depth estimation. Specifically, we take the advantage of convolutional long short-term memory (CLSTM) and propose a novel spatial-temporal CSLTM (ST-CLSTM) structure. Our ST-CLSTM structure can capture not only the spatial features but also the temporal correlations/consistency among consecutive video frames with negligible increase in computational cost. Additionally, in order to maintain the temporal consistency among the estimated depth frames, we apply the generative adversarial learning scheme and design a temporal consistency loss. The temporal consistency loss is combined with the spatial loss to update the model in an end-to-end fashion. By taking advantage of the temporal information, we build a video depth estimation framework that runs in real-time and generates visually pleasant results. Moreover, our approach is flexible and can be generalized to most existing depth estimation frameworks. Code is available at: https://tinyurl.com/STCLSTM

1. Introduction

Static-image depth methods can be computationally costly, while existing approaches do not use temporal information in video. The paper combines ST-CLSTM with an adversarial temporal loss and situates this approach among prior video-depth, CLSTM, and GAN work.

1. Introduction

  • Static-image depth networks can be computationally heavy, and existing methods do not account for the temporal information available in video.The introduction notes that directly extending static-image methods to video is not feasible because of excessive computational cost.Source: §1 · p. 1
  • ST-CLSTM combines spatial feature extraction with temporal-correlation collection to estimate depth from video frames.The CLSTM uses cues from a previous frame to reason about the current frame.Source: §1 · pp. 1–2
  • An adversarial temporal loss uses a 3D CNN discriminator on estimated and ground-truth depth sequences, and is combined with spatial loss for end-to-end training.The paper presents the temporal loss as a way to further enforce consistency among video frames.Source: §1 · p. 2, §1.1 · p. 2, Figure 1 · p. 3
  • The proposed framework is reported to run in real time and to generalize to most existing depth-estimation frameworks.The authors also describe the temporal loss as improving video depth-estimation performance.Source: §1 · p. 2

1.1. Related work

  • Prior video-depth approaches used bundle adjustment with super-resolution or 3D geometric loss to estimate depth, with the latter also estimating ego-motion.The paper instead proposes estimating depth by exploiting temporal information from video.Source: §1.1 · p. 2
  • Prior CLSTM work addressed video tasks such as saliency and skeleton processing, while GANs had also been applied to structured-output learning and depth estimation.The cited examples include CLSTM methods for video saliency and GAN-based depth-map generation.Source: §1.1 · p. 2

2. Our Method

The method combines spatial feature extraction with a convolutional recurrent module to estimate video depth, then trains with spatial and adversarial temporal-consistency losses. Its feature extractor is replaceable, and the framework is designed for real-time processing.

2.1. ST-CLSTM

  • The framework combines spatial feature extraction, temporal correlation collection, and spatial-temporal loss calculation in the ST-CLSTM approach.The ST-CLSTM output is an estimated depth frame.Source: §2.1 · p. 3, §2.2 · p. 4

2.1.1 Spatial feature extraction network

  • The spatial network uses an encoder, decoder, and multi-scale feature fusion, with a shallow ResNet-18 encoder chosen for fast processing.Its decoder uses four up-projection modules, while MFF integrates features at different scales.Source: Figure 2 · p. 3, §2.1.1 · p. 3
  • The spatial feature extractor can be replaced by other depth-estimation models, allowing the framework to be applied with minimal modification.Source: §2.1.1 · pp. 3–4

2.1.2 CLSTM

  • The CLSTM replaces LSTM's Hadamard product with convolution and compresses previous-frame features to eight channels before concatenating them with current-frame features.The updated memory and output-gate features are refined by two convolution layers to produce the depth estimate.Source: §2.1.2 · p. 4

2.2.1 Spatial loss

  • The spatial loss combines depth, gradient, and surface-normal terms to penalize depth errors, edge errors, and differences in surface orientation.The depth term uses a logarithm L1 loss; the gradient term targets errors around edges, and the normal term is sensitive to small depth structures.Source: §2.2.1 · pp. 4–5

2.2.2 Temporal loss

  • A 3D CNN discriminator compares estimated and ground-truth depth sequences, and its cross-entropy loss serves as the temporal consistency loss.The temporal loss is combined with the spatial loss; the discriminator receives concatenated RGB and depth frames, with some generated sequences mixing in ground-truth depth frames.Source: §2.2.2 · p. 5
  • The adversarial training is intended to enforce temporal consistency rather than improve the accuracy of an individual frame's depth.Source: §2.2.2 · p. 5

3. Experiments

The experiments evaluate the framework on indoor NYU Depth V2 and outdoor KITTI video data, using train/test splits based on Eigen et al..

3. Experiments

  • The evaluation covers indoor NYU Depth V2 and outdoor KITTI, comparing the proposed framework with existing depth-estimation approaches.Source: §3 · p. 5

3.1. Datasets

  • NYU Depth V2 uses 249 training videos and 654 test samples from the remaining 215 videos, following Eigen et al..Training images are resized to 320×240 and cropped to 304 × 228 patches.Source: §3.1 · p. 6
  • KITTI uses 32 training videos and 697 test samples from the remaining 29 videos, following Eigen et al..Training inputs are randomly cropped to 480×320 patches from the original images.Source: §3.1 · p. 6

3.2. Evaluation metrics

The evaluation uses spatial depth metrics and temporal metrics that assess whether estimated changes and motions across adjacent frames match the ground truth.

3.2. Evaluation metrics

  • Spatial evaluation includes mean relative error (Rel), root mean squared error (RMS), mean log10 error, and accuracy under thresholds δ < 1.25^i.SSIM is identified as structural similarity.Source: §3.2 · p. 6
  • Temporal consistency means matching estimated changes and motions across adjacent frames to those in the corresponding ground truths.Source: §3.2 · p. 6
  • Temporal change consistency (TCC) and temporal motion consistency (TMC) are introduced to evaluate temporal consistency quantitatively.The evaluation refers to estimated depth maps from consecutive frames and their corresponding ground truths.Source: §3.2 · p. 6

3.3. Implementation details

The framework is trained for 20 epochs, with separate initialization and learning-rate settings for its ST-CLSTM and 3D CNN components.

3.3. Implementation details

  • The ST-CLSTM is trained for 20 epochs from a 0.0001 learning rate, reduced by a factor of 0.1 every five epochs, using an ImageNet-pretrained spatial feature extractor.Source: §3.3 · p. 6
  • The 3D CNN is randomly initialized, with initial learning rates of 0.1 for NYU Depth V2 and 0.01 for KITTI.Before adversarial updates, the ST-CLSTM is trained alone for one NYU epoch or two KITTI epochs to produce plausible depth estimates.Source: §3.3 · p. 6
  • Training augmentation flips images and depth maps horizontally with 50% probability, rotates them by a random angle from −5° to 5°, and varies image brightness, contrast, and saturation by ratios from 0.6 to 1.4.The augmentation methods follow.Source: §3.3 · p. 6

3.4. Benefit of ST-CLSTM

ST-CLSTM improves depth-estimation performance on indoor and outdoor datasets, while results on NYU Depth V2 improve as more input frames are used.

3.4. Benefit of ST-CLSTM

  • On NYU Depth V2, performance increases as the number of input frames increases, consistent with ST-CLSTM capturing more temporal information.Source: §3.4 · p. 6
  • The experiments use five input frames as a compromise that considers computation cost.Source: §3.4 · p. 6
  • The authors report that ST-CLSTM captures temporal information and improves depth-estimation performance on both indoor and outdoor datasets, with results reported in Table 1.Table 1’s first four rows report NYU Depth V2 results, and its last two rows report KITTI results.Source: §3.4 · p. 6, Table 1 · p. 6

3.5. Benefit of generative adversarial learning

The authors evaluate adversarial learning on NYU Depth V2 and KITTI, reporting that the temporal loss enforces consistency and further improves ST-CLSTM performance.

3.5. Benefit of generative adversarial learning

  • The generative adversarial learning scheme and temporal loss are reported to enforce consistency among video frames and further improve ST-CLSTM performance.Table 2 reports results for NYU Depth V2 and KITTI.Source: §3.5 · p. 7, Table 2 · p. 7
  • The evaluation compares ST-CLSTM without adversarial learning against the GAN variant on both NYU Depth V2 and KITTI.Source: §3.5 · p. 7
  • The evaluated scheme is intended to further enforce temporal consistency among video frames.Source: §3.5 · p. 7

3.6. Improvement of temporal consistency

The visual comparisons show improved temporal consistency from ST-CLSTM, with the generative adversarial scheme further improving the results. The 3D-CNN discriminator also improves depth-estimation accuracy and temporal consistency in the reported test sequences.

3.6. Improvement of temporal consistency

  • ST-CLSTM+GAN exhibits better temporal consistency than 2DCNN and ST-CLSTM in the five-frame NYU Depth V2 visual results.The zoomed regions show smoother backgrounds and sharper object edges with ST-CLSTM, with further improvement from the generative adversarial scheme.Source: Figure 5 · p. 7, §3.6 · p. 7
  • Figure 5 compares RGB inputs, ground-truth depths, baseline and proposed estimates, with zoomed regions shown in its bottom rows.The sequence contains four consecutive frames with one frame between each displayed frame.Source: Figure 5 · p. 7, §3.6 · p. 7
  • The 3D-CNN discriminator improves estimation accuracy and temporal consistency on 654 test sequences, each 16 frames long.The reported averages are TCC and TMC; the supplied passages do not include their values.Source: §3.6 · p. 7, Table 3 · p. 8

3.7. Comparisons with state-of-the-art results

On NYU Depth V2 and KITTI, the framework outperforms most state-of-the-art methods, many of which use more complicated network structures. The evaluation uses a shallow ResNet18 backbone; deeper backbones are left for future work.

3.7. Comparisons with state-of-the-art results

  • The framework outperforms most state-of-the-art methods on NYU Depth V2 and KITTI, despite many comparison methods using more complicated network structures.The approach uses a shallow ResNet18 backbone, and the authors note that deeper backbones could improve its performance.Source: §3.7 · p. 8
  • The evaluation uses a shallow ResNet18 backbone, while evaluating deeper backbones is left as future work.Source: §3.7 · p. 8

3.8. Speed analysis

The model's measured processing speed is sufficient for real-time use at the tested resolution. Parallel-plus-serial data loading increases processing speed relative to serial loading.

3.8. Speed analysis

  • The authors report that ST-CLSTM is sufficiently fast for real-time use, with negligible processing-speed drop versus the 2D CNN baseline and a dramatic increase when using PS-mode.Inference used 304 × 228 videos, averaging 600 frames over five epochs; hardware included 8GB RAM, an Intel i7-4790 CPU, and a GTX1080Ti GPU.Source: §3.8 · p. 8
  • Parallel-plus-serial data loading increases processing speed dramatically compared with serial loading.The method feeds 120 frames to the spatial extraction network at once, then processes their spatial features sequentially with CLSTM.Source: §3.8 · p. 8

4. Conclusion

The conclusion combines a shallow 2D CNN with CLSTM to capture spatial features and temporal correlations, and uses adversarial learning for temporal loss. Experiments on indoor and outdoor benchmarks show effective temporal-information capture, strong performance, and real-time execution.

4. Conclusion

  • ST-CLSTM combines a shallow 2D CNN and CLSTM to capture spatial features and temporal correlations among video frames.Source: §4 · p. 8
  • The adversarial-learning temporal loss further enforces consistency among video-frame depth estimates.Source: §4 · p. 8
  • Experiments on indoor and outdoor datasets show effective temporal-information capture, strong performance, and real-time execution.The framework is also described as generalizable to most existing depth-estimation frameworks.Source: §4 · p. 8

Related papers

Loading 1908.03706v1…