PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepVideoMVS: Multi-View Stereo on Video with Recurrent Spatio-Temporal Fusion

Arda Düzçeker, Silvano Galliani, Christoph Vogel, Pablo Speciale, Mihai Dusmanu, Marc Pollefeys

arXiv:2012.02177v3cs.CVcs.LG

TL;DR

Dense depth recovery from posed video calls for an online multi-view method that can use temporal information while remaining lightweight. The paper adds perspective-corrected ConvLSTM fusion to a cost-volume stereo backbone; it reports outperforming existing methods on most evaluated metrics while maintaining real-time performance.

  • Problem

    The paper targets dense depth recovery for posed-video streams in an online, lightweight multi-view stereo setting.

    Source: §1 · p. 1

  • Method

    A lightweight cost-volume stereo network uses ConvLSTM to propagate past latent geometry, warping the prior hidden state toward the current viewpoint using previous depth.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    The approach outperforms existing state-of-the-art methods on most evaluation metrics, generalizes across considered test sets, and runs significantly faster than competitors with low memory consumption.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The proposed fusion improves depth predictions while adding only small computational overhead and memory consumption.

    Source: §1 · p. 2

    Direct fusion assumes the hidden state and current encoding are purpose-wise similar, yet pose-induced image motion can be difficult to capture with near-object disparities, occlusions, and rapid rotations.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

We propose an online multi-view depth prediction approach on posed video streams, where the scene geometry information computed in the previous time steps is propagated to the current time step in an efficient and geometrically plausible way. The backbone of our approach is a real-time capable, lightweight encoder-decoder that relies on cost volumes computed from pairs of images. We extend it by placing a ConvLSTM cell at the bottleneck layer, which compresses an arbitrary amount of past information in its states. The novelty lies in propagating the hidden state of the cell by accounting for the viewpoint changes between time steps. At a given time step, we warp the previous hidden state into the current camera plane using the previous depth prediction. Our extension brings only a small overhead of computation time and memory consumption, while improving the depth predictions significantly. As a result, we outperform the existing state-of-the-art multi-view stereo methods on most of the evaluated metrics in hundreds of indoor scenes while maintaining a real-time performance. Code available: https://github.com/ardaduz/deep-video-mvs

1. Introduction

The paper targets dense depth recovery for posed video streams in an online, lightweight multi-view stereo setting. It propagates past scene geometry through a ConvLSTM while correcting latent representations for viewpoint changes, and reports improved depth accuracy with small computational overhead.

1. Introduction

  • Passive camera-based depth sensing is described as energy- and cost-efficient, compact, and usable across a wide range of conditions.The paper contrasts it with active sensing and notes that stereo devices can require baselines infeasible for mobile devices.Source: §1 · p. 1
  • The system targets dense depth recovery for each keyframe in posed video, where calibrated cameras and known poses support metric reconstruction.The intended setting is online multi-view stereo with limited viewpoint variation between consecutive frames.Source: §1 · p. 1
  • The approach propagates past scene geometry into current predictions through ConvLSTM hidden states, explicitly accounting for perspective projection and viewpoint changes.The authors report that this produces improved and consistent depth outputs.Source: §1 · p. 2
  • The authors report state-of-the-art results on ScanNet, 7-Scenes, TUM RGBD, and ICL-NUIM, alongside the lowest runtime and a small memory footprint.The paper also describes its stereo backbone as compact, cost-volume-based, and reliant solely on 2D convolutions.Source: §1 · p. 2

2. Related Work

Related work spans depth-map MVS architectures and temporal depth estimation from video. This paper positions its approach as lightweight cost-volume MVS with perspective-corrected latent fusion, distinct from prior temporal models and probability-volume aggregation.

Depth Map Prediction with Learned MVS

  • Depth maps are presented as versatile, relatively memory-efficient representations that can support tasks beyond 3D reconstruction and deliver real-time information.Other reviewed approaches infer in 3D space, use point-based or implicit scene representations, or regress TSDFs.Source: §2 · p. 2
  • Learning-based MVS commonly uses plane-swept cost volumes, with 3D-convolution regularization offering high accuracy at substantial computational cost, while 2D-convolution alternatives better suit real-time use.The proposed backbone follows the lightweight 2D-convolution direction.Source: §2 · p. 2

Depth Estimation from Video

  • Prior video-depth methods model temporal relations with ConvLSTMs or frame constraints, but monocular methods lack geometric correctness based on image measurements.Known poses in MVS enable triangulated metric measurements and transfer of past scene-geometry encodings into the current view.Source: §2 · p. 2
  • GP-MVS uses pose-similarity priors for soft constraints and online state-space inference, whereas this approach fuses latent encodings with learned filters after perspective correction.Neural RGBD aggregates depth-probability volumes over time; this paper instead fuses bottleneck latent encodings earlier in the pipeline.Source: §2 · pp. 2–3

3. Method

The method predicts depth from posed video using a lightweight cost-volume pair network and a ConvLSTM fusion module at the encoder-decoder bottleneck. It warps the prior hidden state toward the current viewpoint before fusion, using previous depth to guide the transformation.

3. Method

  • The predictor learns to infer a depth map from current and previous images, their camera poses, and camera intrinsics by minimizing loss against ground-truth depth.The input includes the current frame and a number of previous time steps.Source: §3 · p. 3

3.1. Pair Network

  • The pair network uses a feature pyramid, a plane-sweep cost volume, encoder-decoder regularization, and depth regression at multiple resolutions.Its cost volume uses 64 inverse-depth-spaced hypotheses over 0.25–20 meters; skip connections bring feature maps into the encoder.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2. Spatio-temporal Fusion

  • A ConvLSTM cell at the encoder-decoder bottleneck models spatio-temporal relations using the encoder output and recurrent hidden and cell states.The authors report that ELU performed better than tanh, Layer Normalization stabilized the model, and ConvGRU performed worse despite lower resource cost.Source: §3.2 · pp. 3–4
  • Directly fusing latent encodings from different viewpoints may force the ConvLSTM to learn pose-induced image motion, which can be difficult with near-object disparities, occlusions, and rapid rotations.This concern relies on the assumption that the hidden state and current encoding are purpose-wise similar latent representations.Source: §3.2 · p. 4
  • The proposed fusion warps the prior hidden representation toward the current viewpoint using poses and a partial current-depth estimate derived from the previous depth prediction.Inverse mapping with bilinear grid sampling is described as differentiable and lightweight; downsampling to the bottleneck relaxes the need for perfect warping.Source: §3.2 · pp. 4–5
  • Training begins by using ground-truth depth for warping, then switches late to the testing strategy while fine-tuning only the ConvLSTM cell.The stated purpose is to guide the cell with accurate warpings before gradually adapting it to the testing configuration.Source: §3.2 · p. 5

4. Experiments

The models are trained on ScanNet and tested on ScanNet and sequences from four other datasets. The experiments compare depth quality, runtime, memory, frame selection, and temporal-fusion ablations.

4.1. Dataset

  • Training uses ScanNet’s official split; testing includes its 100-sequence test split and 38 sequences from four other datasets, totaling around 31K images.The fusion network is trained with subsequences of length 8; Table 1 specifies the test-set datasets and measurement-frame setup.Source: §4.1 · p. 5, §4 · p. 5, Table 1 · p. 6

4.3. Comparison with Existing Methods

  • 20.3% improvement in absolute inverse depth error over finetuned GP-MVS; fusion outperforms existing methods on 70% of metrics and generalizes across test sets.The comparison covers ScanNet, 7-Scenes, RGB-D Scenes V2, TUM RGB-D, and Augmented ICL-NUIM.Source: Table 1 · p. 6, §4.3 · p. 7

4.2. Frame Selection

  • Pose-distance view selection balances triangulation quality, matching accuracy, and view-frustum overlap, using indoor-scene thresholds of 0.35 ± 0.05 pose-distance and 10 cm ± 5 cm translation.At test time, the system buffers the last 30 keyframes, adds a frame when pose-distance exceeds 0.1, and prefers relative camera distances of 15 cm when ranking measurement frames.Source: §4.2 · p. 6

4.4. Ablation Studies

  • Additional measurement frames tend to improve performance but gains can stagnate; at three views, abs-inv error increases for MVDepthNet and GPMVS.The comparison uses the proposed view-selection heuristic.Source: Table 2 · p. 7, §4.4 · p. 7

4.3. Comparison with Existing Methods

  • On ScanNet, the methods are fastest and most accurate, with fusion delivering unmatched prediction quality at a high frame rate.Figure 4 relates absolute error to runtime and memory consumption; exact measurements are in the supplementary material.Source: Figure 4 · p. 7, §4.3 · p. 7

4.4. Ablation Studies

  • A simple view-selection heuristic improves predictions by about 10% on average across methods, which also remain robust when every 10th or 20th frame is sampled.The heuristic affects all evaluated MVS methods in a similar manner.Source: Table 3 · p. 7, §4.4 · p. 8

4.3. Comparison with Existing Methods

  • Fusion appears closest to ground truth in qualitative examples, and its more consistent depths yield less noisy TSDF reconstructions with flatter walls and preserved geometry.The TSDF examples are from ScanNet and 7-Scenes.Source: §4.3 · p. 7, Figure 5 · p. 8, Figure 6 · p. 8

4.4. Ablation Studies

  • Naive fusion improves performance by about 10% over the pair network on average, and the proposed hidden-state propagation delivers a similar additional gain.The ablation averages performance over all test sequences.Source: Table 4 · p. 7, §4.4 · p. 7

5. Conclusion

The paper presents an online multi-view stereo system for posed video that estimates depth maps in real time. It uses a lightweight stereo backbone with a memory cell that fuses information over time and transforms its hidden state to account for viewpoint changes.

5. Conclusion

  • The system predicts depth maps from posed-video streams and exploits their temporal structure for online multi-view stereo.Source: §5 · p. 8
  • The approach outperforms existing state-of-the-art methods on most evaluation metrics, generalizes to all considered test sets, and achieves faster inference with low memory consumption.It is designed for online operation while estimating depth maps in real time.Source: §5 · p. 8
  • A memory cell aggregates bottleneck encodings over time, and the ConvLSTM hidden state is transformed during propagation to address viewpoint dependence.Source: §5 · p. 8

Supplementary Material for DeepVideoMVS: Multi-View Stereo on Video

The supplied passage identifies supplementary material accompanying DeepVideoMVS: Multi-View Stereo on Video.

Supplementary Material for DeepVideoMVS: Multi-View Stereo on Video

A. Training Details

Training uses cropped and resized images, separate pair-network and fusion-network training stages, and several image, sequence, and geometric augmentations.

A. Training Details

  • ScanNet images are cropped from 640×480 to 480×480 and downscaled to 256×256; pair-network training computes a reference-frame cost volume using one measurement frame.Source: A. Training Details · p. 11
  • The pair network uses Adam with a 1e−4 learning rate for 600K iterations, freezing pretrained MnasNet layers for the first 200K iterations.Training uses mini-batches of 14 and predicts depth for both input images by alternating their reference and measurement roles.Source: A. Training Details · p. 11
  • Fusion training progresses from a frozen-backbone stage to joint training and then cell-only finetuning, using groundtruth depth for early warping and predicted depth for finetuning.The network is trained with subsequences of length 8; gradients are stopped through the depth prediction used for hidden-state warping.Source: A. Training Details · p. 11
  • Augmentations change image color, reverse subsequence order, vary geometric scale, and horizontally flip cost volumes and extracted features during pair-network training.Scale bounds are adjusted to keep a correct local cost-volume extremum for each pixel after scaling.Source: A. Training Details · p. 11

B. Evaluation Metrics

The passage introduces metric calculations and defines the predicted and groundtruth depth values, valid-pixel count, and indicator function.

B. Evaluation Metrics

  • For metric calculations, d-hat_i and d_i are predicted and groundtruth depth at valid pixel i, N counts valid pixels, and 1 denotes the indicator function.Source: B. Evaluation Metrics · p. 11

C. Inference Time and Memory Consumption

The section describes how inference time and GPU memory were measured across methods and reports a caveat about reproducing DELTAS timing.

C. Inference Time and Memory Consumption

  • Inference measurements used 320×256 inputs, batch size 1, 300 forward passes, and an NVIDIA GTX 1080Ti workstation.Timing excluded CPU time and data transfer; measurements began after 100 warm-start passes, with CPU and GPU temperatures below 40°C.Source: C. Inference Time and Memory Consumption · p. 11, Table 5 · p. 12
  • DELTAS timing used a custom singular-value-decomposition function suggested by its authors to increase inference speed.The authors could not reproduce the roughly 90-millisecond inference time reported for DELTAS on an NVIDIA Titan RTX.Source: C. Inference Time and Memory Consumption · p. 12
  • The comparison reports mean inference time and GPU memory consumption; Neural RGBD used at least two measurement frames while other methods used one.Source: Table 5 · p. 12

D. Evaluation Set Details

Evaluation used four real datasets and one synthetic dataset, each providing RGB-D video and 3D camera poses. The section specifies the selected test sequences for each dataset.

D. Evaluation Set Details

  • The five datasets provide RGB-D videos at 640 × 480 depth resolution and 3D camera poses.Source: D. Evaluation Set Details · p. 12
  • All 100 official ScanNet test sequences, scene0707 through scene0806, were selected.Source: D. Evaluation Set Details · p. 12
  • The selected 7-Scenes sequences were chess, fire, head, office, pumpkin, redkitchen, and stairs scenes.The listed sequence IDs are chess-01/02, fire-01/02, head-02, office-01/03, pumpkin-03/06, redkitchen-01/02, and stairs-02/06.Source: D. Evaluation Set Details · p. 12
  • The evaluation also selected RGB-D Scenes V2, TUM RGB-D SLAM, and Augmented ICL-NUIM sequences.RGB-D Scenes V2 includes scenes 01, 02, 05, 06, 09, 10, 13, and 14; the selected TUM and ICL-NUIM sequence IDs are listed in the paper.Source: D. Evaluation Set Details · p. 12

E. Additional Ablation Studies

Additional ablations compare fusion with increased plane-sweep resolution, recurrent-cell choices, skip connections, and ConvLSTM configurations. The studies also identify stability issues and training conditions that qualify those comparisons.

Additional Ablation Studies

  • These ablations used the ScanNet validation split, and results across subsections are not meaningfully comparable because training strategies may differ.Models within a given subsection used identical training and data pipelines.Source: E. Additional Ablation Studies · p. 12

Activation and Normalization in the ConvLSTM Cell

  • Configuration-5 gave the best depth prediction performance while maintaining stable behavior among the tested activation and normalization options.The authors report that removing layer normalization before the sigmoid gates improved predictions by avoiding an unnecessary constraint on gate values.Source: Figure 8 · p. 12, E. Additional Ablation Studies · p. 13, Table 6 · p. 13
  • Naive ELU activation without normalization could amplify hidden-state noise over time and destabilize predictions, including on long sequences with pose loops.The reported noise test introduced positive additive noise; the loop test used validation sequences containing pose loops. The authors attribute these issues to ELU’s asymmetric, positively unbounded range.Source: E. Additional Ablation Studies · p. 13, Figure 7 · p. 13

ConvLSTM vs. ConvGRU

  • 5.6% lower abs-inv error favored ConvLSTM over ConvGRU when their hidden states were warped during propagation.The authors speculate that warping ConvGRU’s sole hidden state loses information from non-overlapping view frustums, while ConvLSTM has two internal states and warps only the hidden state.Source: E. Additional Ablation Studies · p. 14, Table 7 · p. 14

Finetuning the Cell

Skip Connections from Feature Pyramid to Encoder

  • Multi-skip design was around 4.5% better in abs-inv than single-skip design in the fusion network, despite similar pair-network results.The authors conjecture that the additional skip connections help training by easing the loss landscape in the presence of ConvLSTM.Source: E. Additional Ablation Studies · p. 14, Table 9 · p. 14

Accuracy vs. Time Trade-off of the Fusion

  • 20.4% improvement in abs-inv came from fusion, versus 1.2% from increasing plane-sweep resolution on the ScanNet validation split.The comparison retrained the pair network with M=88 sweep planes to compensate for fusion’s similar inference-time overhead.Source: E. Additional Ablation Studies · p. 14, Table 10 · p. 14

F. Additional Qualitative Results and Supplementary Video

The additional examples show depth predictions from ScanNet, 7-Scenes, and RGB-D Scenes V2. The authors report that fusion makes depth values more coherent across frames and produces less flicker and smoother reconstructions of planar surfaces.

F. Additional Qualitative Results and Supplementary Video

  • Fusion produces less flickering depth predictions and greater geometric consistency than the pair network and existing methods, with planar surfaces reconstructed more smoothly across a sequence.The authors describe this pattern in the additional examples and supplementary video.Source: F. Additional Qualitative Results and… · p. 14
  • The additional depth-prediction examples cover ScanNet, 7-Scenes, and RGB-D Scenes V2.The ScanNet examples are also shown separately.Source: Figure 9 · p. 15, Figure 10 · p. 16
Loading 2012.02177v3…