PaperlayerView exact revision on arXiv

Source-linked AI summary

DynamicStereo: Consistent Dynamic Depth from Stereo Videos

Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, Christian Rupprecht

arXiv:2305.02296v1cs.CVcs.AI

TL;DR

Dynamic depth from stereo needs temporally consistent predictions, including when stereo reconstruction supplies 3D content for virtual or mixed reality. DynamicStereo jointly processes stereo videos with a transformer and introduces Dynamic Replica; using the dataset, it improves over the state of the art in temporally consistent stereo estimation.

  • Problem

    Dynamic depth from stereo requires temporally consistent predictions, while stereo reconstruction can supply 3D content for virtual or mixed reality.

    Source: §1 · p. 1, §6 · p. 8

  • Method

    DynamicStereo jointly processes stereo videos with a transformer, alongside Dynamic Replica, a benchmark dataset for learning and evaluating dynamic depth from stereo.

    Source: §1 · p. 2

  • Results

    DynamicStereo improves over the state of the art in temporally consistent stereo estimation using Dynamic Replica, and other methods benefit from training on it.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Dynamic Replica's realistic scenes can reduce the domain gap between real-world scenes and typical synthetic, abstract training datasets.

    Source: §6 · p. 8

    Despite greater temporal consistency than prior work, the method remains unstable over time; sliding-window evaluation causes 1–2 second low-frequency oscillations, and memory limits prevent longer windows.

    Source: C. Limitations · p. 12

Abstract

from arXiv · show

We consider the problem of reconstructing a dynamic scene observed from a stereo camera. Most existing methods for depth from stereo treat different stereo frames independently, leading to temporally inconsistent depth predictions. Temporal consistency is especially important for immersive AR or VR scenarios, where flickering greatly diminishes the user experience. We propose DynamicStereo, a novel transformer-based architecture to estimate disparity for stereo videos. The network learns to pool information from neighboring frames to improve the temporal consistency of its predictions. Our architecture is designed to process stereo videos efficiently through divided attention layers. We also introduce Dynamic Replica, a new benchmark dataset containing synthetic videos of people and animals in scanned environments, which provides complementary training and evaluation data for dynamic stereo closer to real applications than existing datasets. Training with this dataset further improves the quality of predictions of our proposed DynamicStereo as well as prior methods. Finally, it acts as a benchmark for consistent stereo methods.

1. Introduction

DynamicStereo targets temporally consistent depth from stereo video by using information across space, time, and stereo pairs. The paper also introduces Dynamic Replica, a dataset of realistic synthetic videos of people and animals for learning and evaluating dynamic stereo.

Introduction

  • Stereo video methods that reconstruct frames independently can produce inconsistent depth, despite observations being correlated across time even when cameras move or scenes deform.Stereo disparity matches are triangulated into 3D, but framewise reconstruction ignores that the frames depict the same underlying objects.Source: §1 · pp. 1–2
  • DynamicStereo uses transformer self- and cross-attention across space, time, and stereo pairs, with divided attention for efficient processing.Looking across space and time can improve matching, for example when points occluded in one camera become visible from both cameras at other times.Source: §1 · p. 2
  • Dynamic Replica contains 524 videos of virtual humans and animals in realistic digital scans, and training on it can boost dynamic stereo method quality over existing datasets.The dataset is designed to provide realistic dynamic scenes for learning and evaluation, complementing existing synthetic datasets with abstract scenes and datasets without video or focused on static scenes.Source: §1 · p. 2
  • The paper contributes DynamicStereo, Dynamic Replica as a learning and evaluation benchmark, and state-of-the-art dynamic stereo results across a variety of benchmarks.Source: §1 · p. 2

2. Related work

Prior work addresses stereo matching, video depth consistency, and training data, but DynamicStereo focuses on learning temporally consistent disparity directly from stereo videos. The section also introduces Dynamic Replica as a large-scale semi-realistic synthetic dataset focused on non-rigid objects.

Depth from stereo

  • Stereo methods progressed from local matching costs and energy-based optimization to learned cost volumes [7] [12] [14] [51] [53] [54], which can generalize poorly beyond training data.Earlier learning-based work used CNNs for matching costs [52] and introduced fully learning-based stereo estimation [27].Source: §2 · p. 2
  • Efficient methods include RAFT-Stereo [23], which iteratively updates an epipolar-line cost volume, and CRE-Stereo [18]; DynamicStereo fuses low- and high-resolution features during coarse-to-fine refinement.Unlike CRE-Stereo, DynamicStereo propagates fused features to the final prediction so high-resolution layers can use them and track previous states.Source: §2 · p. 2

Datasets for learning depth from stereo

  • Dynamic Replica is larger in resolution and frame count than previous datasets and provides non-rigid people and animals unavailable at scale in prior datasets.The authors describe it as the first large-scale semi-realistic synthetic dataset focused on non-rigid objects for disparity estimation.Source: Table 1 · p. 3, §2 · p. 3

Depth from stereo

  • Prior attention-based stereo methods [19] [42] [36] [18] estimate disparity for individual frames, whereas DynamicStereo applies attention across time, space, and stereo frames.These methods use attention for dense pixel matching or to improve convolutional features.Source: §2 · pp. 2–3

Dynamic video depth

  • Monocular video depth methods use motion parallax or information fused across frames; CVD [25], RCVD [16], and DVD [55] address motion differently but require per-video fine-tuning.CVD assumes near-rigid motion, RCVD excludes dynamic objects from its objective, and DVD models non-rigid motion with scene flow.Source: §2 · p. 3

Dynamic depth from stereo

  • Unlike CODD, which uses separately trained stereo, motion, and fusion networks with online memory, DynamicStereo learns consistency from data without memory and supports online or offline use.CODD assumes no access to future frames, preventing global optimization over the sequence.Source: §2 · p. 3

3. Method

DynamicStereo estimates disparity from stereo-video sequences rather than frames independently, allowing it to fuse information over time. Its encoder and iterative decoder propagate spatial, stereo-view, and temporal information to produce disparity maps.

3. Method

  • The method predicts disparity for a sequence of stereo frames, enabling temporal information fusion that can improve prediction consistency.Unlike frame-wise methods, it learns a sequence-to-sequence mapping from stereo video to disparity maps aligned with the left frames.Source: §3 · p. 3

3.1. Encoder

  • Divided Space-Stereo-Time attention exchanges information across dimensions individually, and is applied four times only to the lowest-resolution encoder features.The encoder extracts features independently from each frame before the attention blocks propagate information across space, stereo views, and time.Source: §3.1 · p. 4

3.2. Decoder

  • The decoder refines disparity from coarse to fine through iterative correspondence updates, using epipolar-line correlations and a 3D convolutional GRU to propagate spatial and temporal information.The GRU receives fused features and a hidden state initialized from the reference feature map; each update iteration follows correlation lookup and modality fusion.Source: §3.2 · pp. 4–5

3.3. Training

  • Training supervises predictions across all iterations and timestamps, weighting later disparity estimates more heavily.The loss uses exponentially increasing weights toward the final estimate, with γ = 0.9.Source: §3.3 · p. 5

3.4. Inference

  • The model is trained on stereo sequences of fixed length T, and the inference description introduces a test-time procedure intended to increase temporal coherence.The supplied inference passage ends before specifying that procedure.Source: §3.4 · p. 5

4. Dynamic Replica: dynamic stereo dataset

Dynamic Replica is a synthetic stereo-video dataset and benchmark of animated humans and animals in everyday scenes, designed to provide more realistic sequences than SceneFlow’s short clips of randomly textured objects [26].

4. Dynamic Replica: dynamic stereo dataset

  • Dynamic Replica contains 524 videos: 484 training and 20 validation clips of 300 frames each, plus 20 test videos lasting 30 seconds.Training and validation videos are 10 seconds long; the longer test videos are intended to benchmark models on extended sequences.Source: §4 · p. 6
  • Videos are rendered at 1280×720, exceeding the resolutions of Sintel (1024×536) and SceneFlow (960×540).Source: §4 · p. 6
  • The dataset uses Facebook Replica [40] indoor reconstructions, 375 scanned human models, and smooth virtual-camera trajectories mimicking mobile-phone or AR-glasses filming.The human models are animated using motion-capture sequences from real human scans.Source: §4 · p. 6
  • All samples include ground-truth depth, optical flow, foreground/background masks, and camera parameters for both stereo views.Source: §4 · p. 6

5. Experiments

Experiments show that Dynamic Replica benefits stereo training and that DynamicStereo improves temporal consistency; ablations support separate, time-aware update blocks.

5.1. Dynamic Replica

  • Combining Dynamic Replica with SceneFlow boosts generalization across datasets and models, while Dynamic Replica alone improves its own test set but generalizes less well than SceneFlow.This supports including Dynamic Replica in standard disparity-estimation training, despite its primary aim of enabling temporally consistent models on longer sequences.Source: Table 3 · p. 7, §5.1 · p. 7

5.2. Temporal Consistency

  • DynamicStereo is more temporally consistent than prior methods across training datasets and benchmarks, outperforming even CRE-Stereo [18] trained on seven datasets.On a real-world sequence, its reconstruction also shows less flickering than RAFT-Stereo [23].Source: §5.2 · p. 7, Table 4 · p. 7, Table 5 · p. 8

5.3. Ablation Studies

  • Learning separate update blocks for each decoder resolution improves results over sharing weights across resolutions.The authors attribute this to different scales exploiting features at different resolutions.Source: Table 6 · p. 8, §5.3 · p. 8
  • Extending the update block with a 3D convolution across space and time improves performance over the 2D variant, especially temporal consistency.CRE-Stereo results in the Sintel ablation are from the training set and cannot be directly compared.Source: Table 7 · p. 8, §5.3 · p. 8

6. Conclusion

DynamicStereo introduces the Dynamic Replica dataset for training temporally consistent disparity estimators and a method that improves state-of-the-art temporally consistent stereo estimation using the dataset.

6. Conclusion

  • DynamicStereo improves state-of-the-art temporally consistent stereo estimation using the new Dynamic Replica dataset.The method combines spatial, temporal, and stereo information for precise, consistent predictions across time.Source: §6 · p. 8
  • Other methods benefit from training on Dynamic Replica, whose realistic scenes can reduce the domain gap from typical synthetic, abstract datasets such as SceneFlow.The paper reports that each model component contributes to final performance in extensive ablation studies.Source: §6 · p. 8

A. Additional Ablations

The ablations show that attention improves disparity estimation and temporal consistency, with space, stereo, and time attention best for the SST block and space-time attention performing well in the update block.

SST Block Attention

  • SST block attention across space, stereo pairs, and time achieves the best results, while attention layers generally improve disparity accuracy and temporal consistency.Time attention also improves accuracy, potentially by using multiple viewpoints over time to locate correspondences more precisely.Source: A. Additional Ablations · p. 12

Update Block Attention

  • The update block’s combination of space and time attention performs well on both datasets and improves both stereo and temporal metrics.Source: A. Additional Ablations · p. 12

Additional Ablations

B. Implementation details

The implementation details specify balanced dataset sampling, training encodings and augmentations, chunked inference, and efficient attention, recurrent, and upsampling components.

B. Implementation details

  • For dataset-generalization experiments, the same number of frames is sampled from DR and SF; temporal-consistency experiments instead sample the same number of sequences.Source: B. Implementation details · p. 12
  • Inference processes videos in 20-frame chunks with 10-frame overlap, discarding the first and last 5 predicted frames from each chunk.This procedure is used to improve temporal consistency when composing the final disparity sequence.Source: B. Implementation details · p. 12
  • Training uses learnable temporal encodings and spatial sin/cos Fourier features, plus saturation jitter, right-frame stretching, and rectangular erasures with probability 0.5, following.Saturation is sampled uniformly from 0 to 1.4; right-frame stretching simulates imperfect rectification.Source: B. Implementation details · p. 12
  • Space-Stereo-Time attention applies spatial self-attention to each view, cross-attention between views across space, and standard attention across time.Time and position encodings are added before reshaping the feature tensors for attention.Source: B. Implementation details · p. 12
  • Each 3D CNN-based GRU module comprises three separable height-width-time GRUs with kernel sizes (1 × 1 × 5), (5 × 1 × 1), and (1 × 5 × 5).Source: B. Implementation details · p. 12
  • Upsampling passes each update block’s output to a higher-resolution block using convex upsampling from RAFT combined with standard bilinear upsampling.Source: B. Implementation details · p. 12

C. Limitations

DynamicStereo improves temporal consistency but remains limited by window-scale oscillations, difficult textureless surfaces, and reliance on synthetic data for evaluation and training.

C. Limitations

  • Sliding-window evaluation causes low-frequency oscillations over 1–2 seconds, and memory limits currently prevent extending the window size.The method is more temporally consistent than previous work but is not fully stable over time.Source: C. Limitations · p. 12
  • Exceedingly large untextured surfaces, such as walls, remain difficult to predict accurately despite DynamicReplica priors.Training on DynamicReplica mitigates but does not resolve this stereo-matching difficulty.Source: C. Limitations · p. 12
  • Because dense ground truth is difficult to collect, training and evaluation rely on synthetic datasets, limiting assessment of real-world generalization to qualitative results.Those results might not fully reflect performance on artificial scenes.Source: C. Limitations · p. 12

Related papers

Loading 2305.02296v1…