Source-linked AI summary
DynamicStereo: Consistent Dynamic Depth from Stereo Videos
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, Christian Rupprecht
TL;DR
DynamicStereo addresses temporally inconsistent depth predictions from stereo video, a problem relevant to dynamic-scene reconstruction. It jointly processes stereo videos with transformer-based divided attention and introduces Dynamic Replica, whose realistic dynamic scenes improve predictions for DynamicStereo and prior methods. The method is more temporally consistent than previous work, but remains limited by window-scale oscillations and reliance on synthetic data.
Problem
Frame-independent stereo processing produces temporally inconsistent depth predictions, motivating methods that improve consistency in dynamic stereo video.
Method
DynamicStereo jointly processes stereo videos with transformer-based attention across space, time, and stereo, while Dynamic Replica provides synthetic videos of people and animals in realistic scanned environments.
Results
DynamicStereo improves temporally consistent stereo estimation, and training on Dynamic Replica also benefits prior methods.
Takeaways & Limitations
Dynamic Replica serves as training and evaluation data for dynamic stereo, while DynamicStereo combines spatial, temporal, and stereo information for precise and consistent predictions.
Takeaways & Limitations
Sliding-window inference causes low-frequency oscillations over 1–2 seconds, and larger windows are currently precluded by memory limitations.
Abstract
from arXiv · showhide
We consider the problem of reconstructing a dynamic scene observed from a stereo camera. Most existing methods for depth from stereo treat different stereo frames independently, leading to temporally inconsistent depth predictions. Temporal consistency is especially important for immersive AR or VR scenarios, where flickering greatly diminishes the user experience. We propose DynamicStereo, a novel transformer-based architecture to estimate disparity for stereo videos. The network learns to pool information from neighboring frames to improve the temporal consistency of its predictions. Our architecture is designed to process stereo videos efficiently through divided attention layers. We also introduce Dynamic Replica, a new benchmark dataset containing synthetic videos of people and animals in scanned environments, which provides complementary training and evaluation data for dynamic stereo closer to real applications than existing datasets. Training with this dataset further improves the quality of predictions of our proposed DynamicStereo as well as prior methods. Finally, it acts as a benchmark for consistent stereo methods.
1 Meta AI 2 Visual Geometry Group, University of Oxford
Figure 1 presents the paper’s two contributions: Dynamic Replica, a synthetic stereo-video dataset, and DynamicStereo, an efficient video-level stereo-matching method.
- Dynamic Replica supports training and benchmarking temporally consistent disparity estimators for dynamic scenes with people and animals.
- DynamicStereo uses transformer architectures to perform efficient stereo matching at the level of videos.
1. Introduction
The paper targets temporally consistent depth from stereo video, where frame-independent reconstruction overlooks shared information across time. It introduces DynamicStereo for efficient spatiotemporal stereo matching and Dynamic Replica, a realistic synthetic benchmark containing people and animals.
- Dynamic depth from stereo seeks temporally consistent video reconstruction because conventional frame-wise processing ignores observations of the same objects over time.
- Temporal matching can improve reconstruction by using information across space and time, including points occluded in one camera but visible at other times.
- DynamicStereo uses self- and cross-attention across space, time, and stereo pairs, with divided attention for efficient processing.
- Dynamic Replica contains 524 videos of virtual humans and animals embedded in realistic digital scans of physical environments.
- The paper presents DynamicStereo and Dynamic Replica as complementary contributions and reports state-of-the-art dynamic stereo results across varied benchmarks.
2. Related work
Related work spans traditional and learning-based stereo, depth-from-stereo datasets, attention-based matching, and video-depth methods. The paper positions Dynamic Replica as a large-scale dataset for non-rigid dynamic scenes and DynamicStereo as a video-level alternative to frame-wise or online approaches.
- Learning-based stereo methods evolved from CNN matching costs and 3D cost volumes toward more computationally efficient iterative architectures such as RAFT-Stereo and CRE-Stereo.
- Attention-based stereo methods improve feature matching, but earlier approaches focus on individual frames whereas DynamicStereo applies attention across time, space, and stereo frames.
- Monocular video-depth methods model temporal information under motion assumptions, often requiring video-specific fine-tuning, while CODD uses an online memory state without future-frame access.
- Existing depth-from-stereo datasets trade off scale, realism, annotations, and dynamic content, with several lacking video sequences or substantial non-rigid objects.
- Dynamic Replica addresses the lack of large-scale stereo datasets focused on non-rigid objects such as people and animals.
3. Method
DynamicStereo predicts disparity for stereo-video sequences by combining temporal, spatial, and stereo-view information through an efficient encoder–decoder architecture. Its transformer propagation and iterative decoder refine disparities from coarse to fine while maintaining information across dimensions.
- DynamicStereo learns sequence-level disparity predictions, allowing information to be fused across time rather than processing stereo frames independently.
- 3.1. Encoder: The encoder extracts multi-scale features, while a lowest-resolution SST transformer exchanges information across space, stereo views, and time.Divided attention is applied individually across the three dimensions and repeated four times, only at the lowest-resolution feature map.
- 3.2. Decoder: The decoder performs coarse-to-fine disparity recovery using correlation volumes, iterative correspondence refinement, and information exchange across space, views, and time.Correlation volumes are computed along epipolar lines at multiple feature resolutions and scales.
- 3.2. Decoder: At each iteration, the update block samples correlations around the current disparity, fuses them with disparity and features, applies space-time attention, and updates disparity with a 3D convolutional GRU.The update rule is D_hat^(m+1) = D_hat^(m) + g(D_hat^(m), phi).
4. Dynamic Replica: dynamic stereo dataset
Dynamic Replica is a synthetic stereo-video dataset designed for temporally consistent dynamic-depth learning and evaluation in realistic indoor environments. It contains animated humans and animals, longer test videos, and dense ground-truth annotations.
- Dynamic Replica provides stereo videos of animated humans and animals in virtual everyday environments, addressing the need for realistic training data for temporally consistent models.
- The dataset contains 524 videos: 484 training videos, 20 validation videos, and 20 test videos, with 300 frames in 10-second training and validation sequences.The test videos are 30 seconds long to evaluate models on longer sequences.
- Videos are rendered at 1280×720 resolution, higher than the resolutions of Sintel and SceneFlow.
- Each sample includes ground-truth depth, optical flow, foreground/background segmentation masks, and camera parameters for both stereo views.
5. Experiments
The experiments evaluate Dynamic Replica for disparity generalization, assess DynamicStereo’s temporal consistency, and ablate architectural choices. Training on both Dynamic Replica and SceneFlow improves performance, while temporal and multi-resolution processing strengthen consistency.
- 5.1. Dynamic Replica: Combining Dynamic Replica and SceneFlow boosts performance across datasets and disparity estimation models.Dynamic Replica alone improves its own test performance but generalizes less well than SceneFlow-trained models.
- 5.2. Temporal Consistency: DynamicStereo achieves lower temporal variance than the comparison model when reconstructing a 40-frame static video.The evaluation visualizes globally aligned point clouds from 40 frames and marks pixels with variance above 50px².
- 5.2. Temporal Consistency: DynamicStereo is more temporally consistent than prior methods, while training other methods on Dynamic Replica also improves temporal consistency.CRE-Stereo’s Sintel results are training-set results and are not directly comparable.
- 5.3. Ablation Studies: Learning separate update blocks for each decoder resolution improves results over sharing weights across resolutions.The ablation attributes the gain to different scales exploiting features at different resolutions.
- 5.3. Ablation Studies: Extending update-block convolutions across time improves overall performance, with especially large gains in temporal consistency.The comparison contrasts 3D temporal processing with the 2D convolutions used by methods operating on single time steps.
6. Conclusion
The paper contributes Dynamic Replica, a dataset for training temporally consistent disparity estimators, and DynamicStereo, a method that improves state-of-the-art temporal stereo estimation. Other methods also benefit from the dataset, while the model combines spatial, temporal, and stereo information.
- Dynamic Replica enables training temporally consistent disparity estimators, and DynamicStereo improves state-of-the-art temporally consistent stereo estimation using it.
- Other methods benefit from Dynamic Replica, while DynamicStereo combines spatial, temporal, and stereo information for precise and consistent predictions across time.Ablations report that each model component contributes to final performance.
A. Additional Ablations
Additional ablations examine attention choices in the SST block and update block. Attention generally improves both disparity accuracy and temporal consistency, with attention across space, stereo pairs, and time performing best in the SST block.
- SST Block Attention: Attention layers generally improve disparity accuracy and temporal consistency, with space, stereo-pair, and time attention achieving the best SST-block results.
B. Implementation details
The implementation samples balanced data and uses temporal positional encoding and augmentation for training. The architecture combines spatial and temporal attention with separable 3D GRUs, while runtime is measured per video frame.
- Training: Training samples equal numbers of frames or sequences from Dynamic Replica and SceneFlow for generalization and temporal-consistency experiments.
- Implementation details: Learnable temporal positional encoding, Fourier spatial features, saturation changes, rectification stretching, and simulated occlusions augment training.
- Attention and recurrent processing: The model applies linear spatial self-attention, spatial cross-attention between stereo views, and standard temporal attention.
- Recurrent processing and upsampling: Each 3D GRU module uses separable height-width-time kernels, while upsampling combines RAFT-style convex upsampling with bilinear upsampling.
- Runtime: Runtime analysis reports the average seconds required to process one 1280x720 video frame on a GPU.
C. Limitations
DynamicStereo improves temporal consistency but remains limited by sliding-window oscillations, memory constraints, untextured regions, and reliance on synthetic data for dense supervision.
- Sliding-window processing causes low-frequency oscillations over 1–2 seconds, and memory limitations prevent extending the window size.
- Large untextured surfaces remain difficult to predict accurately, although Dynamic Replica provides priors that partially mitigate this limitation.
- Because dense ground truth is nearly impossible to collect, training and evaluation rely on synthetic datasets, limiting real-world assessment to qualitative comparisons.