PaperlayerView exact revision on arXiv

Source-linked AI summary

DynStream: Online Streaming 4D Gaussian Reconstruction of Dynamic Worlds from Unposed Video

Dingwei Xian, Xiaoyu Zhou, Yajiao Xiong, Yongtao Wang, Ming-Hsuan Yang

arXiv:2610.09720v1cs.CV

TL;DR

Long, unposed video streams challenge systems to combine continuous reconstruction with photorealistic rendering. DynStream incrementally reconstructs local temporal windows and aligns them into a globally consistent 4D Gaussian scene without per-scene optimization. The paper reports state-of-the-art reconstruction quality across diverse dynamic scenes.

  • Problem

    Long, unposed streams require continuous processing and photorealistic rendering, while existing approaches struggle to provide both together.

    Source: §1 · p. 2

  • Method

    DynStream reconstructs local temporal windows and incrementally aligns chunks using dynamic-aware relative poses to build a globally consistent 4D Gaussian scene.

    Source: §1 · pp. 2–3

  • Results

    The authors report state-of-the-art reconstruction quality across diverse dynamic scenes and competitive rendering quality for online reconstruction from long video streams.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    DynStream is presented as a scalable approach to persistent online reconstruction of dynamic environments.

    Source: §5 · p. 10

    Metric-scale loss is applied only to training samples with metric-scale supervision and is disabled for non-metric datasets.

    Source: §3.4 · p. 7

Abstract

from arXiv · show

Online reconstruction of dynamic 4D scenes from long, unposed streaming videos requires both continuous processing and photorealistic rendering, which existing methods struggle to achieve simultaneously. Existing feed-forward Gaussian methods are restricted to offline processing, whereas online point-cloud approaches struggle to maintain dense geometry and high-fidelity rendering. We present DynStream, a framework for streaming 4D Gaussian reconstruction from long, unposed videos. Given a continuous video stream, DynStream reconstructs the scene within local temporal windows and incrementally aligns and fuses these local reconstructions into a globally consistent scene, enabling online 4D reconstruction without per-scene optimization. By jointly enforcing cross-window geometric consistency and modeling time-varying scene content, DynStream supports efficient reconstruction and photorealistic rendering over extended video streams. Experiments demonstrate that DynStream enables high-fidelity online dynamic reconstruction and rendering from long video streams, achieving state-of-the-art performance across diverse dynamic indoor and outdoor scenes.

1 INTRODUCTION

DynStream targets long, unposed video streams, where online systems must balance temporal consistency, dynamic scene capture, and rendering quality. It incrementally reconstructs and aligns local chunks into a globally consistent 4D Gaussian scene without per-scene optimization.

1 INTRODUCTION

  • Long, unposed streams require incremental integration that preserves geometric consistency and high-fidelity rendering while capturing changing scene content efficiently.The introduction identifies these combined requirements as the challenge for continuous 4D reconstruction.Source: §1 · p. 2
  • Existing optimization-based methods incur sequence-growing computation and may accumulate tracking and reconstruction errors, while feed-forward methods have bounded context or fidelity and supervision constraints.These limitations make both paradigms ill-suited to long, continuously arriving, unposed streams.Source: §1 · p. 2
  • DynStream reconstructs long, unposed streams as globally consistent 4D Gaussian scenes without per-scene optimization or 3D/pose supervision.It jointly estimates camera motion and time-varying scene representations.Source: §1 · pp. 2–3
  • A dynamic-aware relative-pose strategy propagates camera and scene coordinates across successive chunks, avoiding independent chunk registration and per-scene optimization.Causal attention predicts poses and newly observed Gaussian primitives using past and current frames; chunks use a keyframe and metric-scale alignment.Source: §1 · pp. 2–3
  • DynStream achieves competitive rendering quality while enabling online dynamic reconstruction from long video streams.The introduction reports this outcome across diverse scenes.Source: §1 · p. 3

2 RELATED WORK

Prior online reconstruction often relies on iterative tracking and scene-specific optimization, while feed-forward methods have extended geometric prediction toward streaming. DynStream combines causal feed-forward reconstruction and Gaussian prediction to incrementally align chunks into a shared renderable scene.

2 RELATED WORK

  • Earlier online dynamic reconstruction couples scene integration with correspondence estimation, motion tracking, and iterative optimization, often using explicit depth observations (Yu et al., 2018).More recent Gaussian-based methods still require repeated scene-specific computation as observations arrive (Li et al., 2026).Source: §2 · p. 3
  • Feed-forward methods estimate geometry and motion from visual observations, but their outputs primarily describe geometric quantities rather than a continuously maintained, photorealistic, incrementally updated scene (Wang et al., 2024; Zhang et al., 2025; Wang et al., 2025).Related work also describes feed-forward Gaussian methods for dynamic scenes and causal streaming (Ye et al., 2025; Jiang et al., 2025; Ye et al., 2026; Yang et al., 2025; Chen et al., 2026; Yang et al., 2026; Qiu et al., 2026).Source: §2 · p. 3
  • Existing methods do not continuously build a globally aligned, renderable scene from long, unposed streams when they operate on clips or require known camera parameters.DynStream instead propagates camera and scene coordinates across temporal chunks without reprocessing past frames or per-scene optimization.Source: §2 · p. 3

3 METHOD

DynStream processes an incoming stream in bounded temporal chunks, reconstructing local scene content and incrementally integrating it into a coherent global representation. Its causal reconstruction and dynamic-aware alignment modules support online processing while preserving time-varying content.

3.1 PROBLEM FORMULATION

  • Fixed-length chunks bound the temporal context, with only arrived frames visible; causal local reconstruction and global integration update a coherent scene over time.The current chunk is reconstructed using its visible frames and retained reference information, then aligned with the existing scene state.Source: §3.1 · p. 4

3.2 CAUSAL-ATTENTION ONLINE RECONSTRUCTION

  • A causal-attention module predicts keyframe-relative poses, depth, and pixel-aligned Gaussians, with a metric scale head aligning length-related predictions across chunks.The causal mask restricts each frame to observations available up to that time.Source: §3.2 · pp. 4–5, Figure 2 · p. 5

3.3 RELATIVE-POSE GLOBAL ALIGNMENT AND DYNAMIC-AWARE INTEGRATION

  • At chunk transitions, relative-pose predictions propagate keyframe poses globally, while scale and coordinate transforms align local Gaussians for integration.Static structure accumulates across chunks, while dynamic Gaussians retain their temporal associations rather than being indiscriminately accumulated.Source: §3.3 · p. 6

3.4 LOSS FUNCTION

  • Training uses objectives for appearance, geometry, camera motion, metric scale, and dynamic decomposition; rendering supervision uses only the causal scene state available at time t.Metric-scale loss is disabled for datasets without metric-scale supervision.Source: §3.4 · pp. 6–7
  • Table 1 averages metrics over 14 sampled frames per scene and weights scenes equally; Table 2 reports sequence-level macro-averaged trajectory errors over 11 KITTI sequences.The KITTI benchmark comprises 23,201 frames; lower trajectory errors are better.Source: Table 1 · p. 7, Table 2 · p. 8

4 EXPERIMENTS

DynStream is evaluated on diverse indoor and outdoor dynamic scenes for rendering quality, camera pose, qualitative fidelity, and component ablations. The reported results favor its rendering and pose estimates, while ablations support the roles of dynamic modeling, relative-pose estimation, and metric-scale prediction.

4.1 EXPERIMENTAL SETUP

  • The model is trained on real and synthetic indoor and outdoor datasets and uses VGGT initialization for its geometry transformer, depth head, and relative-pose head.Training uses AdamW, BF16 mixed precision, gradient clipping, and 2–14 context views with varied image heights and weighted losses.Source: §4.1 · p. 8

4.2 QUANTITATIVE RESULTS

  • DynStream achieves state-of-the-art rendering quality on indoor and outdoor dynamic-scene benchmarks across PSNR, SSIM, and LPIPS-Alex.The evaluation compares methods on the same input sequences without image enhancement or post-processing.Source: §4.2 · p. 8
  • On KITTI odometry sequences 00–10, DynStream performs better camera pose estimation than the previous online reconstruction method on long sequences.The evaluation covers 23,201 frames, with the longest sequence spanning 4,661 frames and 5.068 km.Source: §4.2 · p. 9

4.3 QUALITATIVE RESULTS

  • Qualitatively, DynStream produces cleaner geometry and more faithful appearance than static methods with ghosting or dynamic methods with motion-segmentation artifacts.Figure 4 compares scenes by column and ground truth or baseline methods by row.Source: Figure 4 · p. 8, §4.3 · p. 9

4.4 ABLATION STUDY

  • Removing dynamic-aware scene modeling degrades rendering quality on both Waymo and SpatialVID, consistent with preventing dynamic observations from accumulating into the persistent scene.Table 3 reports the ablation across the two datasets.Source: §4.4 · p. 9, Table 3 · p. 9
  • Removing metric-scale prediction notably increases SE(3)-aligned ATE, especially on VKITTI2, while removing relative-pose prediction also degrades trajectory accuracy.Table 4 reports Sim(3)- and SE(3)-aligned ATE on KITTI and VKITTI2.Source: §4.4 · pp. 9–10, Table 4 · p. 9

5 CONCLUSION

DynStream incrementally aligns local temporal reconstructions into a persistent global representation, with state-of-the-art reconstruction quality reported across diverse dynamic scenes. The section also records the use of generative AI for language polishing and privacy, licensing, and anonymization considerations for public artifacts.

5 CONCLUSION

  • DynStream reconstructs long, unposed video streams by aligning local temporal windows into a persistent global representation without per-scene optimization.The conclusion reports state-of-the-art reconstruction quality across diverse dynamic scenes.Source: §5 · p. 10

AI USE STATEMENT

  • Generative AI tools were used to polish language and improve manuscript clarity and readability.Source: AI USE STATEMENT · p. 11

ETHICS STATEMENT

  • Public artifacts require privacy, licensing, and anonymization review because reconstruction can expose people, locations, and private environments.Local possession of data or checkpoints does not grant redistribution rights.Source: ETHICS STATEMENT · p. 11

A DATASETS AND IMPLEMENTATION DETAILS

DynStream is trained on a mix of real and synthetic indoor and outdoor datasets spanning varied dynamics, scales, and camera motion. The appendix also specifies initialization, optimization, input-view sampling, image sizes, and loss weights.

A DATASETS AND IMPLEMENTATION DETAILS

  • Training uses SpatialVID, Waymo Open Dataset, PointOdyssey, DL3DV-10K, HM3D, Replica, ETH3D, Matterport3D, and TartanAir.The datasets cover real and synthetic indoor and outdoor environments with diverse camera motion, scene scale, and dynamics.Source: A DATASETS AND IMPLEMENTATION DETAILS · p. 14, Table I · p. 14
  • The geometry transformer, DPT depth head, and relative-pose head are initialized from VGGT, while remaining prediction layers are initialized randomly.Training uses AdamW with β = (0.9, 0.95), weight decay 0.05, BF16 mixed precision, gradient clipping at 0.5, and warm-up followed by cosine decay.Source: A DATASETS AND IMPLEMENTATION DETAILS · p. 14
  • Training samples 2–14 context views with probability proportional to squared view count, varies image height from 224 to 448 in multiples of 16, and applies metric-scale supervision only when annotations are available.The loss weights are λ1 = 1, λ2 = 10, λ3 = 1, λ4 = 0.1, λ5 = 0.05, and λ6 = 0.1.Source: A DATASETS AND IMPLEMENTATION DETAILS · p. 14

B ADDITIONAL EVALUATION PROTOCOLS

The evaluation protocols specify rendering and camera-pose metrics, matched input sequences, and incremental processing for long sequences. Camera-pose evaluation includes both scale-aligned and raw errors, while the long-sequence setting maintains prior scene content and poses.

B ADDITIONAL EVALUATION PROTOCOLS

  • Rendering quality is measured with PSNR, SSIM, and LPIPS-Alex on the same input sequences for each benchmark, without image enhancement or post-processing.Source: B ADDITIONAL EVALUATION PROTOCOLS · p. 14
  • KITTI odometry evaluation reports Sim(3)-aligned ATE, SE(3)-aligned ATE, raw ATE, translational RPE, and rotational RPE across sequences 00–10.Sim(3) alignment removes global scale; SE(3) alignment preserves scale error and thus reflects metric-scale estimation quality.Source: B ADDITIONAL EVALUATION PROTOCOLS · p. 14
  • All long-sequence inputs are processed incrementally using the current bounded chunk, while previously reconstructed scene content and camera poses remain in the global scene state.Source: B ADDITIONAL EVALUATION PROTOCOLS · p. 14

C ADDITIONAL ANALYSIS OF DYNAMIC SEGMENTATION

The dynamic-segmentation analysis compares predicted probabilities with SAM3-generated foreground masks on Waymo and SpatialVID, while the efficiency analysis evaluates long-stream processing. DynStream reports stronger mask agreement and remains runnable on a 100-chunk stream where competing feed-forward methods fail.

C ADDITIONAL ANALYSIS OF DYNAMIC SEGMENTATION

  • SAM3 masks represent semantic foreground, not physical motion: parked vehicles may be included, while moving vegetation or water may be excluded.Because training labels use the same source, the experiment tests agreement with a shared foreground reference rather than independent zero-shot motion segmentation.Source: C ADDITIONAL ANALYSIS OF DYNAMIC… · p. 15
  • The evaluation covers 100 scenes each from Waymo and SpatialVID, with 14 uniformly sampled 224 × 448 frames per scene and no ground-truth cameras or extra sky masks.Scenes are selected before inference using a fixed random seed and foreground amount, without performance-based filtering.Source: C ADDITIONAL ANALYSIS OF DYNAMIC… · p. 15
  • For SpatialVID, the validation split comes from the current backbone loader, and possible overlap from earlier preprocessing changes cannot be completely excluded.A historical fixed training-scene list is unavailable.Source: C ADDITIONAL ANALYSIS OF DYNAMIC… · p. 15
  • Coverage is TP/(TP + FN), precision is TP/(TP + FP), and AP uses continuous dynamic scores without a binary threshold.Metrics are computed after aggregating each scene's 14 frames, then averaged across scenes.Source: C ADDITIONAL ANALYSIS OF DYNAMIC… · p. 15
  • DynStream obtains substantially higher IoU, Dice, and AP on both datasets, with high foreground coverage useful for avoiding dynamic Gaussians entering the persistent scene.Table II reports consistency with SAM3-generated foreground reference masks.Source: C ADDITIONAL ANALYSIS OF DYNAMIC… · p. 15, Table II · p. 15
  • The reconstruction pipeline uses pdyn > 0.05, while compatible probability outputs are also compared at 0.5 because methods' native dynamic signals and thresholds differ.Table III presents dynamic segmentation using the common probability threshold of 0.5.Source: C ADDITIONAL ANALYSIS OF DYNAMIC… · p. 15, Table III · p. 15

D MODEL EFFICIENCY

  • 315.5 ms per frame at 100 chunks; competing feed-forward methods fail on the 100-chunk stream.DynStream processes new observations incrementally without reprocessing previously reconstructed content.Source: D MODEL EFFICIENCY · p. 16, Table IV · p. 16

E ADDITIONAL QUALITATIVE RESULTS

Additional qualitative results cover driving scenes and diverse indoor and outdoor environments. DynStream better preserves persistent backgrounds while retaining time-specific foreground content.

E ADDITIONAL QUALITATIVE RESULTS

  • Across driving and indoor/outdoor scenes, DynStream better preserves persistent background geometry while maintaining time-specific foreground content.Static reconstruction methods often show ghosting or duplicated geometry around moving objects, and imperfect dynamic separation can leave residual artifacts.Source: E ADDITIONAL QUALITATIVE RESULTS · p. 16

F DETAILED ABLATION STUDIES

The ablations examine temporal aggregation, cross-chunk pose and scale consistency, and dynamic-aware integration. They test how removing each component affects online information exchange or scene reconstruction.

F DETAILED ABLATION STUDIES

  • The ablations cover causal temporal aggregation, keyframe-relative pose prediction, metric scale estimation, and dynamic-aware scene integration.These are the main components tested in the detailed ablation studies.Source: F DETAILED ABLATION STUDIES · p. 16
  • Causal-attention ablation tests temporal information exchange under online constraints, while relative-pose and scale ablations test cross-chunk camera and geometry consistency.Removing dynamic-aware integration tests whether treating all Gaussians as persistent static content introduces artifacts in time-varying regions.Source: F DETAILED ABLATION STUDIES · p. 16

G FAILURE CASES AND LIMITATIONS

DynStream has limitations in long-trajectory pose and scale propagation, dynamic-probability-dependent integration, and accumulated scene memory. The paper also notes privacy considerations for releasing reconstructed scenes or derived data.

G FAILURE CASES AND LIMITATIONS

  • Propagating relative transformations across chunks can accumulate pose and scale errors over very long trajectories without global bundle adjustment or loop closure.Global camera poses are obtained by that propagation.Source: G FAILURE CASES AND LIMITATIONS · p. 16
  • Dynamic-probability errors can mis-fuse moving content into the persistent scene or prevent static geometry from being accumulated; SAM3 supervision emphasizes selected semantic foreground categories, not arbitrary motion.False negatives and false positives create the respective integration errors.Source: G FAILURE CASES AND LIMITATIONS · p. 16
  • Bounded temporal context does not bound accumulated Gaussian-scene size, so memory can keep increasing on very long sequences with large unexplored regions or extensive dynamic content.Source: G FAILURE CASES AND LIMITATIONS · p. 16

H BROADER IMPACTS

  • Reconstructed driving and indoor imagery may contain people, vehicles, locations, or private spaces; releases should respect dataset licenses, privacy requirements, and anonymization procedures.The stated intended use is scene understanding and mapping rather than personal identification.Source: H BROADER IMPACTS · p. 17
Loading 2610.09720v1…