PaperlayerView exact revision on arXiv

Source-linked AI summary

Stream4D: Streaming 4D Reconstruction with Training-free Motion Decoupling via Causal KV-Cache Attention

Wei Zhang, Ying Feng, Mengtong Wu, Shiqiang Gong, Huaqi Zhang, Qi Zhang

arXiv:2610.31320v1cs.CV

TL;DR

Existing streaming reconstruction commonly assumes static scenes, although dynamic objects can disrupt pose estimation and accumulate as long-sequence ghosting. Stream4D uses cached attention cues, depth priors, and soft key masking to decouple motion in causal backbones; experiments report improved pose accuracy, reliable masks, and reduced ghosting.

  • Problem

    Existing streaming reconstruction methods commonly assume static scenes, while dynamic objects can mislead pose estimation and accumulate into ghosting over long sequences.

    Source: §1 · p. 2, §3.1 · p. 3

  • Method

    Stream4D aggregates motion cues from causal attention KV caches, fuses attention saliency with depth priors to form dynamic masks, and applies soft key masking.

    Source: §1 · p. 2, §3.2 · p. 3, §3.3 · p. 4

  • Results

    Across dynamic-scene benchmarks, Stream4D improves pose accuracy, produces reliable dynamic masks, and mitigates long-term ghosting.

    Source: §1 · p. 2, §4.1 · p. 7, §4.2 · p. 8

  • Takeaways & Limitations

    Soft key suppression keeps static backgrounds stable and sharp while retaining dynamic objects only in the current frame, even over long sequences.

    Source: §4.2 · p. 8

Abstract

from arXiv · show

4D scene reconstruction requires robust handling of dynamic objects, yet existing streaming methods commonly assume static scenes, leading to pose drift and ghosting artifacts over long sequences. We introduce Stream4D, a training-free, motion-decoupled streaming reconstruction framework that can be directly integrated into causal Transformer backbones (e.g., StreamVGGT, Stream3R). Our key observation is that causal attention implicitly encodes rich temporal motion cues within the KV cache. We adaptively aggregate these cues from multiple attention layers and across temporal windows, and fuse them with depth priors to handle challenging cases such as occlusions and textureless regions, producing precise dynamic masks that decouple static and dynamic regions. We then apply a soft key masking scheme that leverages these dynamic masks to suppress interference from dynamic regions in causal attention, thereby improving camera pose estimation accuracy. Stream4D inherits the efficiency of causal attention and remains compatible with existing streaming architectures. On diverse dynamic-scene benchmarks, Stream4D significantly improves camera pose estimation and static-scene reconstruction quality, yields reliable dynamic masks, and effectively mitigates long-sequence ghosting.

1 Introduction

Stream4D addresses dynamic-scene interference in causal streaming reconstruction with a training-free framework that decouples motion and improves reconstruction outcomes. Its design aggregates motion cues from cached attention and uses dynamic masks to suppress interference.

  • Causal streaming methods use KV-cache historical memory to reduce computation and storage on long sequences, but commonly assume static scenes.Source: §1 · p. 2
  • Dynamic objects disrupt photometric consistency and epipolar geometry, mislead pose estimation, and accumulate into ghost artifacts; prior dynamic-segmentation methods are incompatible with causal streaming or limited in temporal consistency.Source: §1 · p. 2
  • Stream4D is a training-free, pluggable motion-decoupled framework compatible with causal backbones such as StreamVGGT [59] and Stream3R [9].Source: §1 · p. 2
  • Across dynamic-scene benchmarks, Stream4D improves pose accuracy, produces reliable dynamic masks, and mitigates long-term ghosting without additional training or fine-tuning.Source: §1 · p. 2
  • It adaptively aggregates temporal motion cues encoded in the KV cache across multiple attention layers and temporal windows.Source: §1 · p. 2
  • Its dynamic-mask pipeline fuses attention saliency with depth priors to handle occlusions and textureless regions, then soft key masking suppresses dynamic regions in cached attention.Source: §1 · p. 2

2 Related Work

Prior reconstruction methods trade off global interaction, incremental processing, and scalability, while most 4D approaches remain poorly suited to causal streaming. Stream4D targets that gap with training-free motion decoupling through selective KV-cache updates.

  • Global-attention methods such as Fast3R [40] and VGGT [23] require reprocessing growing sequences, while StreamVGGT [59] and Stream3R [9] use causal KV-cache inference for long-sequence efficiency.Source: §2 · p. 2
  • Many dynamic-scene pipelines require strong priors, extensive test-time optimization, fine-tuning, or post-processing, and training-free variants are often backbone-specific.Source: §2 · p. 3
  • Most existing dynamic-scene methods rely on global bidirectional architectures, leaving them incompatible with causal streaming and long-sequence accumulation of dynamic information.Source: §2 · p. 3
  • Stream4D is presented as a training-free framework for causal streaming that decouples motion through selective KV-cache updates and supports online long-sequence reconstruction.Source: §2 · p. 3

3 Method

Stream4D estimates dynamic-object masks, camera motion, and a static point cloud through causal streaming reconstruction. It aggregates motion cues from cached attention, fuses them with depth priors, and uses masks to reduce dynamic interference during reconstruction.

3 Method

  • Stream4D estimates dynamic-object masks, camera motion, and a static scene point cloud in a pipeline organized around causal streaming, dynamic segmentation, and motion-decoupled reconstruction.The static point cloud is updated incrementally with high-confidence static points after outlier filtering.Source: §3 · p. 3, §3.3 · p. 4

3.1 Streaming 4D Geometry Reconstruction

  • Causal attention restricts each frame to past frames, while the KV Cache stores historical key-value pairs for incremental online reconstruction.The cache is updated with each frame’s keys and values, avoiding redundant computation.Source: §3.1 · p. 3, Figure 2 · p. 3

3.2 Streaming Dynamic Segmentation

  • Dynamic saliency aggregates attention statistics across layers and adaptive historical frames, then fuses attention cues with depth and temporal-change measures to produce binary masks.Depth priors are included to address occlusions and textureless regions; temporal smoothing, Otsu thresholding, and morphological refinement produce the masks.Source: §3.2 · pp. 3–4

3.3 Motion-Decoupled Streaming Reconstruction

  • Dynamic masks guide soft key masking in the KV Cache, reducing dynamic interference during historical aggregation and improving camera pose estimation accuracy.Photometric and geometric consistency losses are restricted to mask-identified static regions.Source: §3.3 · p. 4

4 Experiments

Experiments assess causal, single-pass performance across dynamic segmentation, DyCheck reconstruction and pose estimation, and long-sequence efficiency. Stream4D is compared with streaming and offline baselines on DAVIS and DyCheck, with scalability tested on sequences of 10–200 frames.

4.2 4D Reconstruction

  • Table 2 presents 4D reconstruction results on DyCheck, reporting pose metrics ATE, RTE, and RRE, where lower values are better, and classifies methods as offline or single-pass causal streaming.Source: Table 2 · p. 6

4.1 Dynamic Object Segmentation

  • Against StreamVGGT, Stream4D gains +8.48 (43.39 vs. 34.91), +9.12 (46.45 vs. 37.33), and +9.86 (43.06 vs. 33.20) mean IoU (J𝑀) on DAVIS-2016, DAVIS-2017, and DAVIS-all, respectively.Its segmentation maintains stable masks and complete boundaries across slow and fast motions.Source: §4.1 · p. 7

4.2 4D Reconstruction

  • Soft key masking suppresses dynamic regions in cached keys, limiting repeated fusion that creates trajectory replicas and can corrupt nearby static surfaces.Stream4D retains dynamic objects in the current frame while keeping the background stable and sharp over long sequences.Source: §4.2 · p. 8

4.3 Efficiency and Scalability

  • At N = 200, Stream4D takes 93.1s versus StreamVGGT’s 84.1s, while Easi3R reaches 260.0s at N = 100 and fails before N = 200.Stream4D computation grows near-linearly; its latency overhead over StreamVGGT is approximately 10%.Source: §4.3 · p. 8, Figure 8 · p. 8

5 Conclusion

Stream4D uses causal-attention motion cues, depth priors, and soft key masking for training-free motion-decoupled reconstruction in causal streaming. Experiments report competitive or better segmentation and pose estimation alongside efficient long-sequence inference; future work targets richer cues and dynamic-object reconstruction.

  • Stream4D aggregates multi-level causal attention cues with depth priors to derive dynamic masks, then applies soft key masking to suppress dynamic features during historical aggregation.The framework operates under single-pass causal inference constraints.Source: §5 · p. 8
  • On DAVIS and DyCheck, Stream4D achieves competitive or superior streaming-baseline performance in dynamic segmentation and pose estimation while maintaining efficient long-sequence inference.Source: §5 · p. 8
  • Current aggregation relies primarily on attention statistics and depth priors; future work will explore cross-scale temporal patterns, semantic-aware saliency, and dynamic-object reconstruction.Source: §5 · p. 8
Loading 2610.31320v1…