Source-linked AI summary
LongStream: Long-Sequence Streaming Autoregressive Visual Geometry
Chong Cheng, Xianda Chen, Tao Xie, Wei Yin, Weiqiang Ren, Qian Zhang, Xiaoyang Guo, Hao Wang
TL;DR
Long-sequence streaming 3D reconstruction remains difficult because existing models can fail under first-frame anchoring, long-horizon cache effects, and scale drift. LongStream decouples pose coordinates and metric scale, aligns cache use during training and inference, and reports stable metric-scale reconstruction at 18 FPS across kilometer-scale sequences. Its scope remains bounded by static-world assumptions, heuristic keyframe scheduling, and the absence of explicit loop-closure optimization.
Problem
The paper addresses streaming 3D reconstruction that must remain stable over long sequences despite first-frame anchoring, attention degradation, scale drift, and extrapolation errors.
Method
LongStream uses keyframe-relative poses, orthogonal scale learning, cache-consistent training, and periodic cache refresh in a gauge-decoupled streaming framework.
Results
LongStream achieves state-of-the-art streaming reconstruction across indoor and outdoor benchmarks, maintaining stable metric-scale reconstruction at 18 FPS over kilometer-scale sequences.
Takeaways & Limitations
The reported design supports strictly online reconstruction with stable pose, geometry, and scale over ultra-long sequences and diverse environments.
Takeaways & Limitations
The method assumes a largely static world, uses a heuristic keyframe schedule, and does not perform explicit loop-closure optimization.
Abstract
from arXiv · showhide
Long-sequence streaming 3D reconstruction remains a significant open challenge. Existing autoregressive models often fail when processing long sequences because they anchor poses to the first frame, leading to attention decay, scale drift, and extrapolation errors. We introduce LongStream, a novel gauge-decoupled streaming visual geometry model for metric-scale scene reconstruction across thousands of frames under a strictly online, future-invisible setting. Our approach is threefold. First, we discard the first-frame anchor and predict keyframe-relative poses. This reformulates long-range extrapolation into a constant-difficulty local task. Second, we introduce orthogonal scale learning. This method fully disentangles geometry from scale estimation to suppress drift. Finally, we identify attention bias issues in Transformers, including attention-sink reliance and long-term KV-cache saturation. We propose cache-consistent training combined with periodic cache refresh. This approach suppresses attention biases and contamination over ultra-long sequences and reduces the gap between training and inference. Experiments show that LongStream achieves state-of-the-art performance, enabling stable, metric-scale reconstruction over kilometer-scale sequences at 18 FPS. Project Page: https://3dagentworld.github.io/longstream/
1. Introduction
LongStream addresses long-sequence streaming reconstruction failures by decoupling pose coordinates and metric scale while aligning Transformer training with cache-based inference. It achieves stable online reconstruction across long sequences with real-time throughput.
- Motivation: Existing streaming methods process sequences incrementally but can suffer catastrophic extrapolation failure, trajectory collapse, and tracking failure within tens of meters.These failures arise despite linear-time reconstruction and are documented for long sequences.
- Motivation: First-frame anchoring forces absolute-pose prediction, making large-index extrapolation difficult under train-short, test-long conditions.The resulting domain gap contributes to increasingly difficult long-sequence prediction.
- Gauge-decoupled design: LongStream predicts keyframe-relative poses instead of using a fixed first-frame anchor, converting long-range extrapolation into a constant-difficulty local task.The predictions become invariant to the global coordinate choice.
- Gauge-decoupled design: Orthogonal scale learning separates scale-invariant geometry optimization from independent global-scale prediction, reducing scale entanglement and stabilizing metric outputs.The geometry branch learns shape while a dedicated scale head predicts the global scale factor.
- Streaming architecture: Cache-consistent training and periodic cache refresh address attention-sink reliance and long-term KV-cache contamination by aligning training and inference contexts and reducing stale memory.The approach targets temporal degradation and drift in ultra-long streaming sequences.
- Results: 18 FPS enables real-time, metric-scale reconstruction across kilometer-scale sequences, with state-of-the-art streaming reconstruction on indoor and outdoor datasets.The reported benchmarks include KITTI, vKITTI, Waymo, TUM-RGBD, ETH3D, and 7Scenes.
2. Related Work
Prior work spans classical, offline, and streaming 3D reconstruction, but long-sequence online stability remains difficult. LongStream is presented against methods that trade accuracy, scalability, or temporal consistency in different ways.
- Classical SfM and MVS: SfM estimates camera poses and sparse structure through feature matching and bundle adjustment, while MVS densifies reconstruction with pixelwise depth.These optimization-heavy pipelines offer accuracy and interpretability but scale poorly to large or dynamic scenes and are difficult to deploy in real time.
- Offline 3D reconstruction: Offline neural methods such as DUSt3R and MASt3R operate on image pairs, requiring global alignment or costly fusion for multi-view scenes.Pointmaps improve computational efficiency and support applications including SLAM and neural rendering.
- Streaming 3D reconstruction: Streaming methods update geometry frame by frame, but recurrent or causal architectures degrade on long sequences because of long-term dependency and cached-token problems.CUT3R struggles with long-term dependencies, while Stream3R exhibits attention collapse as cached tokens dominate.
- Streaming 3D reconstruction: StreamVGGT improves consistency through temporal causal attention, cache updates, and distillation, yet cache contamination still limits long-horizon stability.Existing streaming methods generally degrade as sequence length grows and fail to generalize to much longer streams.
3. Methodology
LongStream decouples streaming geometry from global coordinate and metric-scale gauges by predicting keyframe-relative poses and learning scale separately. Cache-consistent training and periodic refresh address attention-sink dependence and long-term cache saturation for stable ultra-long streaming.
- Gauge-decoupled framework: LongStream jointly predicts keyframe-relative pose, depth, pointmaps, and global scale within a unified spatiotemporal Transformer.Its architecture uses a ViT encoder, causal aggregator with a shared KV cache, and task-specific heads.
- Gauge-decoupled formulation: Relative pose T_i←k = T_i ∘ T_k^-1 makes prediction invariant to global SE(3) coordinate reparameterization.Using the preceding keyframe bounds the index gap and converts long-range extrapolation into a local estimation task.
- Reference-aware attention: Reference-aware attention restricts non-keyframes to their keyframe and intervening frames, while keyframes attend only to the preceding keyframe interval.The design ensures that the model learns relative relationships rather than relying on the entire history.
- Orthogonal scale learning: Orthogonal scale learning separates normalized geometry optimization from dedicated global scale estimation for Sim(3) gauge decoupling.The scale head predicts a positive scale factor, while scale affects translation, depth, and pointmaps but not rotation or field of view.
- Probabilistic framework: The probabilistic objective jointly models images, pointmaps, depth, relative pose, and global scale, with translation supervision computed in normalized coordinates.This prevents translation loss from implicitly encoding global scale.
- Cache consistency: Cache-consistent training passes and trims KV caches between chunks, matching training visibility to frame-by-frame inference and suppressing attention-sink dependence.Periodic cache refresh then clears stale context while preserving continuity in the keyframe-relative coordinate system.
4. Experiments
LongStream is evaluated on diverse indoor and outdoor sequences, where it maintains stable streaming reconstruction under long, challenging trajectories. It achieves strong accuracy and real-time operation while reducing long-sequence errors relative to existing approaches.
- 3D Reconstruction: Full-sequence reconstruction on 7Scenes and TUM is competitive with offline approaches using Chamfer Distance and F1@0.25.The evaluation uses test sequences without subsampling; some baseline failures produce extremely large Chamfer Distance values.
- Outdoor Sequences: LongStream preserves trajectory continuity and metric accuracy across several hundred meters, including large loop closures without explicit loop-closure modules.Existing streaming methods accumulate drift over long outdoor trajectories, while VGGT-SLAM encounters memory and tracking limitations.
- Indoor Sequences: LongStream maintains coherent poses and 3D structure on indoor trajectories with strong viewpoint changes, occlusions, and repeated back-tracking.The qualitative comparison contrasts LongStream with drift-prone Stream3R, StreamVGGT, and VGGT-SLAM runs.
- Ablation Study: Combining all four components reduces ATE from 8.043 to 0.115, nearly two orders of magnitude.The ablation evaluates the keyframe-relative pose head, scale branch, cache-consistent training, and periodic cache refresh.
5. Conclusion
LongStream provides stable, metric-scale reconstruction over ultra-long sequences by combining gauge-decoupled poses with cache-consistent training. The model remains effective across diverse indoor and outdoor settings, though its scope is limited by static-world and heuristic-scheduling assumptions.
- Conclusion: LongStream delivers stable, metric-scale reconstruction over ultra-long sequences while preserving consistent geometry and scale across thousands of frames.The conclusion also reports strong accuracy and real-time performance on diverse indoor and outdoor datasets.
- Limitations: The model assumes a largely static world, relies on a heuristic keyframe schedule, and shows mild pointmap-consistency degradation over very long windows.These limitations are identified as directions for improving robustness and generality.
7. Gauge Invariance of Relative Pose and Scale
LongStream’s relative-pose and scale objectives remove dependence on arbitrary global coordinates and separate shape learning from metric-scale estimation. Together, these choices yield a gauge-consistent training objective for long-sequence reconstruction.
- SE(3) Gauge Invariance: Keyframe-relative pose T_i←k = T_i T_k^-1 remains unchanged under any global SE(3) coordinate transformation.Absolute poses transform with the inverse global transformation, which cancels in the relative-pose product.
- SE(3) Gauge Invariance: This invariance ensures that training is unaffected by arbitrary choices of world coordinates.The relative-pose target is therefore independent of the selected global coordinate frame.
- Scale Parameterization: The predicted metric point cloud is represented as X = s X_hat_raw, with s > 0 supplied by the dedicated scale head.The scale factor affects metric outputs while the geometry branch operates on normalized predictions.
- Sim(3) Scale Decoupling: Normalization cancels the global scale factor from the geometry objective, making the geometry loss independent of s.The normalized prediction divides out the same scale factor in numerator and denominator.
- Sim(3) Scale Decoupling: Global scale is learned solely through the dedicated scale objective, while normalized geometry supervision trains 3D structure separately.The paper summarizes this as orthogonal scale learning and gauge-consistent training.
8. Additional Attention Visualization Analysis
The analysis shows that baseline streaming models over-rely on the first frame and distant history, destabilizing long-range prediction, whereas cache-consistent training shifts attention toward nearby frames and improves stability.
- Attention bias: Baseline attention concentrates disproportionately on the first frame and distant history instead of recent frames relevant to local geometric consistency.This imbalance causes rapid RPE growth and unstable long-range predictions.
- Cache-consistent training: Cache-consistent training suppresses the attention sink and produces more balanced attention across causal and windowed inference settings.The accompanying RPE heatmaps report stable and best accuracy with CCT.
- Attention bias: In windowed inference, retaining the attention sink accelerates degradation, while removing it causes collapse, revealing strong dependence on the initial frame.
- Cache-consistent training: CCT allocates relatively more attention to nearby frames, improving stability across full-window and sliding-window inference.As sequence length approaches ∼80 frames, attention still shifts gradually toward earlier history, consistent with cache saturation.
- Interpretation: Overall, CCT aligns attention patterns more closely with temporal geometric coherence than baseline streaming inference.
9. Long-Sequence Stability Analysis
Long-sequence experiments on Waymo and KITTI show nonlinear error growth for streaming methods as sequences lengthen, while LongStream remains stable through first-frame decoupling and cache management.
- Observed degradation: Streaming baselines and LongStream without refresh exhibit nonlinear error growth as sequence length increases.Longer histories or revisiting do not necessarily improve performance under strictly online constraints.
- LongStream stability: LongStream remains stable over long sequences by removing first-frame anchoring and mitigating long-history effects with cache-consistent training and periodic cache refresh.
- Evaluation setting: 135 m Waymo and 561 m KITTI sequences are used to evaluate long-sequence stability as sequence length increases.
10. Additional Hyperparameter Analysis
The ablations identify practical operating points for keyframe switching and cache length, while showing that excessively sparse keyframes, frame-to-frame tracking, or long caches degrade performance.
- Keyframe interval: N = 1 degenerates into frame-to-frame tracking and causes rapid error accumulation, while N = 15 degrades because sparse switches receive too few supervision signals.
- Cache window size: W = 30 significantly impairs accuracy, with ATE rising to 0.516, supporting the geometric-saturation explanation for excessively long histories.
- Keyframe interval: N = 10 provides the best trade-off between drift accumulation and training dynamics.
- Loop closure: Without loop-closure correction, LongStream shows mild drift when revisiting the same place.Online loop-closure cues are identified as a promising direction for improving global consistency.
- Cache window size: W = 10 maintains sufficient context while preventing geometric saturation and reducing computational cost.
11. Additional Limitation
LongStream does not perform explicit loop-closure optimization, limiting trajectory correction during revisits compared with offline global bundle adjustment.
- Scope limitation: LongStream lacks explicit loop-closure optimization and therefore does not obtain the strong trajectory correction available from offline global bundle adjustment.Lightweight online loop-closure cues are left as future work to improve global consistency, especially in large loops.