PaperlayerView exact revision on arXiv

Source-linked AI summary

HorizonStream: Long-Horizon Attention for Streaming 3D Reconstruction

Chong Cheng, Peilin Tao, Nanjie Yao, Guanzhi Ding, Xianda Chen, Yuansen Du, Xiaoyang Guo, Wei Yin, Weiqiang Ren, Qian Zhang, Zhengqing Chen, Hao Wang

arXiv:2605.23889v1cs.CV

TL;DR

Streaming 3D reconstruction degrades on long sequences because existing methods apply uniform, recency-based influence to geometrically heterogeneous evidence. HorizonStream factorizes this influence across bounded long-range propagation and local 3D matching, remaining stable beyond 10,000 frames while using constant memory and linear time.

  • Problem

    Streaming 3D reconstruction still degrades on long sequences because existing architectures organize history by recency despite geometric evidence having heterogeneous lifetimes.

  • Method

    HorizonStream factorizes geometric evidence influence into bounded multi-timescale temporal propagation, gated local 3D matching, and metric readout for scale and pose.

  • Results

    HorizonStream remains stable on sequences exceeding 10,000 frames, outperforming streaming methods with constant memory and linear time after training on 48-frame clips.

  • Takeaways & Limitations

    The framework provides sequence-length-independent geometric propagation that generalizes from short training clips to ultra-long streaming reconstruction.

  • Takeaways & Limitations

    For extremely long sequences with repeated revisits, the fixed-size recurrent state can miss fine-grained details, while dynamic foreground objects can corrupt local geometric evidence.

Abstract

from arXiv · show

Online 3D reconstruction requires estimating camera pose and scene geometry under strict causal and bounded-memory constraints. Existing methods often suffer from drift, jitter, or collapse on long sequences. We trace these failures to a fundamental mismatch. Streaming geometry is inherently temporally heterogeneous, with evidence ranging from short-lived correspondences to persistent global scale. However, current architectures impose uniform and pathological influence patterns. For example, sliding windows enforce hard cutoffs, while ungated recurrence and causal attention cause cache saturation and spike-like attention sinks. To resolve this, we formalize geometric propagation as an \emph{evidence influence kernel} and propose HorizonStream, a long-horizon Transformer that explicitly factorizes this kernel. For the long-range temporal factor, Geometric Linear Attention learns channel-wise decay rates to enable bounded, multi-timescale propagation of geometric evidence. For the short-range spatial factor, Geometric Local Attention with Spatiotemporal RoPE performs reliable 3D matching while suppressing attention sinks. Finally, Metric Readout Tokens recover stable scale and rigid pose directly from the persistent geometric state. Extensive experiments show that HorizonStream, trained on only 48-frame clips, generalizes stably to sequences exceeding 10,000\ frames with constant memory and linear time, achieving state-of-the-art streaming 3D reconstruction performance. Project Page: https://3dagentworld.github.io/horizonstream/

1 Introduction

The introduction identifies recency-based memory as the cause of long-sequence streaming 3D reconstruction failures and frames geometric propagation as temporally heterogeneous evidence aggregation. HorizonStream addresses this mismatch with a factorized, bounded influence kernel combining multi-timescale temporal propagation and local spatial matching.

  • Problem: Strictly causal streaming reconstruction degrades over long sequences, causing collapse, pose jitter, and scale instability when history is organized purely by recency.The passage attributes these failures to recency-based history organization.
  • Motivation: Geometric evidence has heterogeneous lifetimes: local 2D-3D correspondences are short-lived, whereas global scale and scene structures persist.Recent evidence may become invalid while older evidence remains reliable.
  • Problem: Existing memory designs impose pathological influence kernels, with sliding windows forgetting immediately and full attention retaining everything permanently.Both extremes lack bounded, flexible temporal retention tailored to geometric channels.
  • Method: HorizonStream factorizes the influence kernel using Geometric Linear Attention for bounded O(1) multi-timescale propagation and Geometric Local Attention for short-range 3D matching.The temporal component learns channel-wise exponential decay rates, while the spatial component performs local content matching.
  • Results: 48-frame training generalizes stably to tens of thousands of frames without pose degradation and outperforms all streaming 3D reconstruction methods.The proposed local, bounded kernel yields a sequence-length-independent propagation rule for arbitrary-length streams.

2 Related Work

Prior 3D reconstruction methods span offline feed-forward models and causal online systems that exchange information through windows, recurrent states, memories, or cache operations. HorizonStream instead advocates bounded, multi-timescale control of geometric evidence to preserve useful long-range geometry while down-weighting stale evidence without cache resets.

  • Offline feed-forward 3D reconstruction: Offline methods extend dense pairwise geometry prediction to sequences, arbitrary image collections, reduced-memory inference, and longer inputs through spatial memory, attention-map reuse, chunking, or accumulated weights.The cited passage names DUSt3R, MASt3R, Spann3R, MonST3R, VGGT, FastVGGT, VGGT-Long, and LoGeR.
  • Online feed-forward 3D reconstruction: Online methods use causal masks with sliding windows, persistent recurrent or pointer states, KV-cache pruning, or keyframe memory to transfer information across windows.The passage identifies STream3R, StreamVGGT, CUT3R, TTT3R, Point3R, InfiniteVGGT, and Lingbot-map as examples.
  • Limitations of existing online methods: LongStream links long-sequence degradation to attention sinks and state saturation, while periodic cache refresh discards accumulated context at each boundary and weakens long-range revisit.This limitation motivates alternatives that retain accumulated geometric context across boundaries.
  • HorizonStream’s motivation: HorizonStream proposes bounded, multi-timescale control of evidence influence by learning channel-wise propagation scales that preserve useful long-range geometry and down-weight stale evidence without cache reset.This is presented as the required direction for a better online 3D reconstruction pipeline.

3 Method

HorizonStream factorizes geometric evidence influence into causal local matching, bounded multi-timescale cross-window propagation, and metric readout. Its O(1) recurrent geometric state, channel-wise retention, gated spatial attention, and metric tokens address correspondence reliability, long-range accumulation, scale, and pose.

  • Overview: The architecture causally processes the latest W frames while maintaining an O(1) geometric state for cross-window structure and scale.Geometric Local Attention handles within-window matching, Geometric Linear Attention performs cross-window propagation, and Metric Readout Tokens recover scale.
  • Kernel Factorization: The evidence influence kernel is factorized into spatial and temporal components, with dedicated mechanisms for local correspondence selection, bounded propagation, and metric preservation.Kspatial uses image content and 3D proximity, Ktime uses channel-wise exponential decay, and Metric Readout completes the factorization.
  • Geometric Linear Attention: Geometric Linear Attention summarizes correspondence, motion, structure, and scale cues in a bounded recurrent state that revises stale information and preserves long-lived geometry.Discounting closes the open-form temporal influence that causes unbounded accumulation.
  • Geometric Linear Attention: Channel-wise retention assigns distinct temporal influence horizons, allowing low-γ channels to revise transient correspondences while high-γ channels preserve structure and metric cues.This replaces a scalar lifetime that cannot represent the different persistence requirements of local correspondences, motion, scene structure, and scale.
  • Geometric Local Attention: Geometric Local Attention combines head-wise reliability gating with three-axis Spatiotemporal RoPE to select reliable local evidence using content and relative space-time layout.Gating downweights unreliable heads and preserves heads supporting local matching; temporal indices are periodically reset to avoid unbounded positional growth.
  • Metric Readout Tokens: Metric Readout Tokens read scale from high-retention recurrent-state channels and use it to rescale predicted translation and depth across windows.Each frame includes a learned Metric Readout Token, extending metric readout from local context to sequence-level evidence.

4 Experiments

Experiments evaluate HorizonStream across indoor, outdoor, synthetic, and ultra-long sequences, where it remains stable and competitive with offline methods. Ablations attribute this performance to bounded long-term retention, local spatial matching, and metric pose/scale readout.

  • Long-short sequence generalization: HorizonStream achieves the best overall performance among online methods across indoor, KITTI-scale, and ultra-long sequences exceeding 10,000 frames.Indoor evaluations use full sequences without downsampling; competing streaming methods degrade through jitter, pose degradation, or collapse as sequence length grows.
  • KV-cache contamination: Without periodic refresh, CUT3R, TTT3R, and LoGeR degrade sharply, while HorizonStream avoids refresh by discounting stale evidence in a bounded geometric state.The comparison indicates temporal-state contamination rather than limited model capacity as the cause of refresh-related degradation.
  • Reconstruction and depth accuracy: HorizonStream achieves the best online reconstruction quality across four benchmarks and approaches the best offline methods on KITTI depth.The reconstruction advantage is mainly attributed to more accurate pose estimation; 7Scenes baseline errors are inflated on Chess, Pumpkin, and RedKitchen.
  • Ablation study: Removing Geometric Linear Attention causes severe drift, while disabling channel-wise gating or replacing it with TTT-like fast weights degrades longer-horizon performance.Learned effective lifetimes form a continuous channel- and layer-wise spectrum, supporting multi-timescale retention.
  • Ablation study: Removing Geometric Local Attention severely degrades performance, and removing either head-wise output gating or Spatiotemporal RoPE substantially increases long-sequence drift.These components support fine-grained spatial matching and complementary attention control within each local window.
  • Readout and limitations: Metric Readout Tokens and multi-token pose aggregation each provide consistent gains, while the 10-frame pose window reduces memory cost but may miss fine-grained details during repeated revisits.The fixed-size recurrent state can miss details in extremely long sequences with repeated revisits.

5 Conclusion

HorizonStream unifies long-term temporal memory and short-term spatial matching through an evidence influence kernel for streaming 3D reconstruction. Trained on 48 frames, it generalizes beyond 10,000-frame sequences with constant memory and linear time.

  • HorizonStream is a streaming 3D reconstruction framework built on an evidence influence kernel.The kernel unifies long-term temporal memory with short-term spatial matching.
  • 48 frames of training enable generalization to sequences exceeding 10,000 frames.
  • Constant memory and linear time support streaming reconstruction over these long sequences.

A Geometric Attention Dilution · B Extended Theoretical Analysis

The analysis formalizes why causal softmax attention fails as long-range cross-window memory: geometrically relevant frames are bounded, so their attention share vanishes as sequences grow. Even perfect score discrimination cannot prevent dilution, while sink-token concentration leaves the underlying signal loss and O(t) cache cost unchanged.

  • A Geometric Attention Dilution: Causal softmax attention cannot serve as long-range cross-window memory for streaming 3D reconstruction.
  • A Geometric Attention Dilution: The geometric relevance of frame i to frame t is the co-visibility ratio r(i, t) = |Ωi ∩Ωt|/|Ωt|.For exploration of new regions, the relevant-frame set Rt is bounded by |Rt| ≤ Wgeo, determined by scene geometry and camera speed.
  • A Geometric Attention Dilution: The total attention on geometrically relevant frames is bounded by the corresponding relevant-mass expression.This bound compares attention assigned to the bounded co-visible set against all prior frames.
  • A Geometric Attention Dilution: For t > Wgeo(1 + e2M), more than half the attention mass falls on geometrically irrelevant frames.Even with perfect score discrimination, the relevant fraction decays as O(Wgeoe2M/t).
  • B Extended Theoretical Analysis: Assigning score +M to all Wgeo relevant frames and −M to all others gives the best-case attention bound.The derivation divides numerator and denominator by Wgeo · eM to obtain (11), while any suboptimal assignment worsens the bound.
  • B Extended Theoretical Analysis: For t ≫ Wgeoe2M, the relevant-attention bound is O(Wgeoe2M/t) → 0.Thus, even under optimal scoring, geometrically useful attention vanishes over sufficiently long sequences.
  • B Extended Theoretical Analysis: Sink tokens can absorb wasted irrelevant attention, but the geometrically useful signal fraction still vanishes as O(1/t).Causal softmax therefore remains unsuitable for cross-window memory because it retains O(t) KV-cache cost.

B.1 Zero-Forgetting Contamination and Stability … C Implementation Details

The appendix establishes that selective, channel-wise forgetting prevents initialization contamination, provides adaptive multi-timescale memory, and keeps the recurrent state stable. It also formalizes Geometric Linear Attention as discounted test-time training, while the supplied passages contain no information about C Implementation Details.

  • B.1 Zero-Forgetting Contamination and Stability: Under zero forgetting, the initial state contributes to every output with undiminished magnitude, and new evidence cannot dilute it.This permanently anchors the recurrent state to initialization regardless of camera motion.
  • B.1 Zero-Forgetting Contamination and Stability: With channel-wise retention satisfying γ̄ < 1, the initial-state contribution decays exponentially and the recurrent state remains adaptive to incoming evidence.The passages present γ̄ < 1 as a sufficient condition for closing initial-state influence.
  • B.2 Effective Memory Horizon: For channel c, the effective memory horizon is τ(c) = −1/log γ(c), and observations older than 3τ(c) contribute below 5% of their original weight.The supplied derivation gives the 5% threshold at t−i = 3τ(c).
  • B.2 Effective Memory Horizon: Learned gates partition channels into fast and slow subspaces, providing short-term feature buffering and long-term geometric memory with different horizons.The fast horizon is described as τfast ≪ T, while the slow horizon is τslow ≫ W; the partition adapts end-to-end to training data.
  • B.3 State Norm Boundedness: Under γ̄ < 1 and bounded inputs, Proposition 5 establishes a uniformly bounded Frobenius norm for the persistent state.The proposition addresses boundedness as t → ∞.
  • B.3 State Norm Boundedness: Ungated linear attention instead permits linear state-norm growth, eventually requiring resets to prevent overflow.The cited comparison gives ∥St∥F ≤ ∥S0∥F + t·BkBv and links reset requirements to TTT-based methods.
  • B.4 Formal Connection to Test-Time Training: Geometric Linear Attention is equivalent to discounted test-time training for an online linear model optimized with a discounted objective.When γt ≡ 1, the formulation reduces to standard online linear regression, which is equivalent to linear attention.
  • B.4 Formal Connection to Test-Time Training: The discounted TTT interpretation follows by discounting the previous iterate by γt and applying one gradient-descent step to the current prediction loss.The supplied proof evaluates the gradient at S′ = γtSt−1 before updating the state.

C.1 Architecture Details · C.2 Training Hyperparameters

HorizonStream uses a 24-layer alternating frame/global Transformer with interleaved persistent-state geometric attention, lightweight pose/depth heads, and explicit training choices for metric scale and augmentation. Its design maintains nearly constant memory while scaling to 10K-frame sequences.

  • C.1 Architecture Details: 24 Transformer layers alternate between frame and global blocks, with Geometric Linear Attention at layers 4, 11, 17, and 23.Each Geometric Linear Attention layer updates the persistent state before Geometric Local Attention operates.
  • C.1 Architecture Details: 2.0 head-wise gate bias initializes attention to preserve pretrained behavior, while Geometric Linear Attention gates begin with γ ≈1.The gates gradually learn channel-wise propagation behavior during training.
  • C.1 Architecture Details: 10K frames: HorizonStream keeps peak memory nearly constant and scales smoothly, whereas competing methods require increasing memory or higher runtime on long sequences.This scaling behavior is reported in the memory and runtime comparison.
  • C.1 Architecture Details: The pose consensus head refines translation, rotation quaternion, and focal length through residual corrections over K lightweight-transformer rounds.The depth head uses DPT-style multi-scale fusion from four intermediate layers.
  • C.2 Training Hyperparameters: 518×518 input images feed a Geometric Linear Attention state with dimension S ∈Rd×d and d=1024.These are the stated input and model dimensions.
  • C.2 Training Hyperparameters: Scale loss is restricted to metric-scale samples, while depth uses confidence-weighted SmoothL1 loss.Training also applies random color jitter, random cropping, and random horizontal flip.

C.3 Training Data

HorizonStream is trained on 24 diverse datasets using variable-stride video sampling and pseudo-temporal sequences from unordered image collections. Geometric Linear Attention converges faster and reaches lower final loss, while loop closure reduces ATE on revisited-region sequences without harming other sequences.

  • Training Data: Training spans 24 datasets across indoor, outdoor, driving, and synthetic environments.The datasets provide varied scene and capture conditions.
  • Training Data: Video sampling uses variable temporal strides from 1 to 8, while unordered image collections become pseudo-temporal sequences through camera graph traversal.Per-dataset sampling ratios are listed in Table 7.
  • Training Data: Geometric Linear Attention with channel-wise gating converges faster and reaches a lower final loss for cross-window propagation.This comparison is shown in Figure 8 under different attention mechanisms.
  • Training Data: Loop closure reduces ATE on long sequences with revisited regions while maintaining performance elsewhere.Figure 9 reports this effect across long sequences.

D Evaluation Dataset Details

The evaluation uses full-length sequences without frame subsampling across diverse real, synthetic, indoor, outdoor, and ultra-long datasets. Protocols include held-out driving segments, coordinate alignment, and sequence-specific coverage.

  • Evaluation protocol: All sequences are evaluated at full length without frame subsampling.This establishes the common evaluation protocol across datasets.
  • Dataset coverage: KITTI, vKITTI2, and 7Scenes cover all 11 KITTI sequences, morning scenes in five virtual environments, and sequence 01 across seven scenes.The evaluated vKITTI2 environments are Scene01, Scene02, Scene06, Scene18, and Scene20; 7Scenes includes Chess, Fire, Heads, Office, Pumpkin, RedKitchen, and Stairs.
  • Dataset coverage: Waymo Open evaluation uses 9 training-disjoint segments to measure generalization on unseen driving scenes.The selected segments contain 196–199 frames and span 42–351 m, although Waymo is included in training data.
  • Dataset coverage: ScanNet++ contributes 5 scenes, while Oxford Spires includes all 14 subsets with image-to-ground-truth point-cloud alignment.Aligned-image counts vary across Oxford Spires subsets, increasing evaluation difficulty.
  • Long-sequence evaluation: VBR evaluates all 7 sequences at full length, spanning 8,815–18,846 frames and trajectories up to 5.2 km.TUM RGB-D and ETH3D use standard protocols with full sequences.

E Additional Experimental Results · E.1 Channel-to-Geometry Linear Probing

Additional experiments show that HorizonStream maintains accuracy and efficiency on long sequences, while probing reveals that its recurrent states encode metric information across multiple retention timescales. Loop closure improves revisited-region performance but does not eliminate failures in dense-revisit or ambiguous scenes.

  • E Additional Experimental Results: HorizonStream achieves the lowest average ATE among online methods across the 12 Oxford Spires evaluation sites, with especially large margins on college5 and christ2.The result is reported in Tab. 8 and concerns per-sequence absolute trajectory error.
  • E Additional Experimental Results: Optional loop closure reduces ATE on sequences with revisited regions while maintaining comparable performance.Its effect is shown on long sequences in Fig. 9.
  • E Additional Experimental Results: Across sequence lengths from 200 to 10,000 frames, HorizonStream maintains nearly constant peak GPU memory and smooth runtime scaling.Competing methods either run out of memory or show super-linear runtime growth.
  • E Additional Experimental Results: Geometric Linear Attention with channel-wise gating converges faster and reaches lower final loss than ungated and softmax-attention variants.The comparison attributes the improvement to bounded multi-timescale retention during training.
  • E Additional Experimental Results: Failures mainly arise in ultra-long sequences with dense revisits or visually ambiguous regions, where the fixed-size recurrent state lacks fine-grained information for precise relocalization.The optional loop-closure module partially mitigates these failures.
  • E.1 Channel-to-Geometry Linear Probing: Frozen Geometric Linear Attention states are evaluated using ridge regressors trained on KITTI sequences 00 and 02 and tested on held-out sequence 05.The experiment extracts 1024-dimensional features from all four Geometric Linear Attention layers for each chunk.
  • E.1 Channel-to-Geometry Linear Probing: The probes predict local translation error, long-range scale log error, long-range translation error, and segment scale log error.These four targets test whether recurrent geometric states contain linearly decodable geometric information.
  • E.1 Channel-to-Geometry Linear Probing: Segment-scale log error is the most reliably predictable target, with probe weights distributed across short-, medium-, and long-retention channels.This supports metric representation across the learned retention spectrum rather than concentration in one band.
Loading 2605.23889v1…