PaperlayerView exact revision on arXiv

Source-linked AI summary

LoGeR: Long-Context Geometric Reconstruction with Hybrid Memory

Junyi Zhang, Charles Herrmann, Junhwa Hur, Chen Sun, Ming-Hsuan Yang, Forrester Cole, Trevor Darrell, Deqing Sun

arXiv:2603.03269v2cs.CVcs.LG

TL;DR

Feedforward geometric models struggle to scale dense 3D reconstruction beyond short contexts because attention is costly and recurrent memory is limited. LoGeR processes chunks with hybrid SWA and TTT memory, achieving strong long-sequence reconstruction, including 19k-frame evaluation, while reducing KITTI ATE from 72.86 to 18.65. Its effective range remains constrained by TTT length generalization, with periodic resets needed beyond the training context.

  • Problem

    Feedforward geometric models are limited to bounded scenes because quadratic attention restricts context and short-context training does not support long-range dependencies at inference.

  • Method

    LoGeR processes video in chunks and combines SWA for high-fidelity adjacent alignment with TTT memory for long-range global consistency.

  • Results

    LoGeR processes thousands of frames without optimization and reduces KITTI ATE from 72.86 to 18.65 while achieving a 55.2% relative improvement on extremely long VBR sequences.

  • Takeaways & Limitations

    The hybrid architecture enables globally consistent long-sequence reconstruction while retaining local geometric detail and high-fidelity intra-chunk reasoning.

  • Takeaways & Limitations

    TTT struggles to generalize beyond its training chunk count, so sequences beyond about 1,000 frames can accumulate error and require periodic state resets.

Abstract

from arXiv · show

Feedforward geometric foundation models achieve strong short-window reconstruction, yet scaling them to minutes-long videos is bottlenecked by quadratic attention complexity or limited effective memory in recurrent designs. We present LoGeR (Long-context Geometric Reconstruction), a novel architecture that scales dense 3D reconstruction to extremely long sequences without post-optimization. LoGeR processes video streams in chunks, leveraging strong bidirectional priors for high-fidelity intra-chunk reasoning. To manage the critical challenge of coherence across chunk boundaries, we propose a learning-based hybrid memory module. This dual-component system combines a parametric Test-Time Training (TTT) memory to anchor the global coordinate frame and prevent scale drift, alongside a non-parametric Sliding Window Attention (SWA) mechanism to preserve uncompressed context for high-precision adjacent alignment. Remarkably, this memory architecture enables LoGeR to be trained on sequences of 128 frames, and generalize up to thousands of frames during inference. Evaluated across standard benchmarks and a newly repurposed VBR dataset with sequences of up to 19k frames, LoGeR substantially outperforms prior state-of-the-art feedforward methods--reducing ATE on KITTI by over 74%--and achieves robust, globally consistent reconstruction over unprecedented horizons.

1. Introduction

LoGeR addresses the context and data walls limiting feedforward 3D reconstruction by combining chunk-wise processing with hybrid memory for local and global coherence. It achieves long-horizon reconstruction while substantially improving KITTI trajectory error and evaluating sequences up to 19k frames.

  • Feedforward geometric models remain confined to bounded scenes because quadratic bidirectional attention limits context and short-context training leaves long-range inference dependencies unsupported.
  • Chunk-wise processing preserves in-distribution local reasoning but introduces the challenge of maintaining coherence across chunk boundaries.
  • The hybrid memory balances lossless local geometric details, global structural consistency, and linear computational cost with sequence length.
  • LoGeR combines non-parametric SWA for uncompressed local detail with parametric TTT memory that anchors the global coordinate frame and prevents scale drift.
  • 19k frames and 11.5 km in trajectory define the new VBR benchmark, while KITTI ATE drops from 72.86 to 18.65.

2. Related Work

Related work spans learning-based SLAM, feedforward reconstruction, external or recurrent memories, and efficient attention mechanisms. LoGeR uses chunk-wise processing and hybrid memory to retain bidirectional multi-frame reasoning while propagating information across chunks.

  • Learning-based visual SLAM methods learn strong 3D priors but retain expensive graph, loop-closure, and global-optimization backends.
  • Feedforward reconstruction directly predicts canonical pointmaps but is limited by global-attention memory costs and bounded input sequences.
  • Long-sequence feedforward methods use external spatial memory, recurrent states, sparse attention, or causal attention, but lack demonstrated results on exceptionally long sequences.
  • LoGeR processes streams chunk-wise, preserving bidirectional multi-frame reasoning while using hybrid memory to propagate information across chunks.
  • The method combines local SWA for precise adjacent alignment with parametric TTT for long-range global consistency under linear sequence-length cost.

3. Preliminary

The preliminary section introduces short-context geometric reconstruction, TTT as fast-weight memory, and the hybrid-memory motivation. TTT stores compressed context efficiently, while the paper motivates complementary mechanisms for preserving high-fidelity local information.

  • Bidirectional transformer models predict local pointmaps and global camera poses from multi-view images or video frames but remain limited to bounded scenes.
  • TTT stores context from prior forward calls by updating fast weights during training and inference while keeping base slow weights frozen during inference.
  • TTT defines a network f_W that maps keys toward values so its fast weights encode a compressed form of the KV cache.
  • Figure 2 presents chunked processing with SWA for neighboring-chunk consistency and TTT layers for long-range global consistency.
  • Updated fast weights process queries to produce outputs, but their compressed memory is inherently lossy despite linear-complexity context propagation.

4. Approach

LoGeR reconstructs long video streams chunk by chunk, combining bidirectional intra-chunk reasoning with complementary memories for local alignment and global propagation. This design targets bounded computation while preserving geometric consistency across long horizons.

  • Design requirements: The architecture requires local bidirectional reasoning, lossless short-range transfer, and linear-time fixed-size memory for long-range propagation.These requirements jointly address dense geometric fidelity, adjacent-boundary alignment, and global consistency across thousands of frames.
  • Chunk-wise processing: LoGeR processes video sequentially in chunks using a strong bidirectional geometry backbone for high-quality dense predictions.Each chunk may contain a variable number of frames and can use backbones such as VGGT or π3.
  • Hybrid memory: Chunk-wise TTT maintains fast weights that apply historical information to each chunk and update those weights with the current chunk’s information.The fast weights provide long-term, lossy compression across many chunks, using LaCT for chunk-aligned test-time updates.
  • Hybrid memory: SWA attends to tokens from the previous and current chunks, creating a bounded-compute pathway for lossless high-fidelity feature transfer across boundaries.SWA layers are inserted sparsely, with only four layers used to remain compute-bound.
  • Chunk-wise processing: Within each chunk, bidirectional attention performs geometric reasoning under a bounded context window before lightweight decoders output pointmaps and camera poses.The prediction heads follow π3 and produce dense pointmap predictions and per-frame camera poses.

5. Experiments

LoGeR is evaluated on long-sequence reconstruction benchmarks and shorter established datasets, where its hybrid memory supports accurate trajectories, reconstruction, and state estimation. Ablations further show that the memory components, large-scale training data, and curriculum each contribute to performance.

  • Long-sequence evaluation: LoGeR surpasses VGGT-Long by 32.5% on average on KITTI, particularly mitigating accumulated drift on open-loop trajectories without loop closure.
  • Shorter-sequence evaluation: LoGeR produces visually superior reconstructions on ScanNet and TUM-Dynamics, recovering structural details while avoiding severe distortions and geometric artifacts.
  • Shorter-sequence evaluation: Both LoGeR and the proposed baseline substantially outperform prior work on camera pose evaluation, with relative gains of 80.0% on ScanNet and 66.1% on TUM.
  • Long-sequence evaluation: On VBR, LoGeR increasingly outperforms Pi3-Chunk as sequence length grows and maintains global scale consistency up to 20k frames.Pi3-Chunk’s local SIM(3) scale estimation accumulates errors over extended distances, whereas TTT anchors LoGeR’s global scale.
  • Ablation study: Removing either TTT or SWA causes noticeable ATE degradation, while qualitative ablations associate TTT with global consistency and SWA with local smoothness.
  • Ablation study: Training without five large-scale datasets significantly reduces performance, supporting the importance of diverse long-horizon data for generalization.The excluded datasets are TartanAir, TartanAirV2, Waymo, Virtual KITTI 2, and OmniWorld-Game.
  • Ablation study: Progressive curriculum training consistently improves both LoGeR variants across datasets, stabilizing recurrent-layer optimization and feedforward alignment.

6. Conclusion

The paper concludes that LoGeR combines SWA and TTT to address long-sequence feedforward 3D reconstruction. It processes thousands of frames without optimization and outperforms methods using alternative memory or attention designs.

  • LoGeR combines Sliding Window Attention with a Test-Time Training layer as a hybrid architecture for long-sequence feedforward 3D reconstruction.
  • LoGeR processes thousands of frames without optimization and outperforms methods based on explicit memories, recurrent states, or causal attention.
  • The architecture opens potential directions for long-context spatio-temporal reasoning in dynamic scenes, video understanding, and robotics.

Discussion and Future work

The discussion identifies effective context length and training-data scale as remaining constraints, while pointing to broader applications for hybrid memory in tasks requiring local detail and global consistency.

  • TTT fast weights have fixed memory but struggle to generalize beyond the number of training chunks, limiting effective range to the training context length.
  • Sequences longer than 1,000 frames can accumulate error and trajectory drift, and periodic state resets currently mitigate this at the cost of long-term context.
  • Expansive, high-quality training data remains a significant bottleneck for long-context reconstruction.
  • The hybrid SWA + TTT architecture may extend to domains requiring both long-term global consistency and strong local dependencies.

Impact Statement

The paper’s potential societal consequences come from improving conversion of video into dense 3D scenes. The stated applications include VR, generative 3D, and robotics.

  • Potential applications of improved video-to-3D conversion include VR, generative 3D, and robotics.

A.1. Details of Training Data

LoGeR is trained on a diverse, standardized mixture of large-scale real and synthetic data, emphasizing long-range navigation scenes. Its π3-like backbone combines TTT throughout the residual blocks with sparse SWA layers for temporal context.

  • 14 large-scale datasets span real-world and synthetic indoor, outdoor, and autonomous-driving scenes.
  • Training inputs are standardized as multi-view sequences with 48 views, or 128 views for H200 training, at 504 × 280 resolution.
  • Navigation datasets receive heavy sampling weights to provide long-range geometric signals, while DL3DV is also heavily weighted for its complementary diversity.
  • The π3-like backbone uses a DINO patchifier, 18 residual blocks, and heads for local pointmaps, global camera poses, and confidence maps.
  • TTT layers appear in all 18 residual blocks, while SWA layers are inserted sparsely at blocks 6, 10, 14, and 18 with learnable temporal-overlap tokens.

A.3. More Training Details

Training and inference use staged optimization, memory-saving implementations, and chunk-wise processing. The Pi3-Chunk baseline aligns overlapping chunks with scale-aware transformations, while LoGeR is evaluated against feedforward and optimization-based methods.

  • More Training Details: AdamW training freezes the encoder and prediction heads while using decoupled learning rates for new memory layers and unfrozen base parameters.
  • More Training Details: The objective weights rotation, translation, and global pointmap losses at λr = 0.1, λt = 10, and λglobal = 1.
  • More Training Details: Curriculum training progressively adjusts chunk and overlap sizes across hardware-specific stages to build long-context reasoning.
  • More Inference Details: Gradient checkpointing and FlexAttention reduce memory and compute costs for long-sequence unrolling and chunk-wise SWA.
  • More Inference Details: Inference uses chunk size 64 with overlap 3 for short sequences, while larger scenes use smaller LoGeR chunks to mitigate trajectory drift.
  • More Inference Details: On a 500-frame sequence, inference speed changes slightly with chunk size while memory consumption remains constant.
  • More Details on the Proposed Baseline: Pi3-Chunk stitches chunk predictions through overlapping frames, while its SIM(3) alignment compensates for each chunk’s relative scale ambiguity.
  • More Details on the Proposed Baseline: The relative scale factor is estimated robustly from median point-coordinate norms over valid overlap pixels, then applied to current-chunk pointmaps and pose translations.

B.2. Depth Evaluation

LoGeR is evaluated for video depth and long-sequence reconstruction across standard benchmarks. It outperforms prior methods on Bonn depth estimation and achieves strong results on 7-Scenes and qualitative long-trajectory comparisons.

  • Depth Evaluation: On 7-Scenes at 1k frames, LoGeR reports 90.3% and 72.1% error reductions against TTT3R and VGG-T3, respectively, with comparable runtimes.
  • Depth Evaluation: On 7-Scenes, LoGeR is 84.1% faster than VGGT with 31.0% better performance.
  • Depth Evaluation: At 500 frames on Bonn, LoGeR achieves a 21.05% error reduction in Abs Rel compared with TTT3R.
  • Depth Evaluation: On Bonn, both Pi3-Chunk and LoGeR significantly outperform prior methods, despite depth estimation requiring less long-term global consistency than pose estimation or reconstruction.
  • Depth Evaluation: The VBR evaluation finds that LoGeR’s methods achieve the best results on almost all sequences.

B.5. More Qualitative Results on 3D Reconstruction

Qualitative evaluations extend LoGeR’s reconstruction beyond standard sequences to minutes-long, large-scale indoor and outdoor videos. These results illustrate the method’s long-horizon applicability across in-the-wild and VBR footage.

  • Additional qualitative results use in-the-wild videos and the VBR dataset.
  • Figure 14 presents minutes-long reconstruction results on large-scale indoor and outdoor scenes.
Loading 2603.03269v2…