Source-linked AI summary
Revisiting Local Context for Long-Horizon Streaming 3D Reconstruction
Jiarong Han, Jincheng Xiong, Yuzhou Liu, Linzhe Shi, Changjie Wu, Ning Guo, Mu Xu, Hang Zhang, Ming Qian
TL;DR
Extremely long video reconstruction requires online camera and geometry estimation under bounded resources, while long-range reference targets become harder as sequences grow. ABot-Recon keeps learned context and prediction targets local, composing adjacent-frame measurements into global results and refining rotations to limit drift. On Oxford Spires, it reduces ATE and RPE-R by approximately 40% relative to the respective best prior results.
Problem
Ultra-long streaming reconstruction must estimate camera motion and dense geometry causally with bounded memory and computation as temporal separation and camera displacement increase.
Method
ABot-Recon uses a fixed 12-frame local context, predicts current-frame geometry and adjacent-frame motion, and composes these reference-frame-consistent measurements globally with rotation refinement and composition-aware supervision.
Results
ABot-Recon delivers superior long-horizon tracking and reconstruction across challenging benchmarks, reducing Oxford Spires ATE and RPE-R by approximately 40% relative to the respective best prior results.
Takeaways & Limitations
Accurate and stable ultra-long trajectories can be recovered using local temporal context and reliable composition rather than persistent learned long-range memory.
Takeaways & Limitations
Gains are less pronounced on compact indoor benchmarks, where reconstruction results are not uniformly the best and persistent geometric context may remain useful.
Abstract
from arXiv · showhide
Streaming 3D reconstruction from extremely long videos requires estimating camera motion and scene geometry online under bounded memory and computation. Early streaming models achieve causal, bounded-cost inference using finite context buffers or compact recurrent states, yet their estimates often deteriorate as sequences grow. Recent methods improve long-horizon stability by coupling short-range context with persistent or multi-level long-range memory. We pursue a different route: we keep the learned temporal state strictly local and formulate predictions whose targets remain independent of sequence length. We present ABot-Recon, a simple streaming model that caches KV features from only the preceding 11 frames. It predicts a point map in the current camera coordinate system together with an adjacent-frame relative pose. These predictions remain equivariant under changes of reference frame, and global poses and geometry are recovered through sequential composition. To reduce accumulated drift, a lightweight temporal refiner improves relative rotations using recent visual and motion context, while a composition-aware pose loss supervises multi-step pose composition. Extensive evaluations on challenging long-sequence benchmarks demonstrate the superior long-horizon performance of our local-context approach. On Oxford Spires, ABot-Recon achieves an ATE of 4.35 m and an RPE-R of $0.12^\circ$, reducing both errors by approximately 40\% relative to the best prior results.
1 Introduction
Ultra-long streaming reconstruction must maintain bounded causal inference despite growing temporal separation and camera displacement. ABot-Recon addresses this by keeping predictions and learned context local, then composing local measurements into global trajectories and geometry.
- Ultra-long video reconstruction requires causal camera and geometry estimation with bounded memory and per-frame computation despite large temporal gaps.
- Local reference-frame targets keep the estimation range fixed as sequences grow, avoiding elaborate long-range cache management.The formulation supports training on short sequences while applying the model to ultra-long streams.
- Global poses and scene geometry are recovered by composing adjacent-frame transformations and transforming local point maps into the global reconstruction.
- A temporal rotation refiner and composition-aware pose supervision reduce drift from accumulated adjacent-frame pose errors without extending temporal context.The supervision accounts for the accumulated effect of multi-step pose composition.
- On Oxford Spires, ABot-Recon reduces both ATE and RPE-R by approximately 40% relative to the respective best prior results.The method achieves this across challenging long-sequence evaluations while using only local temporal context.
- ABot-Recon uses a 12-frame horizon, caching KV features from the preceding 11 frames while maintaining no persistent learned long-range state.It predicts a current-camera point map, confidence map, and adjacent-frame transformation.
2 Related Work
Related work evolves from feed-forward geometric priors and streaming temporal states toward stronger long-range memory and local-to-global decomposition. ABot-Recon instead relies on bounded local predictions with reliable composition.
- Feed-forward reconstruction methods replace substantial portions of classical SfM and SLAM pipelines with learned geometric priors for cameras, point maps, depth, and tracks.
- Streaming systems propagate historical information through external spatial memories, recurrent states, cached temporal keys and values, sliding windows, or explicit spatial pointers.
- Later methods improve long-sequence scaling through gated memories, keyframe-relative prediction, cache refresh, feature compression, adaptive state updates, and persistent attention.
- Local-to-global approaches separate bounded local motion estimation from global trajectory recovery through composition, graph optimization, submap alignment, or loop closure.
- ABot-Recon predicts bounded local poses in an equivariant reference frame, supervises multi-span pose chains, and refines rotations using recent context without learned long-range memory.An optional training-independent loop-closure backend can further refine trajectories at inference time.
3 Method
ABot-Recon performs streaming reconstruction with strictly local, fixed-window predictions and composes them into global geometry and camera motion. A temporal rotation refiner and composition-aware supervision target long-horizon drift while preserving bounded memory.
- Local prediction formulation: ABot-Recon predicts local point maps, confidence maps, and adjacent-frame poses, then recovers global geometry and camera motion through composition.Point maps are expressed in the current camera coordinate system, while relative poses connect adjacent frames.
- Rotation refinement: A motion–visual rotation refiner fuses recent motion and dense visual evidence, then predicts a residual rotation to correct unreliable relative-rotation estimates.A lightweight gated TCN aggregates the current pairwise feature with the K −1 preceding features while preserving causality.
- Composition-aware supervision: Composition-aware supervision compares multi-step composed transformations with ground truth instead of supervising adjacent transformations independently.Supervised frame pairs span gaps up to K −1, with weights proportional to temporal gap to emphasize accumulated chain errors.
- Training objective: Training combines composed relative-pose losses with residual regularization and local point-map, surface-normal, and confidence losses.The residual term encourages small-magnitude and temporally consistent rotation refinements, while λ coefficients balance the supervision terms.
4 Experiments
Experiments show that ABot-Recon maintains strong long-horizon pose and reconstruction accuracy using compact local context, while composition-aware supervision, longer training clips, and rotation refinement improve stability.
- ABot-Recon achieves strong trajectory accuracy across KITTI, Oxford Spires, and VBR using compact local temporal context without persistent global memory.
- 4.35 m average ATE is achieved on Oxford Spires without loop closure, while optional loop closure further reduces average ATE to 4.02 m.
- ABot-Recon maintains the lowest running adjacent-frame pose error throughout KITTI-02 without progressive degradation as the processed horizon grows.
- ABot-Recon achieves the lowest average rotational and translation errors at every evaluated KITTI multi-gap, with slower error growth at longer gaps.
- The model processes streams at 24.45 FPS using 6.71 GB of GPU memory, while CUT3R and TTT3R use fewer resources but exhibit substantially larger long-sequence drift.
- On Oxford Spires, ABot-Recon achieves the lowest CD and highest F1 among compared methods, while remaining competitive on 7Scenes and TUM-Dynamic.
- Composition-aware pose supervision improves consistently over the 32-frame baseline by directly constraining multi-step transformations and relative-pose consistency.
- Increasing training clips from 32 to 128 frames reduces ATE by 19.3%, 36.7%, and 17.8% on KITTI, Oxford, and VBR, respectively.
5 Discussion
The discussion identifies local prediction as especially effective for long, temporally continuous streams, while loop closure can add optional global correction when revisits provide long-range constraints.
- The local formulation is particularly effective for long, temporally continuous streams, whereas gains are less pronounced and reconstruction is not uniformly best on compact indoor benchmarks.
- Loop closure complements local prediction by adding sparse long-range constraints at inference time without changing the learned streaming formulation.
- Future extensions include stronger handling of dynamic scenes, external memory, larger-scale pretraining, and selective long-range constraints when local continuity becomes ambiguous.
6 Conclusion
ABot-Recon performs ultra-long streaming 3D reconstruction using only a fixed twelve-frame horizon and composable local predictions. Rotation refinement and composition-aware supervision support reliable long-term trajectory assembly.
- ABot-Recon operates entirely within a fixed twelve-frame temporal horizon.Its learned predictor avoids persistent long-range state while maintaining bounded model memory and per-frame computation.
- The model predicts current-camera-frame geometry and adjacent-frame motion, then composes these local measurements into global trajectories and scene geometry.
- A lightweight motion-visual rotation refiner and composition-aware pose supervision improve the reliability of multi-step pose composition.
- Experiments across long driving, handheld, and landmark-scale sequences show accurate camera tracking and dense reconstruction.
- The results indicate that persistent learned long-range memory is not required for effective ultra-long streaming reconstruction when local predictions remain composable.
7 Contributions
This section records the project’s contributors, sponsors, leaders, and additional supporters. It provides attribution rather than technical contributions or experimental findings.
- Jiarong Han, Jingcheng Xiong, Yuzhou Liu, Ming Qian, Linzhe Shi, and Changjie Wu are listed as contributors.
- Mu Xu and Ning Guo are listed as project sponsors.
- Hang Zhang, Jiarong Han, and Ming Qian are listed as project leaders.
- The paper expresses gratitude to eight additional supporters for their valuable support and contributions.
A.1 Temporal Sampling
Training sequences are constructed using temporal sampling for videos and pose-graph paths for unordered multi-view data. These policies control chronology, continuity, valid motion, and revisiting.
- Temporal sampling: Video training samples are ordered sequences, with interval [a, b] specifying frame-index steps between consecutive inputs.
- Temporal sampling: Forward fixed-stride sampling selects one interval and traverses long trajectories chronologically.
- Temporal sampling: Adaptive forward sampling reduces the maximum interval when the remaining trajectory cannot fit the configured clip length.
- Temporal sampling: Foldback sampling reverses traversal at sequence boundaries and resamples the stride, extending short videos while preserving local continuity.
- Multi-view construction: Unordered multi-view sequences use pose-graph neighbors, motion and turning costs, beam search, and a minimum unique-view ratio to select valid paths.
- Multi-view construction: US4K requires unique views, whereas HyperSim and BlendedMVS permit constrained revisiting when necessary.
A.2 Data Filtering
The training data undergo dataset-specific filtering to remove annotation, image, depth, pose, and geometry problems. These interventions include excluding unreliable scenes and sanitizing corrupted values.
- Filtering strategy: Inspection of sampled examples and loss-spike samples identifies problematic data for dataset-specific filtering.Removing erroneous samples substantially improves model performance.
- Dataset-specific filtering: BlendedMVS excludes 38 scenes containing sideways or upside-down images.
- Dataset-specific filtering: TartanAir and TartanAir-v2 invalidate erroneous sky-region depths and exclude the Ocean scene because of water-surface depth errors.
- Dataset-specific filtering: OmniWorld-Game removes unreliable scenes after validating cross-frame correspondences using SIFT matching and fundamental-matrix RANSAC.
- Dataset-specific filtering: DL3DV removes scenes with rotation above 70° or translation above 15× the scene’s median inter-frame translation, and denoises depth maps.
- Dataset-specific filtering: HyperSim excludes two sequences with completely black RGB frames, while ScanNet++ excludes scenes whose floating mesh artifacts occlude valid depth.
- Dataset-specific filtering: ARKitScenes LiDAR depth maps are not used for supervision because their low resolution is insufficiently accurate.
B Runtime and Memory Evaluation Protocol
The runtime and memory evaluation processes the complete 4,661-frame KITTI-02 sequence at stride 1 on a single NVIDIA H100 GPU using controlled streaming measurements.
- 4,661 frames are processed from the full KITTI-02 sequence at stride 1.
- Measurements use a single NVIDIA H100 GPU and each method’s specified inference precision and input resolution.
- The protocol includes untimed 64-frame warm-up, streaming-state reset, GPU-preloaded inputs, and released chunk-based modes for LongStream and HorizonStream.
C More Results
Additional evaluations test generalization beyond benchmarks across low-viewpoint robot navigation and heterogeneous large-scale real-world videos, with results reported qualitatively where accurate ground truth is unavailable.
- Evaluation settings: The out-of-benchmark evaluation covers low-viewpoint robot navigation and large-scale real-world traversal without accurate ground-truth camera trajectories.These results are intended as qualitative evidence rather than quantitative comparisons.
- Low-viewpoint robot navigation: Low-mounted robot cameras create ground-dominant views with reduced visibility of distant structures, making frame-to-frame correspondence more challenging.
- Low-viewpoint robot navigation: Despite this domain shift, ABot-Recon produces more coherent trajectories across robot-navigation sequences than other methods.The examples provide qualitative evidence of stable trajectory estimates under less frequently represented viewpoints and motion patterns.
- Large-scale real-world sequences: The four large-scale videos span urban driving, campus roads, repeated structures, intersections, and varied camera motion and visual appearance.They complement standard benchmarks with less curated and more heterogeneous long-video streams.
- Large-scale real-world sequences: ABot-Recon produces more consistent large-scale trajectory structure than compared streaming baselines across the four sequences.Its adjacent-relative-transform target remains unchanged as explored area or sequence length increases.
- Large-scale real-world sequences: Lightweight temporal correction suppresses locally correlated rotational errors before they accumulate through pose composition.