PaperlayerView exact revision on arXiv

Source-linked AI summary

Geometric Context Transformer for Streaming 3D Reconstruction

Lin-Zhuo Chen, Jian Gao, Yihang Chen, Ka Leong Cheng, Yipengjing Sun, Liangxiao Hu, Nan Xue, Xing Zhu, Yujun Shen, Yao Yao, Yinghao Xu

arXiv:2604.14141v2cs.CV

TL;DR

Streaming 3D reconstruction must retain enough geometric context for accurate, temporally consistent predictions without allowing state and computation to grow uncontrollably. LingBot-Map addresses this with learned Geometric Context Attention that combines anchors, a local pose-reference window, and trajectory memory, achieving strong benchmark performance and approximately 20 FPS inference. Its scope includes mode-dependent trade-offs and limitations from compressed memory, absent loop closure, and no test-time optimization.

  • Problem

    Streaming reconstruction needs rich geometric context for long-term consistency while keeping its state compact and efficient, a balance existing methods handle imperfectly.

  • Method

    LingBot-Map uses end-to-end learned GCA with anchor context, a local pose-reference window, and trajectory memory for coordinate grounding, local geometry, and global consistency.

  • Results

    LingBot-Map achieves state-of-the-art performance across multiple benchmarks and enables stable inference at around 20 FPS.

  • Takeaways & Limitations

    The structured streaming state supports long-range 3D reconstruction while retaining computational efficiency and rich geometric context.

  • Takeaways & Limitations

    Direct mode is preferred within approximately 3,000 frames, while longer VO-mode sequences incur accumulated alignment drift; the model also lacks explicit loop closure and test-time optimization.

Abstract

from arXiv · show

Streaming 3D reconstruction aims to recover 3D information, such as camera poses and point clouds, from a video stream, which necessitates geometric accuracy, temporal consistency, and computational efficiency. Motivated by the principles of Simultaneous Localization and Mapping (SLAM), we introduce LingBot-Map, a feed-forward 3D foundation model for reconstructing scenes from streaming data, built upon a geometric context transformer (GCT) architecture. A defining aspect of LingBot-Map lies in its carefully designed attention mechanism, which integrates an anchor context, a pose-reference window, and a trajectory memory to address coordinate grounding, dense geometric cues, and long-range drift correction, respectively. This design keeps the streaming state compact while retaining rich geometric context, enabling stable efficient inference at around 20 FPS on 518 x 378 resolution inputs over long sequences exceeding 10,000 frames. Extensive evaluations across a variety of benchmarks demonstrate that our approach achieves superior performance compared to both existing streaming and iterative optimization-based approaches.

1 Introduction

LingBot-Map addresses the challenge of streaming 3D reconstruction by selectively managing geometric context for long-term consistency and efficient inference. It combines complementary context types and reports strong benchmark performance across camera pose estimation and dense reconstruction.

  • Streaming reconstruction must balance rich geometric context for long-term consistency with a compact state for efficient inference.
  • Existing approaches trade off state retention, long-range context, memory, and computation through recurrent compression or near-complete historical caching.CUT3R may forget geometric priors, while causal-attention methods retain redundancy as sequences grow.
  • LingBot-Map introduces GCA with anchor context, a local pose-reference window, and trajectory memory for coordinate grounding, dense local geometry, and global consistency.The learned attention framework adaptively weights, encodes, and compresses information within each context type.
  • The training recipe combines progressive training, context parallelism, and relative loss to support stable optimization on long sequences.Evaluation covers Oxford Spires, 7-Scenes, Tanks and Temples, and ETH3D.
  • LingBot-Map achieves state-of-the-art performance across multiple benchmarks, outperforming existing streaming approaches in reconstruction quality and inference speed.

2 Related Work

Prior 3D reconstruction spans optimization-based traditional systems, feed-forward foundation models, and streaming adaptations. Streaming methods remain constrained by hand-crafted pipelines, state forgetting, or growing context and computation.

  • Traditional SfM and SLAM recover camera poses and scene geometry from multi-view observations, while MVS focuses on dense reconstruction.SfM typically operates offline; SLAM processes video streams online and commonly uses bundle adjustment.
  • Feed-forward foundation models directly predict 3D reconstructions or geometric outputs, but early approaches require limited views or optimization-based alignment for additional views.VGGT extends feed-forward reconstruction to more than two views through cross-view attention.
  • Streaming reconstruction methods include hybrid SLAM-based systems and end-to-end feed-forward approaches using recurrent states, test-time training, caching, or causal attention.
  • Existing streaming methods often struggle to maintain performance over long sequences and complex scenes.

3 Method

LingBot-Map processes each arriving frame causally with alternating frame attention and GCA, using structured contexts to preserve geometric information while bounding streaming cost. Its training combines geometric losses, progressive views, and context parallelism, while optimized runtime reaches approximately 20 FPS in the reported setting.

  • Overview: Each arriving frame is encoded by a ViT and processed using alternating frame-wise attention and GCA before task-specific pose and depth prediction.
  • Geometric Context Attention: The streaming state separates coordinate grounding, dense local geometry, and drift correction into anchor context, a recent pose-reference window, and trajectory memory.Anchors establish scale, the local window retains full image tokens, and trajectory memory summarizes older observations compactly.
  • Geometric Context Attention: GCA combines the three contexts in a structured attention mask that retains long-range consistency with bounded per-frame cost.Unlike causal attention, its context does not retain full image-token history.
  • Complexity: For T frames, GCA uses (n + k) · M + 6T context tokens, whereas causal attention uses T · (M + 6).
  • Training: The composite training objective includes depth, absolute-pose, and relative-pose terms, with relative losses computed over already-observed frame pairs in the sliding window.The relative terms use geodesic rotation error and ℓ1 translation error and remain causal.
  • Training: Progressive view training increases sequence length during training, while context parallelism distributes views across GPUs to address cross-frame attention memory costs.
  • Inference: ∼20 FPS is achieved at 518 × 378 resolution for sequences up to 1000 frames with a 64-frame sliding window using FlashInfer.The implementation uses paged KV-cache management and optimized paged and sparse KV attention kernels.

4 Training & Inference

LingBot-Map uses a staged training strategy and compact context management for streaming inference. It offers Direct and VO modes, balancing trajectory accuracy against scalability to very long sequences.

  • Training: A two-stage curriculum trains an offline base model on short diverse multi-view data, then introduces Geometric Context Attention for long-sequence streaming.The curriculum addresses error propagation that destabilizes direct optimization on long sequences.
  • Inference: GCA-based inference maintains anchor, pose-reference window, and trajectory-memory contexts, and adaptive keyframe selection controls KV-cache growth on long inputs.Keyframes are added when predicted pose-and-depth optical flow exceeds a threshold.
  • Training: Progressive view training increases sequence length from 24 to 320 views, while context parallelism distributes views across GPUs to manage quadratic cross-frame attention.The training setup uses Ulysses context parallelism with a parallelism dimension of 16.
  • Training: The training corpus comprises 29 datasets spanning multi-view collections and temporally continuous video sequences, with later training emphasizing long trajectories.A foldback sampler generates temporally coherent subsequences with varying frame rates and no forward-time bias.
  • Inference: Direct Output mode continuously accumulates context and predicts absolute poses and dense depth maps without external alignment, remaining stable for approximately 3,000 frames.This is about 10× the training length of 320 views.
  • Inference: VO mode resets state across overlapping local windows, enabling arbitrarily long sequences with bounded memory but adding alignment drift at window boundaries.Direct mode is preferred within approximately 3,000 frames, whereas VO mode is intended for substantially longer inputs.

5 Evaluation Benchmark

The evaluation benchmark measures camera pose and dense reconstruction across complementary indoor, outdoor, object-centric, and large-scale environments. It combines varied sequence properties with task-specific metrics and alignment procedures.

  • Benchmark scope: The benchmark covers camera pose estimation and 3D reconstruction across five datasets: Oxford Spires, ETH3D, 7-Scenes, Tanks and Temples, and NRGBD.Together, the datasets span indoor and outdoor environments, object-centric captures, and multi-scene trajectories.
  • Datasets: Oxford Spires includes complex transitions, revisits, and scale variation, with sparse 320-frame and dense 3,840-frame settings testing different sequence lengths.Ground-truth camera trajectories come from a high-precision LiDAR-inertial SLAM system.
  • Datasets: ETH3D supplies high-resolution indoor and outdoor scenes with laser-scanner depth, while 7-Scenes contains motion blur, textureless surfaces, and repetitive structures challenging pose estimation.ETH3D uses an F1 threshold of d = 0.1 m.
  • Datasets: Tanks and Temples provides LiDAR depth and ground-truth 3D shapes across six diverse scenes, whereas NRGBD targets cluttered room-scale reconstruction with fine geometric detail.NRGBD dense reconstruction quality is evaluated using the F1 metric.
  • Metrics: Pose evaluation reports AUC at 3° and 30° thresholds, ATE after Sim(3) alignment, and translational and rotational relative pose errors.These metrics cover both global trajectory consistency and local frame-to-frame accuracy.
  • Metrics: Reconstruction evaluation reports F1, Accuracy, and Completeness after Umeyama alignment and ICP refinement, with dataset-specific thresholds and voxel processing.ETH3D uses a 0.25 F1 threshold and 0.039 m voxels; 7-Scenes and NRGBD use a 0.05 threshold after voxel downsampling.

6 Experiments

LingBot-Map is evaluated against offline, optimization-based, and streaming methods for pose estimation and 3D reconstruction. It maintains accuracy on long sequences, generalizes across datasets, and gains efficiency through selective geometric context.

  • Camera Pose Estimation: LingBot-Map achieves the best results on nearly all Oxford Spires sparse-setting metrics, outperforming offline and optimization-based methods.It reaches AUC@15 61.64 versus DA3’s 49.84 and reduces ATE to 6.42 from DA3’s 12.87 and VGGT’s 24.78.
  • Camera Pose Estimation: ATE increases only from 6.42 to 7.11 as Oxford Spires expands from 320 to 3,840 frames, while competing feed-forward methods degrade substantially.CUT3R rises from 18.16 to 32.47 and Wint3R from 21.10 to 32.90.
  • Camera Pose Estimation: Across Tanks and Temples, ETH3D, and 7-Scenes, LingBot-Map achieves leading trajectory accuracy, including AUC@30 92.80 and ATE 0.20 on Tanks and Temples.It also reports ATE 0.22 on ETH3D and 0.08 on 7-Scenes.
  • Qualitative Analysis: Qualitative comparisons show that LingBot-Map preserves trajectories and coherent geometry where competing methods drift, blur, duplicate, or collapse structures.The advantage appears in complex transitions, dark staircases, and challenging multi-building outdoor scenes.
  • 3D Reconstruction: LingBot-Map achieves the best 3D reconstruction F1 scores, including 98.98 on ETH3D and 64.26 on NRGBD.On ETH3D, its F1 exceeds Wint3R’s 77.28 by 21.70 points; on NRGBD, it exceeds Wint3R’s 56.96 by 7.30 points.
  • Ablation Study: Ablations show that anchor initialization, trajectory-memory context tokens, relative pose loss, and Video RoPE each improve long-sequence pose estimation.Anchor initialization improves AUC@3 by 3.83, context tokens by 2.12, and Video RoPE reduces ATE by 1.48.
  • Ablation Study: A bounded pose-reference window improves both efficiency and several accuracy measures over full causal attention.It raises speed from 11.87 to 20.29 FPS, reduces memory from 36.06 to 13.28 GB, and lowers ATE from 6.60 to 5.98.

7 Conclusion and Discussion

LingBot-Map uses Geometric Context Attention to support long-range streaming 3D reconstruction with compact context and real-time inference. The authors report strong benchmark performance while identifying loop closure, fine-grained memory retention, and test-time optimization as remaining limitations.

  • Conclusion: Geometric Context Attention decomposes streaming state into anchor, local pose-reference window, and trajectory memory inspired by classical SLAM.The three contexts are learned end-to-end for coordinate grounding, local geometry, and long-range drift correction.
  • Conclusion: 80× lower per-frame context growth than causal attention enables stable inference over arbitrarily long sequences at around 20 FPS.
  • Conclusion: LingBot-Map achieves state-of-the-art performance among streaming methods and surpasses offline and optimization-based approaches on large-scale datasets such as Oxford Spires.
  • Limitations: The model currently lacks explicit loop-closure detection, fixed-token trajectory compression may lose fine-grained details, and no test-time optimization is performed.These limitations may matter for revisited regions, sequences spanning tens of thousands of frames, and challenging scenarios, respectively.
  • Future Directions: The authors propose bundle-adjustment-like refinement, loop closure, dynamic-scene handling, multimodal inputs, and downstream applications as future directions.

A Data Processing Pipeline

Training LingBot-Map requires a unified pipeline for large-scale, diverse 3D data with accurate camera poses and depth maps. The corpus spans 29 heterogeneous datasets with differing formats, conventions, representations, and quality characteristics.

  • Data Requirements: The training corpus spans 29 datasets from heterogeneous sources with varied formats, coordinate conventions, depth representations, and quality.
  • Pipeline: A multi-stage processing strategy unifies heterogeneous datasets into a consistent training pipeline.
  • Data Requirements: The pipeline is designed to support training a general-purpose streaming 3D reconstruction model using accurate ground-truth camera poses and depth maps.

A.1 Existing Data Processing

The existing data-processing pipeline standardizes 29 public datasets into a common representation for multi-dataset training. It unifies coordinates, depth scales, frame validity, and metadata.

  • Overview: 29 publicly available datasets are standardized into a unified format through preprocessing steps.
  • Coordinate System Unification: Camera poses are converted to a consistent camera-to-world representation, with additional rotations aligning non-standard axes to the OpenCV standard.
  • Depth Scale Normalization: Depth values from formats including millimeter PNGs, centimeter color images, floating-point arrays, and OpenEXR maps are normalized into consistent units.
  • Corrupted Frame Filtering: Corrupted or degenerate frames are filtered using consistency checks, minimum valid-frame thresholds, invalid-depth handling, and outlier removal.
  • Metadata Format Unification: All datasets are serialized into a common pickle metadata structure containing scene lists, frame mappings, image and depth paths, intrinsics, and 4×4 camera trajectories.

A.2 Generation Data Settings

The generation-data settings render normalized 3D assets from Objaverse and Texverse into controlled multiview training scenes. Blender Cycles produces 512 × 512 RGBA images and metric depth with varied spherical camera viewpoints and HDR lighting.

  • Asset Rendering: Objaverse and Texverse assets are rendered using Blender Cycles after scene normalization to [−0.9, 0.9]3.
  • Camera Settings: A pinhole camera captures 512 × 512 images with horizontal field of view sampled uniformly from 40° to 70°.
  • View and Output Settings: Camera positions are sampled on a sphere at multiple elevations and oriented toward the scene center, with HDR lighting and RGBA color plus metric depth rendered in OpenEXR format.

A.3 Gaming Data Processing Pipeline

The gaming data pipeline builds a diverse, accurately annotated source of long-sequence 3D data using varied environments and camera behaviors. It includes indoor and outdoor scenes with multiple navigation and observation patterns.

  • Data acquisition: The runtime acquisition pipeline captures dense visual and geometric annotations from modern game engines for scalable long-sequence 3D data.The sequences maintain smooth camera motion, sufficient inter-frame overlap, and high visual quality without common exposure or motion-blur artifacts.
  • Indoor scenes: The collection protocol covers indoor scenes such as buildings, rooms, and corridors.Indoor trajectories include free roaming, closed-loop roaming, and transitions between interiors or outdoor areas.
  • Outdoor scenes: The protocol also covers outdoor scenes with broader navigation and observation modes.Outdoor behaviors include stochastic free roaming, loop roaming, scene-boundary transitions, dynamic sightseeing, and object orbiting.
  • Outdoor scenes: The dataset includes dynamic sightseeing and object orbiting to capture moving elements and multi-view observations.These patterns expose the reconstruction model to pedestrians, vehicles, animals, and views gathered around objects.

A.4 MatrixCity Data Sequencing

The MatrixCity data is spatially organized rather than temporally continuous, so the pipeline reorganizes aerial and street imagery into view-continuous sequences for sequential reconstruction.

  • Data organization: MatrixCity provides aerial and street-level multi-view images with known camera poses, but its original organization does not provide temporal continuity.Aerial images are sampled on a regular grid, while street images are stored as independent road segments.
  • Sequence construction: The pipeline reorganizes both MatrixCity data types into view-continuous sequences for downstream video generation and sequential 3D reconstruction.This addresses the mismatch between spatial-coverage-oriented storage and temporally continuous input requirements.

A.4.1 Aerial Data

Aerial MatrixCity frames are stored in row-major grid order, which creates discontinuities at row boundaries. The sequencing method instead uses 8-connected random walks, mapped across grid loops, to produce spatially continuous trajectories.

  • Data layout: Aerial data is arranged on an H×W regular grid in row-major order, with each scene potentially containing multiple grid loops.Frame index i maps to grid coordinates through r = ⌊i/W⌋ and c = i mod W.
  • Data layout: Row-major storage does not form a natural camera trajectory because row boundaries can connect spatially distant frames.Within rows, consecutive frames are neighbors, but the last frame of one row may be far from the first frame of the next.
  • Sequencing method: The method treats the grid as an 8-connected graph and performs random walks to generate spatially continuous trajectories.Each step excludes the previous position to suppress immediate backtracking, with a corner fallback when no candidates remain.
  • Sequencing method: Grid coordinates are mapped to global frame indices using the grid-loop index and the number of frames per loop.The mapping is idx = l · Nloop + r · W + c, and the number of generated sequences scales with data size.
Loading 2604.14141v2…