Source-linked AI summary
BAT3R: Robust Online 3D Reconstruction with Bayesian Adaptive State Updates
Lijian Li, Yuanpeng He, Yudian Zheng, Chung-ju Huang, Linyu Li, Dongming Jin, Wenpin Jiao, Zhi Jin, Chi-Man Pun
TL;DR
Long video reconstruction is difficult for global-attention models because their cost grows quadratically, while recurrent streaming methods can accumulate drift. BAT3R combines state anchoring with Bayesian, uncertainty-aware updates; it reports lower camera-pose ATE than CUT3R and TTT3R at similar computational cost.
Problem
Global-attention reconstruction has quadratic cost with sequence length, and recurrent streaming methods can progressively drift without accounting for observation reliability and historical-state confidence.
Source: §1 · pp. 1–2Method
Results
Camera pose ATE falls by up to 67.8% versus CUT3R and 32.3% versus TTT3R, while computational cost remains similar.
Source: §1 · p. 3Takeaways & Limitations
The framework protects well-consolidated regions from redundant updates while retaining plasticity for novel scene content.
Source: §1 · p. 2, §5 · p. 9BASU initializes per-token uncertainty at σ0^2 = 10^3, reflecting complete prior ignorance.
Source: §3.3 · p. 5
Abstract
from arXiv · showhide
Feed-forward transformer models have driven remarkable progress in 3D vision,yet their quadratic complexity renders them impractical for long video sequences. Streaming 3D reconstruction addresses this bottleneck by processing frames sequen?tially with constant memory. However, existing recurrent architectures inevitably suffer from progressive degradation over long sequences, as they fail to account for the intrinsic reliability of observations and the accumulated confidence of the historical state, leading to severe error accumulation. To address these issues, we introduce BAT3R, a training-free Bayesian adaptation framework comprising two synergistic modules: State Innovation Gating (SIG) and Bayesian-guided Adaptive State Updating (BASU). SIG acts as a distributional regularizer that anchors state evolution to its geometric initialization, effectively arresting representational drift during long-term inference. Building on this, BASU reformulates state update as a recursive Bayesian inference process within a Kalman filtering framework. By jointly modeling the temporal reliability of observations via information-theoretic gain and spatial relevance via cross-attention maps, BASU achieves an asymmetric spatio-temporal state fusion: well-consolidated historical regions are protected from redundant updates, while genuinely novel scene content is aggressively incor?porated. Extensive experiments on standard benchmarks (7-Scenes, TUM, ScanNet, KITTI, etc.) demonstrate that BAT3R consistently outperforms state-of-the-art online 3D reconstruction methods, with the most significant gains observed on long sequences and complex scenes.
1 Introduction
Streaming reconstruction avoids the quadratic sequence cost of global-attention transformers, but recurrent methods can accumulate geometric drift as frames arrive. BAT3R combines state anchoring with Bayesian observation fusion and reports improved sequence-length generalization and camera-pose accuracy at similar computational cost.
1 Introduction
- Global-attention transformer reconstruction has quadratic computational and memory cost with sequence length, limiting its practicality for continuous long-video perception.Streaming methods address scalability by processing frames incrementally and compressing historical context into fixed-length recurrent state.Source: §1 · p. 1
- Recurrent streaming methods can suffer progressive distortion of global geometry because continuous state updates do not revisit prior observations.The introduction identifies this as cumulative structural drift over long sequences.Source: §1 · p. 2
- BAT3R is a training-free framework combining State Innovation Gating (SIG), which anchors state evolution to geometric initialization, and Bayesian-guided Adaptive State Updating (BASU), which fuses observations recursively.BASU uses information-theoretic gain and cross-attention to balance temporal reliability and spatial relevance.Source: §1 · p. 2
- Camera pose ATE falls by up to 67.8% versus CUT3R and 32.3% versus TTT3R, while computational cost remains similar.The paper reports these gains alongside improved sequence-length generalization.Source: §1 · pp. 2–3
2 Related Works
Streaming reconstruction methods manage growing video context through causal processing, memory, or recurrent state, while trading off resource use and long-term consistency. Test-time-training methods update persistent states online, and BAT3R is presented as replacing heuristic updates with Bayesian filtering.
2.1 Streaming Online Reconstruction Foundation Models.
- Recurrent architectures such as CUT3R maintain persistent latent memory and stable inference memory usage, but forgetting can degrade long-term consistency as views accumulate.Other approaches include causal transformers, decoder-only models, external spatial memory, and pointer-based memory.Source: §2.1 · p. 3
2.2 Test-Time Training for 3D Reconstruction.
- BAT3R replaces heuristic recurrent updates with a Bayesian filter that models spatial-observation confidence for probabilistic state updates.The related-work discussion places it among training-free adaptive approaches to catastrophic forgetting in recurrent frameworks.Source: §2.2 · p. 3
3 Method
The method starts from TTT3R’s recurrent fast-weight update, then constrains state drift with SIG and performs uncertainty-aware, spatially selective fusion with BASU. BASU uses observation confidence, state uncertainty, and cross-attention relevance to regulate how each token is updated.
3.1 Preliminaries
- TTT3R treats the recurrent state as fast weights updated at inference, with cross-attention supplying the pseudo-gradient and a per-token gate blending candidate and previous states.Its soft-gated updates lack explicit probabilistic bounds and can drift from the initial state or overwrite consolidated memory.Source: §3.1 · pp. 3–4
3.2 State Innovation Gating
- SIG measures each token’s deviation from the first-frame state S0 and adaptively merges the decoder candidate with the previous state to constrain drift.Large deviations favor the historical state; tokens close to initialization pass with minimal modification, while η preserves a baseline level of plasticity.Source: §3.2 · pp. 4–5
3.3 Bayesian-guided Adaptive State Updating
- BASU augments the state with per-token uncertainty in a Gaussian belief, initialized at Σ0 = σ0^2 1_N with σ0^2 = 10^3 to reflect complete prior ignorance.The uncertainty records accumulated observation quality and progressively decreases as reliable frames are integrated.Source: §3.3 · p. 5
- BASU maps prediction-head confidence to observation-noise covariance and uses innovation magnitude for adaptive process noise, so rapid motion loosens the filter while static scenes keep it tight.Well-textured regions receive low observation noise; occluded or textureless areas receive high observation noise.Source: §3.3 · p. 5
- The Kalman gain balances predicted state uncertainty against observation noise, driving stronger updates when the state is uncertain and the observation is reliable.A weight interpolates between the standard ratio form and a square-root form for improved numerical stability across large scale differences.Source: §3.3 · p. 5
- Figure 3 evaluates GPU memory and inference speed on ScanNet across varying numbers of input views.Source: Figure 3 · p. 6
- BASU uses symmetric KL divergence between posterior and prior to quantify information gain, penalizing anomalously large or negligibly small updates while favoring informative ones.The posterior statistics are converted into a per-token temporal reliability gate.Source: §3.3 · p. 6
- Cross-attention-derived spatial relevance modulates temporal reliability, so tokens with reliable observations and high coverage receive the strongest updates while invisible or unreliable tokens are preserved.The updated covariance is carried forward to close the recursive state-estimation loop.Source: §3.3 · p. 6
4 Experiments
Across camera pose, video depth, and 3D reconstruction, BAT3R improves long-sequence results over recurrent baselines. Ablations identify contributions from Bayesian updating, State Innovation Gating, cross-attention, and adaptive process noise.
4.1 Experimental Setup
- Evaluation covers camera pose on TUM dynamics, ScanNet, and Sintel; video depth on Bonn, KITTI, and Sintel; and reconstruction on 7-Scenes and NRGBD.Baselines include recurrent methods, long-sequence extensions, and training-free test-time-training frameworks.Source: §4.1 · p. 7
4.2 Camera Pose Estimation
- At 1000 frames, BAT3R reduces ScanNet ATE by 67.8% vs CUT3R and 32.3% vs TTT3R, and TUM-dynamics ATE by 66.1% and 44.4%, respectively.It maintains competitive GPU memory consumption and inference speed on ScanNet.Source: §4.2 · p. 7, Figure 4 · p. 7
4.3 Video Depth Estimation
- At 500 frames, BAT3R reduces Bonn Abs Rel by approximately 15.5% vs CUT3R, and KITTI Abs Rel by 24.5% vs CUT3R and 13.6% vs TTT3R.Its δ < 1.25 remains more stable and consistently higher as sequence length increases.Source: §4.3 · p. 8, Figure 5 · p. 8
4.4 3D Reconstruction
- At 400 frames on 7-Scenes, BAT3R reduces Chamfer Distance from 0.134 to 0.024 vs CUT3R and from 0.039 to 0.024 vs TTT3R.Normal Consistency is 0.561 vs 0.558 for TTT3R; detailed 7-Scenes and NRGBD results show error reductions across sequence lengths.Source: §4.4 · pp. 8–9, Figure 6 · p. 8
4.5 Ablation Studies
- Removing Bayesian Recursive Update raises ScanNet ATE from 0.058 to 0.099; omitting SIG raises Bonn Abs Rel from 0.061 to 0.065.Excluding cross-attention lowers KITTI δ < 1.25 from 89.2 to 88.1, while removing adaptive process noise raises ATE to 0.077.Source: §4.5 · p. 9, Table 1 · p. 9
5 Conclusion
BAT3R combines training-free Bayesian adaptation with State Innovation Gating and Bayesian-guided Adaptive State Updating for streaming 3D reconstruction. The paper reports consistent gains over online methods, especially on long sequences, while offline full-attention approaches remain constrained by quadratic complexity.
5 Conclusion
- BAT3R uses SIG to anchor state evolution to geometric initialization and BASU to fuse observations with historical confidence through recursive Bayesian inference.The asymmetric fusion protects consolidated regions from redundant updates while incorporating novel scene content.Source: §5 · p. 9
- Experiments across seven benchmarks show consistent improvements over state-of-the-art online methods, with particularly strong gains on long sequences.The framework requires no training or fine-tuning and is described as immediately applicable to pretrained streaming reconstruction models.Source: §5 · p. 9
A.1 Offline Reconstruction Foundation Models.
- Full-attention methods face an O(N^2) bottleneck that can cause out-of-memory errors on consumer GPUs and limit processable sequence length.This constrains offline foundation models for continuous perception despite inference acceleration techniques.Source: §A.1 · p. 16
B.1 Experimental Settings
The supplement distinguishes long-sequence evaluation from short-sequence comparisons with offline or memory-intensive methods. It situates BAT3R among online approaches and reports the short-sequence benchmark context and implementation setup.
B.1 Experimental Settings
- Long-sequence evaluation tests recurrent memory and geometric consistency over hundreds of frames; short-sequence evaluation enables comparisons with offline or memory-intensive baselines that may run out of memory on longer videos.Source: §B.1 · p. 16
- Experiments use a single RTX 4090 GPU and a pretrained CUT3R checkpoint, with no fine-tuning, training, or test-time gradient updates.The Bayesian state variance is initialized to 1e3 per patch.Source: §B.1 · p. 16
B.2 Baselines
- Online comparisons include recurrent methods and long-sequence extensions using pointmap memory or KV caching, while BAT3R adds SIG and BASU without fine-tuning CUT3R or training parameterized components.Source: §B.2 · p. 16
- Table 2 reports short-sequence camera-pose comparisons on Sintel, TUM-dynamic, and ScanNet; TTT3R leads among online methods but remains below strong offline methods.Figure 8 provides qualitative comparisons of eight TUM RGB-D camera trajectories.Source: Table 2 · p. 17, Figure 8 · p. 17
B.3 Camera Pose Estimation
Long-sequence camera-pose results favor BAT3R among online methods on TUM-dynamics and ScanNet, while strong offline methods retain an accuracy advantage at higher memory and processing cost. Qualitative trajectories also show greater fidelity to ground truth than recurrent baselines.
B.3 Camera Pose Estimation
- Evaluation uses Sim(3)-aligned trajectories and reports Absolute Translation Error, Relative Translation Error, and Relative Rotation Error.Source: §B.3 · p. 18
- On long sequences, BAT3R records TUM-dynamics ATE 0.016 and ScanNet ATE 0.058, leading online methods on both datasets.On TUM-dynamics it reduces ATE 65.2% vs CUT3R and 42.9% vs TTT3R; on ScanNet it reduces ATE 41.4% vs CUT3R and 9.4% vs TTT3R.Source: §B.3 · p. 18, Figure 9 · p. 18
- Strong offline methods such as VGGT retain a performance advantage, but use substantially more memory and process more slowly.Source: §B.3 · p. 18
- In trajectory visualizations, BAT3R stays closer to ground truth across motion patterns than TTT3R and CUT3R.The text attributes the observed global consistency to BASU.Source: §B.3 · p. 18
B.4 Video Depth Estimation
BAT3R reports strong results on video depth estimation and long-sequence 3D reconstruction, including substantial error reductions on 7-Scenes and NRGBD. Its depth evaluation spans aligned and metric-scale settings across three datasets.
B.4 Video Depth Estimation
- Video depth evaluation covers KITTI, Sintel, and Bonn, using scale-aligned relative-depth metrics and unaligned metrics for models producing metric-scale pointmaps.The primary metrics are Abs Rel and δ < 1.25 threshold accuracy.Source: §B.4 · p. 18
- On metric-scale KITTI, BAT3R achieves an Abs Rel of 0.099, reducing error by 18.9% versus CUT3R (0.122) and 10.0% versus TTT3R (0.110).It frequently ranks in the top two on Sintel and Bonn; reported comparisons include a 19.2% Abs Rel improvement over CUT3R on scale-invariant Bonn and a 9.6% reduction on metric-scale Sintel.Source: §B.4 · p. 19, Table 3 · p. 19
- Figure 10 examines Average ATE sensitivity to the minimum update floor η, scaling factor κ, and shifting threshold τ.The supplied figure caption identifies the parameters and metric but does not report the sensitivity outcomes.Source: Figure 10 · p. 20
B.5 3D Reconstruction
- At 500 frames, BAT3R’s 7-Scenes Mean Accuracy error is 0.026, about 86.3% below CUT3R (0.190) and 60.6% below TTT3R (0.066).On NRGBD at 500 frames, Mean Accuracy is 0.097 and Mean Completeness is 0.032, corresponding to 41.6% and 63.2% error decreases relative to TTT3R.Source: Table 4 · p. 19, §B.5 · p. 20
B.6 Parameter Analysis
The parameter analysis finds different sensitivities across SIG and BASU settings, while the following theoretical section addresses stability and optimality of Bayesian-guided state transitions.
B.6 Parameter Analysis
- Average ATE is relatively insensitive to SIG’s update floor η, while κ and τ show sharp initial gains that plateau above 1.0 and 5.0, respectively.The reported experiments use TUM-dynamics; Figure 10 presents the three SIG parameter sensitivities.Source: §B.6 · pp. 20–21, Figure 11 · p. 21
- BASU’s relative covariance floor performs best at 0.0, APN scaling α at 1.0, and Kalman Gain interpolation weight ω at 0.2 and 0.6.A slightly higher covariance floor can temporarily spike ATE; lower α fails to accommodate rapid motion, while higher α injects excessive uncertainty.Source: §B.6 · p. 21, Figure 11 · p. 21
C Theoretical Analysis
- The theoretical analysis focuses on the stability and optimality of Bayesian-guided state transition rules.The supplied passage states the scope of the analysis without giving its proofs or results.Source: C Theoretical Analysis · p. 21
C.1 Stability Analysis of SIG
The theoretical analysis characterizes SIG’s anchoring behavior and BASU’s uncertainty control, fusion optimality, and information-sensitive updating.
C.1 Stability Analysis of SIG
- SIG penalizes deviations from the initial embedding S0, keeping the latent state within a bounded region of the decoder’s reliable manifold.Its gate shifts weight toward the previous state when a candidate exhibits anomalous drift, preventing the state from being driven solely by noisy observations.Source: §C.1 · p. 22
C.2 Geometric Manifold Consistency
- Under bounded process and observation noise covariances, BASU’s posterior state covariance is strictly bounded by the maximum observation noise.The result holds across sequence lengths when incoming observations follow the confidence-driven noise model, so representational uncertainty does not grow with sequence length.Source: §C.2 · p. 22
C.3 Bayesian Fusion Optimality
- Assuming Gaussian state innovation, BASU’s Kalman gain yields a Best Linear Unbiased Estimator that minimizes expected state-estimation MSE.BAT3R uses a square-root-stabilized variant to handle scale mismatches in high-dimensional transformer latent spaces while retaining the MSE-minimizing objective.Source: §C.3 · pp. 22–23
C.4 Information-Theoretic Gain and Update
- The symmetric-KL information gain is minimized for observations with no novel information and maximized for significant, reliable structural change.The analysis describes suppression of redundant observations and anomalous outliers, preserving state plasticity for informative updates consistent with scene geometry.Source: §C.4 · p. 23