Source-linked AI summary
3D Reconstruction with Spatial Memory
Hengyi Wang, Lourdes Agapito
TL;DR
Dense reconstruction pipelines and DUSt3R require sequential processing or optimization-based alignment to obtain globally consistent geometry. Spann3R uses transformer-based spatial memory to predict globally coordinated pointmaps incrementally, achieving competitive reconstruction and generalization while enabling real-time online processing. Its scope remains constrained for large-scale, continuously forward-moving or multi-room scenes and by limited training data and resolution.
Problem
Existing reconstruction approaches retain complex noise-sensitive pipelines or require optimization-based alignment to combine local pointmaps globally.
Method
Spann3R uses an external spatial memory and transformer-based architecture to query previous 3D predictions and regress each next-frame pointmap in a common coordinate system.
Results
Spann3R achieves competitive reconstruction quality and generalization across varied and unseen datasets while enabling online reconstruction at over 50 fps without test-time optimization.
Takeaways & Limitations
The approach supports real-time incremental reconstruction from RGB images without prior camera parameters or test-time optimization.
Takeaways & Limitations
The model may fail on continuously forward-moving or large multi-room scenes because training uses limited memory and five-frame, limited-region sequences.
Abstract
from arXiv · showhide
We present Spann3R, a novel approach for dense 3D reconstruction from ordered or unordered image collections. Built on the DUSt3R paradigm, Spann3R uses a transformer-based architecture to directly regress pointmaps from images without any prior knowledge of the scene or camera parameters. Unlike DUSt3R, which predicts per image-pair pointmaps each expressed in its local coordinate frame, Spann3R can predict per-image pointmaps expressed in a global coordinate system, thus eliminating the need for optimization-based global alignment. The key idea of Spann3R is to manage an external spatial memory that learns to keep track of all previous relevant 3D information. Spann3R then queries this spatial memory to predict the 3D structure of the next frame in a global coordinate system. Taking advantage of DUSt3R's pre-trained weights, and further fine-tuning on a subset of datasets, Spann3R shows competitive performance and generalization ability on various unseen datasets and can process ordered image collections in real time. Project page: \url{https://hengyiwang.github.io/projects/spanner}
1. Introduction
Spann3R addresses the complexity and optimization burden of conventional learned reconstruction pipelines by using spatial memory to predict globally aligned pointmaps incrementally. It builds on DUSt3R and supports real-time online reconstruction without test-time alignment.
- 1. Introduction: Learned reconstruction methods retain sequential pipelines whose sensitivity to noise and engineering demands remain limiting.These pipelines include matching, triangulation, sparse reconstruction, camera estimation, and dense reconstruction.
- 1. Introduction: DUSt3R directly regresses pointmaps from image pairs, but requires per-scene optimization to align local-coordinate predictions globally.Its predictions are expressed in each image pair’s local coordinate system.
- 1. Introduction: Spann3R maintains external spatial memory of previous states and predicts each new image’s pointmap in a common coordinate system.The model queries relevant stored information to predict the next frame.
- 1. Introduction: The method uses a transformer-based memory encoder, query and key projections, and curriculum training on five-frame video subsequences.Memory management maintains a compact representation during inference while learning short- and long-term dependencies.
- 1. Introduction: Spann3R enables online incremental reconstruction at over 50 fps without test-time optimization and achieves competitive quality and generalization on unseen datasets.The approach is evaluated on multiple datasets after training on a subset of available data.
2. Related Works
Prior reconstruction systems either use sequential geometric pipelines, DUSt3R’s pairwise regression with global alignment, or optimization-heavy neural rendering. Spann3R replaces DUSt3R’s optimization step with learned spatial memory for real-time incremental reconstruction.
- 2. Related Works: Classic SfM and multi-view stereo use sequential correspondence, registration, triangulation, bundle adjustment, and dense reconstruction stages.These pipelines are vulnerable to noise propagating across subtasks.
- 2. Related Works: DUSt3R unifies reconstruction subtasks by directly mapping image pairs to 3D, then uses optimization-based global alignment across pairs.Spann3R further replaces this optimization with end-to-end learning.
- 2. Related Works: Spann3R adopts memory-based processing to enable online incremental reconstruction in real time.Its design is positioned as a learned replacement for DUSt3R’s global alignment stage.
- 2. Related Works: Neural rendering methods can reconstruct high-fidelity scenes but generally require known camera parameters and lengthy optimization.The cited methods include NeRF and Gaussian splatting-based approaches.
- 2. Related Works: External memory networks model long-term dependencies and have been adapted from natural-language reasoning to sequential vision tasks.Spann3R is inspired particularly by memory-based video object segmentation methods.
3. Method
Spann3R encodes prior predictions in a spatial memory, retrieves them with cross-attention, and decodes the next pointmap in a shared coordinate system. Dense working and sparse long-term memory support sequential inference, with clipping and curriculum training improving robustness and temporal coverage.
- 3. Method: Spann3R maps each input frame to a pointmap expressed in the initial frame’s coordinate system using spatial memory of previous predictions.The network is designed for ordered sequences and next-frame reasoning.
- 3.1. Network architecture: A ViT encodes each frame, while intertwined target and reference decoders produce query features and memory-conditioned geometric predictions.The target decoder supplies query features; the reference decoder predicts from memory readout and geometric features.
- 3.2. Spatial memory: The model encodes previous decoder features and pointmaps into memory keys and values, then retrieves fused features through query-key cross-attention.Memory features combine visual and geometric information, allowing readout based on appearance and distance.
- 3.2. Spatial memory: The spatial memory combines dense recent-frame working memory with sparse long-term memory for compact sequential storage.Working memory retains the most recent five frames and drains older features into long-term memory when full.
- 3.3. Training and Inference: Training uses five randomly sampled frames with curriculum-adjusted temporal windows, while unordered collections can be ordered using confidence-based graph strategies.Inference can use a minimum spanning tree or predicted confidence to select the next image.
4. Experiments
Spann3R is evaluated on unseen datasets, ablations, online reconstruction, attention behavior, and generalization. It achieves competitive quality without test-time optimization, but accumulated errors, limited resolution, and training-memory constraints remain important boundaries.
- Scene-level reconstruction: Spann3R provides competitive online reconstruction quality while being significantly faster than offline methods requiring optimization-based alignment.Its common-coordinate pointmaps avoid test-time optimization; few-view performance is on par with DUSt3R†.
- Scene-level reconstruction: 224×224 training creates a performance gap relative to DUSt3R using 512 × 384 images, especially on NRGBD scenes with many thin structures.The comparison is affected by input resolution, while DUSt3R’s original-resolution evaluation may benefit from increased visual overlap.
- Spatial-memory ablation: Without long-term memory, the model drifts quickly; clipping small attention weights improves robustness against outlier memory values.The working memory contains the most recent 5 frames, and 4000 long-term memory tokens are sufficient for most scenes.
- Online reconstruction: Online reconstruction handles challenging textureless walls by exploiting indoor regularity, but loop closing may leave geometry inaccurate because of accumulated errors and outliers.Noisy points appear around windows in the reported indoor examples.
- Analysis and generalization: Attention retrieval distinguishes visually similar patches at different locations, supporting robustness across challenging scenes and datasets.Examples include the toy’s right eye and foot; qualitative results span indoor, outdoor, object-level, and scene-level datasets.
- Limitations: Large-scale forward-moving or multi-room reconstruction may fail because five-frame training spans only a limited spatial region and constrains memory size.Suggested directions include scalable training sampling or a more structured inference-time memory system.
- Limitations: Training uses four datasets and five 224×224 RGB-D images sampled from sequences, leaving room for broader, higher-resolution, or self-supervised training.The authors note reliance on a substantial amount of posed RGB-D data.
5. Conclusion
Spann3R performs incremental RGB-image reconstruction without camera parameters by using spatial memory for next-frame prediction. The resulting forward-pass pipeline enables real-time online reconstruction while retaining competitive quality and generalization.
- Conclusion: Spann3R reconstructs incrementally from RGB images without prior camera parameters.Its spatial memory encodes previous states for predicting the next frame.
- Conclusion: A transformer-based forward pass eliminates test-time optimization and enables real-time online reconstruction.The model predicts each next-frame pointmap using information stored in spatial memory.
- Conclusion: The model demonstrates competitive reconstruction quality and generalization across varied scenarios.Future work targets large-scale scenes, bundle adjustment, and self-supervised learning from casual videos.
6. Additional details
Spann3R trains with confidence-aware regression and scale losses, while curriculum sampling adjusts frame intervals to support varied temporal dependencies.
- Training loss: The confidence-aware loss weights valid-pixel regression using learned per-pixel confidence scores.Confidence is an exponential function of the network’s raw output.
- Training loss: The confidence loss encourages greater weight for pixels with smaller depths rather than explicitly encoding inverse depth.The learned weighting function is controlled by α.
- Training loss: The scale loss penalizes predictions whose average distance from the origin exceeds the ground-truth scale.This discourages trivial solutions with overly large predicted scale.
- Curriculum training: The hyperparameter α is selected so the overall training loss becomes negative after 30% of epochs, with α ≥0.4 achieving the best reported results.Insufficient α is associated with a large scale loss during training.
- Curriculum training: The training uses a curriculum that changes the sampling interval between adjacent frames.The interval is interpolated between Tmin and Tmax using the active ratio ηa.
7. Additional analysis
Additional analyses compare confidence-based view selection and DUSt3R-derived components, showing competitive indoor reconstruction while identifying challenging-scene and dataset-specific weaknesses.
- View selection: Offline view selection replaces exponential confidence with a sigmoid function because exponential confidence over-weights highly confident patches.The two alternatives are compared in Table 4.
- DUSt3R ablation: DUSt3Rours has on-par median accuracy and completion with consistently better normal consistency than DUSt3R† on indoor scenes.The comparison reuses DUSt3R’s encoder and repurposes its two decoders.
- Per-scene performance: On DTU, weaker results may reflect limited object-centric training scenes and random sampling instead of DUSt3R’s internal pair-selection model.These are presented as possible explanations for the dataset-specific gap.
- Per-scene performance: Spann3R achieves competitive per-scene results compared with DUSt3R, but challenging scenes can produce more outliers.The NRGBD mirror scene is reported as an example.