Source-linked AI summary
Continuous 3D Perception Model with Persistent State
Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A. Efros, Angjoo Kanazawa
TL;DR
The paper addresses online 3D perception from continuously arriving observations by introducing a persistent-state framework that updates and reads scene information at every step. It produces world-frame pointmaps and camera parameters for online reconstruction, supports unseen-region inference, and achieves competitive or state-of-the-art performance across multiple 3D tasks.
Problem
Online 3D perception must reconcile prior knowledge with continuously arriving observations while handling sparse inputs, dynamic objects, and unobserved scene regions.
Method
The framework maintains a persistent recurrent state, jointly updates and reads it for each image, and outputs world-frame pointmaps and camera parameters that accumulate into dense reconstructions.
Results
The method achieves competitive or state-of-the-art performance across monocular and consistent video depth estimation, camera pose estimation, and 3D reconstruction, while inferring unseen structures and refining reconstructions online.
Takeaways & Limitations
A single online formulation handles videos and unordered photo collections, static and dynamic scenes, sparse observations, and metric-scale inference of unseen structures.
Takeaways & Limitations
Very long sequences may drift without global alignment, deterministic structure generation can become blurry for distant viewpoints, and recurrent-network training is time-consuming.
Abstract
from arXiv · showhide
We present a unified framework capable of solving a broad range of 3D tasks. Our approach features a stateful recurrent model that continuously updates its state representation with each new observation. Given a stream of images, this evolving state can be used to generate metric-scale pointmaps (per-pixel 3D points) for each new input in an online fashion. These pointmaps reside within a common coordinate system, and can be accumulated into a coherent, dense scene reconstruction that updates as new images arrive. Our model, called CUT3R (Continuous Updating Transformer for 3D Reconstruction), captures rich priors of real-world scenes: not only can it predict accurate pointmaps from image observations, but it can also infer unseen regions of the scene by probing at virtual, unobserved views. Our method is simple yet highly flexible, naturally accepting varying lengths of images that may be either video streams or unordered photo collections, containing both static and dynamic content. We evaluate our method on various 3D/4D tasks and demonstrate competitive or state-of-the-art performance in each. Project Page: https://cut3r.github.io/
1. Introduction
The paper introduces an online 3D perception framework that combines learned scene priors with a recurrent state to reconstruct, refine, and extrapolate 3D scenes from image streams. It produces common-frame pointmaps online and supports virtual-view inference of unseen regions.
- The framework unifies reconstruction from few observations, continuous refinement with additional observations, and inference of unobserved 3D regions.
- A recurrent model maintains and incrementally updates a persistent scene state while predicting dense pointmaps and camera parameters for each observation.Pointmaps contain a 3D point per pixel in a world coordinate frame.
- Accumulated world-frame pointmaps enable an online dense reconstruction that improves as new images arrive.
- A virtual camera query, represented as a raymap, lets the model extract pointmaps and colors for unseen scene views.
- The framework trains on diverse 3D data and accepts varying numbers of images from videos or photo collections across static and dynamic scenes.
- Across monocular depth, video depth, camera pose, and 3D reconstruction tasks, the method achieves competitive or state-of-the-art performance.The evaluation also tests unseen-structure inference and continuous reconstruction refinement.
2. Related Work
Related work spans tabula rasa reconstruction, learning-based systems, continuous online reconstruction, dynamic-scene methods, and 3D scene-prior approaches. The paper distinguishes its method through a persistent state that supports both online reconstruction and unobserved-structure inference.
- Tabula rasa 3D Reconstruction: Tabula rasa methods reconstruct each scene from current observations, making sparse or ill-posed settings challenging.Examples include SfM, SLAM, NeRF, and 3D Gaussian Splatting.
- Learning-Based 3D Reconstruction: Learning-based reconstruction methods add data-driven priors to components, pipelines, or end-to-end systems, while also predicting dense 3D geometry.
- Continuous Reconstruction Methods: Existing continuous reconstruction methods include monocular SLAM and recurrent approaches, but commonly require known intrinsics, posed images, or object-centric settings.
- Continuous Reconstruction Methods: Unlike a spatial-memory cache, the paper’s compressed state captures observed content and enables inference of unobserved structures.
- Reconstructing Dynamic Scenes from Monocular Videos: Dynamic-scene reconstruction methods can rely on time-consuming per-video optimization to align depth across moving content.
- 3D Scene Priors: Regression-based few-shot novel-view methods often generalize across a class of scenes, while newer work transfers image and video priors into 3D.
3. Method
CUT3R processes image streams through a persistent state that is simultaneously updated and read to produce metric 3D representations, poses, and reconstructions. The state can also be queried from virtual views to infer unseen scene regions without adding new scene content.
- Online reconstruction: Accumulating world-frame pointmaps over incoming views produces an online dense reconstruction that updates as observations arrive.The recurrent formulation accepts image streams from videos or photo collections and varying numbers of images.
- State-input interaction: Each image is encoded into tokens that interact bidirectionally with persistent state tokens through interconnected transformer decoders.State-update integrates current-image information, while state-readout retrieves stored context for prediction.
- 3D and pose outputs: The model predicts pointmaps in camera and world frames, confidence maps, and 6-DoF camera-to-world poses in metric scale.World pointmaps use the initial image’s coordinate frame, enabling views to share a common reference.
- Unseen-view querying: A virtual camera represented by a raymap queries the state to predict pointmaps and colors for unobserved scene regions without updating the state.The raymap encodes each pixel’s ray origin and direction; shared decoder weights process the query against the current state.
- Training objective: The objective combines confidence-aware pointmap regression, pose supervision, and RGB reconstruction for raymap inputs.Metric ground-truth pointmaps use shared scale normalization so the model learns metric-scale predictions.
- Training strategy: Training uses diverse real and synthetic datasets, including static and dynamic scenes and partial annotations, with staged curriculum training.The model is trained on 32 datasets, and later training emphasizes multi-view scene-level reasoning.
4. Experiments
CUT3R is evaluated across depth estimation, camera pose estimation, and 3D reconstruction, with competitive or state-of-the-art performance while operating online. Additional analyses test reconstruction refinement through state updates and inference of unseen scene regions.
- Monocular and Video Depth Estimation: CUT3R achieves state-of-the-art or competitive monocular depth performance across Sintel, Bonn, KITTI, and NYU-v2.The evaluation uses absolute relative error and δ < 1.25, with per-frame median scaling.
- Monocular and Video Depth Estimation: CUT3R consistently outperforms DUSt3R and MASt3R under per-sequence scale alignment for video depth estimation.
- Camera Pose Estimation: CUT3R performs comparably or better than MonST3R on KITTI while remaining online and achieving nearly 50× speedup.It also significantly outperforms MASt3R for most metrics in the metric-scale setting.
- Camera Pose Estimation: CUT3R achieves the best overall performance among online camera-pose methods, particularly in dynamic scenes.The comparison covers Sintel, TUM-dynamic, and ScanNet, while optimization-based methods retain an overall advantage over online methods.
- 3D Reconstruction: On 7-Scenes and NRGBD, CUT3R significantly outperforms Spann3R and matches or surpasses DUSt3R-GA while operating online at 25× the speed.The reconstruction evaluation uses sparse image collections with 3–5 frames for 7-Scenes and 2–4 frames for NRGBD.
- 3D Reconstruction: CUT3R operates online across static and dynamic scenes, avoiding static-scene global alignment assumptions that can impair moving-object reconstruction.Its implicit state-based alignment remains online and adapts to both scene types.
- Analysis: Revisiting improves performance over online processing, especially accuracy, by letting the state access full scene context before generating predictions.This analysis evaluates whether the persistent state updates effectively with additional observations.
- Analysis: CUT3R infers unseen metric-scale structures from single or multiple views without requiring input camera intrinsics or poses.On unseen MapFree and ARKitScenes validation scenes, state readout supports viewpoint changes and generates structures absent from the input.
5. Conclusion
CUT3R uses a continuously updating persistent state to support online 3D perception across videos and photo collections, including static and dynamic scenes. Its state also enables inference of unseen structures, while long-sequence drift, blurry far-view extrapolation, and training cost remain limitations.
- CUT3R updates a persistent state and reads from it at every observation to predict camera parameters and world-frame pointmaps.Accumulated pointmaps form a dense scene reconstruction over time.
- The framework handles varying-length videos and photo collections across static and dynamic scenes while supporting multiple 3D/4D tasks.
- State probing with a raymap enables inference of new structures that were unobserved in the input views.
- Very long sequences may drift without global alignment, and deterministic structure generation can become blurry when viewpoints are far from those provided.
- Recurrent-network training can be time-consuming.
A. Training Datasets
The model is trained on diverse datasets spanning scene types, modalities, and annotation regimes. Several datasets require reconstructed dense depth, camera-only supervision, filtering, or special treatment of single-view data.
- The training data covers static and dynamic environments, indoor and outdoor scenes, and object-centric scenarios.
- Dense depth maps are generated by multi-view stereo for MapFree and DL3DV, while three other datasets supervise only camera prediction.RealEstate10K, CoP3D, and MVImgNet lack dense depth maps and use provided camera parameters for camera supervision.
- Independent views from several single-view datasets are stacked into sequences, with the recurrent state reset after each view.This permits joint training of multi-view and single-view data within the same batch.
- PointOdyssey scenes with incorrect depth or unrealistic motion and material are filtered, and BEDLAM scenes with panorama backgrounds are removed.
B. More Implementation Details
Training combines video sequences and unordered photo collections with dataset-specific sampling and staged supervision. Metric-scale raymap inputs are used selectively, while the architecture injects pose information into world-coordinate pointmap prediction.
- Training data combines video sequences and unordered photo collections, with video frames subsampled at dataset-dependent random intervals.
- Frames are fed to the model in temporal order using a fixed interval.
- Metric-scale sequences randomly replace 20% of noninitial frames with raymap inputs using ground-truth intrinsics and poses.Raymap mode is disabled for unknown-scale annotations to avoid scale inconsistency.
- Headworld uses pose-token modulation in self-attention and then produces the final world-frame pointmap through a linear or DPT head.
- The pose token has dimension 768, and Headpose is a two-layer MLP with hidden size 768.
- Training proceeds in stages, with long-context training excluding single-view datasets and using only multi-view datasets.
C. More Comparisons
The supplementary comparisons broaden evaluation across video depth and camera-pose estimation, using diverse single-frame, video, joint, learning-based, and optimization-based baselines. The training-dataset table documents annotation categories used to characterize the data.
- Video Depth Estimation: Video-depth evaluation compares single-frame depth, video-depth, and joint depth-and-pose methods.The reported baselines include Marigold, Depth-Anything-V2, NVDS, ChronoDepth, DepthCrafter, Robust-CVD, CasualSAM, DUSt3R, MASt3R, MonST3R, and Spann3R.
- Training Datasets: Table 6 classifies datasets as dynamic, camera only, or single view according to their available annotations.
- Camera Pose Estimation: Camera-pose evaluation includes learning-based visual odometry, optimization-based methods, online methods, and DUSt3R without global alignment.The results are presented in Table 8.
Sintel BONN KITTI
The paper evaluates video depth across Sintel, Bonn, and KITTI, and camera pose estimation across Sintel, TUM-dynamic, and ScanNet.
- Video Depth Evaluation: Video depth evaluation covers Sintel, Bonn, and KITTI using scale&shift-invariant, scale-invariant, and metric depth accuracy.The evaluation also reports KITTI FPS at 512×144 resolution, except Spann3R at 224×224.
- Camera Pose Estimation: Camera pose estimation is evaluated on Sintel, TUM-dynamic, and ScanNet.
- Camera Pose Estimation: Some camera-pose methods require ground-truth camera intrinsics, unlike the remaining methods.