Source-linked AI summary
4RC: 4D Reconstruction via Conditional Querying Anytime and Anywhere
Yihang Luo, Shangchen Zhou, Yushi Lan, Xingang Pan, Chen Change Loy
TL;DR
Existing 4D reconstruction methods often decouple geometry and motion or provide limited dynamic attributes. 4RC instead learns a holistic representation from monocular videos, enabling arbitrary spatio-temporal queries, and reports consistent gains across diverse 4D reconstruction benchmarks. Its performance remains constrained by difficult geometry and extreme motion.
Problem
Existing 4D reconstruction approaches often decouple motion from geometry or provide limited attributes such as sparse trajectories, motivating joint modeling of 3D structure and temporal evolution.
Method
4RC uses a unified feed-forward transformer to encode an entire monocular video once into a compact 4D latent, then conditionally queries geometry and motion at arbitrary frames and timestamps.
Results
4RC consistently outperforms prior methods across challenging 4D reconstruction benchmarks and performs competitively across camera pose, depth, point-cloud, tracking, and motion tasks.
Takeaways & Limitations
Joint geometry-and-motion modeling with on-demand querying provides a unified and flexible approach to feed-forward 4D reconstruction.
Takeaways & Limitations
4RC struggles with inherently difficult geometry, including extreme depth, transparent objects, and floating artifacts, and degrades under extreme or chaotic motion.
Abstract
from arXiv · showhide
We present 4RC, a unified feed-forward framework for 4D reconstruction from monocular videos. Unlike existing approaches that typically decouple motion from geometry or produce limited 4D attributes such as sparse trajectories or two-view scene flow, 4RC learns a holistic 4D representation that jointly captures dense scene geometry and motion dynamics. At its core, 4RC introduces a novel encode-once, query-anywhere and anytime paradigm: a transformer backbone encodes the entire video into a compact spatio-temporal latent space, from which a conditional decoder can efficiently query 3D geometry and motion for any query frame at any target timestamp. To facilitate learning, we represent per-view 4D attributes in a minimally factorized form by decomposing them into base geometry and time-dependent relative motion. Extensive experiments demonstrate that 4RC outperforms prior and concurrent methods across a wide range of 4D reconstruction tasks.
1. Introduction
4RC addresses fragmented 4D reconstruction by jointly modeling scene geometry and motion in a unified feed-forward framework. Its compact factorized representation supports efficient querying of geometry and motion for arbitrary frames and timestamps.
- 4D reconstruction jointly models 3D scene structure and temporal evolution, supporting applications such as video synthesis, scene understanding, and robotics.
- Existing methods often separate motion estimation from reconstruction, rely on iterative or post-hoc optimization, or produce only sparse trajectories.
- 4RC uses a unified feed-forward transformer that jointly encodes geometry and motion from monocular videos without auxiliary estimators or per-scene optimization.
- Its encode-once, query-anywhere and anytime paradigm stores the video in a compact 4D latent representation for retrieving dense geometry and motion at arbitrary query frames and target timestamps.
- 4RC minimally factorizes each frame into viewpoint-invariant base geometry and time-dependent relative motion represented as 3D displacements.
- 4RC achieves competitive performance across camera pose estimation, video depth prediction, point cloud reconstruction, 3D point tracking, and dense motion modeling.
2. Related Work
Prior work spans classical and feed-forward 3D reconstruction, 2D and 3D point tracking, and optimization-based or feed-forward 4D reconstruction. Across these areas, a recurring transition is toward direct prediction, while dynamic methods remain constrained by efficiency, temporal scope, or staged processing.
- Classical SfM and MVS recover camera parameters and dense geometry through multi-stage optimization, whereas feed-forward methods predict 3D attributes directly in one neural-network pass.
- DUSt3R and VGGT show that feed-forward architectures can support dense stereo reconstruction, camera pose estimation, and depth prediction across multiple views.
- Optical-flow methods estimate dense pixel displacements but are typically limited to short temporal windows and can suffer drift over long videos.
- Recent 3D point-tracking methods lift 2D tracking using depth estimators, but their multi-stage pipelines remain limited in efficiency and flexibility.
- 4D reconstruction seeks representations that capture both 3D structure and temporal evolution, with early methods relying on costly per-scene test-time optimization.
3. Method
4RC reconstructs monocular videos through a unified, feed-forward 4D representation that separates base geometry from time-dependent motion. A single encoded video latent supports conditional decoding of dense geometry and motion across source frames and target timestamps.
- Overview: 4RC encodes an entire monocular video into a compact 4D representation that can be queried on demand for geometry and motion.The representation supports reconstruction of any query frame at an arbitrary target timestamp.
- 4D Representation: Each time-indexed point map combines base 3D geometry at the source time with a relative displacement to the target time.This factorization avoids separately representing every source-target pair and preserves explicit temporal motion.
- 4D Representation: Recent feed-forward methods have limited output flexibility, whereas 4RC queries dense motion from any single source frame to any target timestamp.The paper contrasts this capability with pairwise, sparse, or jointly attended alternatives.
- 4D Representation Encoder: A transformer encoder alternates frame-wise and global self-attention while appending camera and time tokens to produce ordered spatio-temporal token sets.Time tokens condition subsequent target-time motion decoding.
- Conditional 4D Decoder: The geometry decoder predicts depth, rays, and camera parameters, from which base point maps are obtained under a perspective camera model.The geometry branch uses spatial and camera tokens, while the motion branch operates separately.
- Conditional 4D Decoder: A lightweight motion decoder uses query-frame tokens, target-time embeddings, and cross-attention to output dense 3D displacement fields.Adaptive Layer Normalization injects the target-time condition, and the resulting design supports dense motion estimation and point tracking.
- Training Scheme: 4RC is trained end to end with joint geometry and motion supervision using depth, ray, camera, and motion losses.Aleatoric uncertainty down-weights unreliable regions, while spatial and temporal gradient constraints encourage smooth geometry and temporally consistent motion.
4. Experiments
4RC is evaluated across diverse 4D and 3D reconstruction benchmarks, tracking settings, in-the-wild videos, and ablations of its motion-decoding design. The results show strong performance across tasks, while ablations support the roles of target-time cross-attention and factorized motion.
- Tracking: 4RC achieves state-of-the-art dense tracking among concurrent 4D reconstruction methods and gains 36% in APD over V-DPM on Waymo.The evaluation samples 24 frames and queries the middle view; 4RC uses flexible per-frame decoding rather than global aggregation decoding.
- Tracking: 4RC outperforms SpatialTrackerV2 on 3 of 4 sparse point-tracking datasets, despite comparison with tracking-specific methods.Sparse trajectories are evaluated in world coordinates after global Sim(3) alignment with RANSAC.
- 3D Reconstruction: 4RC provides top-tier camera pose estimation and reconstruction quality, achieving the best ATE and RPEt on TUM-dynamics among all compared methods.The unified model remains competitive with specialized 3D reconstruction methods such as Pi3.
- 3D Reconstruction: 4RC achieves the best δ < 1.25 score on Bonn and strong multi-view reconstruction results, including 0.034 accuracy on 7-Scenes versus TraceAnything’s 0.240.On Sintel, 4RC has a small gap relative to specialized 3D methods trained exclusively on larger 3D datasets.
- Ablation Studies: Ablations show that removing any motion-head component degrades performance, while direct point prediction performs worse than factorized motion representation.Cross-attention removal particularly harms detailed non-rigid motion, and factorization reduces geometry recomputation while improving temporal consistency.
5. Conclusion
4RC is a unified feed-forward transformer framework that learns a compact 4D representation of entire monocular videos for flexible geometry and motion reconstruction. Experiments report consistent improvements over prior methods across challenging 4D reconstruction benchmarks.
- 4RC learns a compact 4D representation of an entire video that can be queried to recover geometry and motion at arbitrary time instances.The paradigm connects global spatio-temporal modeling with on-demand reconstruction.
- 4RC jointly reasons about scene geometry and motion within a unified feed-forward transformer framework.
- 4RC consistently outperforms prior methods across a wide range of challenging 4D reconstruction benchmarks.
Impact Statement
The work aims to advance machine learning for 4D reconstruction, with potential relevance to robotics, augmented and virtual reality, and content creation. Its implementation uses a large pretrained transformer encoder with dedicated geometry and motion components.
- The paper targets advances in machine learning, particularly 4D reconstruction.
- The proposed approach may benefit robotics, augmented/virtual reality, and content creation applications.
- 4RC uses a ViT-Giant encoder initialized from Depth Anything 3, with separate geometry and motion heads.The encoder has 40 transformer layers, feature dimension 1,536, and 24 attention heads.
A.2. Dataset Details
4RC is trained on seven datasets spanning dynamic and static environments, with motion supervision that ranges from dense motion maps to sparse trajectories. Static datasets provide zero-motion annotations.
- 4RC is trained on a mixture of seven datasets covering dynamic and static environments.The training mixture includes four dynamic datasets with ground-truth motion: PointOdyssey, Dynamic Replica, Waymo, and Kubric.
- The motion annotations vary by dataset, ranging from dense motion maps to sparse trajectories.
- Static datasets naturally provide motion annotations corresponding to zero movement.
A.3. Training Details
Training uses standard visual augmentations, temporally ordered frame sampling, and mixed dense or sparse motion supervision. A streaming S-4RC variant supports causal online reconstruction through cached latent representations and per-frame processing.
- A.3. Training Details: Training applies Gaussian blur, ColorJitter, and RandomGrayscale augmentations, while sampling frames in temporal order with intervals of 1 to 5 frames.The augmentation probabilities are 0.2, 0.1, and 0.05, respectively.
- A.3. Training Details: Motion supervision uses all available ground truth in 20% of iterations and sparse supervision in the remaining 80%.
- A.3. Training Details: S-4RC replaces the standard encoder with a pretrained STream3R backbone that enforces unidirectional causal attention.
- A.3. Training Details: S-4RC processes videos sequentially with per-frame latency and caches 4D latents for flexible temporal motion queries and point tracking.
- A.3. Training Details: S-4RC enables online and streaming 4D reconstruction with competitive performance compared to 4RC despite lacking global temporal context.
B.2. Additional Quantitative Evaluation on 4D Reconstruction
4RC is evaluated on dense- and sparse-view point tracking under global median scale alignment, where it outperforms dedicated tracking and concurrent 4D reconstruction methods. It achieves state-of-the-art results on 4 of 6 datasets.
- Global median scale alignment supports evaluation of 4RC on both dense-view and sparse-view tracking protocols.Predictions and ground-truth points share a world coordinate system defined by the first-frame camera.
- 4RC outperforms approaches designed specifically for point tracking and concurrent 4D reconstruction methods.
- 4RC achieves state-of-the-art results on 4 out of 6 datasets.
B.3. Additional Quantitative Evaluation on Depth Estimation
Additional evaluation compares 4RC with single-frame, video-depth, and joint depth-and-pose methods on Bonn, Sintel, and KITTI. The results show stronger depth-estimation performance than existing depth methods and competitive performance against Pi3, while also modeling dynamic object motion.
- 4RC significantly outperforms existing depth estimation approaches and achieves competitive performance compared to Pi3.
- Unlike methods focused solely on 3D reconstruction, 4RC models dynamic object motion without training on large-scale 3D reconstruction datasets.
- Table 8 evaluates video depth on Bonn, Sintel, and KITTI using per-sequence scale and shift alignment.
- Additional visualizations show camera poses, static reconstruction, dynamic reconstruction, and 3D tracking on in-the-wild videos.
C. Limitations
4RC remains constrained by difficult geometric conditions and extreme motion. The authors attribute these limitations to weak geometric boundaries and insufficiently diverse motion supervision.
- Geometric recovery struggles in extreme-depth regions, transparent objects, and floating artifacts with weak depth boundaries.
- Performance degrades in scenes with extreme or highly chaotic motion.The authors link this limitation mainly to insufficient supervision for complex dynamics in existing datasets.
- Future improvements target stronger depth estimation and training data covering broader motion patterns and kinematic diversity.