Source-linked AI summary
PointVGGT: Zero-Shot Multiview RGB-D Point Cloud Registration with Visual Geometry Foundation Priors
Haobo Jiang, Liang Yu, Jianmin Zheng
TL;DR
Multiview RGB-D registration needs globally aligned poses, but existing methods often use RGB only as an auxiliary matching cue, leaving visual geometry priors underused. PointVGGT grounds foundation-model poses with sensor depth, builds dense correspondences using voxelized spatial hashing, and refines poses with robust bundle adjustment. It reports zero-shot performance across six benchmarks and inference times reduced from minutes to seconds.
Problem
Existing methods often use RGB only as an auxiliary matching cue, while multiview RGB-D registration remains unreliable and computationally expensive in challenging real-world conditions.
Source: §1 · pp. 2–3Method
PointVGGT grounds scale-ambiguous foundation-model poses with sensor depth, constructs correspondences through voxelized spatial hashing, and refines poses using robust motion-only bundle adjustment.
Source: §1 · p. 3Results
PointVGGT reports outstanding zero-shot performance across six benchmarks and reduces inference time from minutes to seconds.
Source: §5 · p. 16Takeaways & Limitations
The framework is plug-and-play with diverse VGGT-style foundation models and can benefit from their continued advances.
Source: §1 · p. 3In unconstrained real-world settings, feed-forward reconstructions may retain residual scale ambiguity or cross-view scale inconsistency.
Source: §3.1 · p. 6
Abstract
from arXiv · showhide
This paper addresses multiview RGB-D point cloud registration, aiming to estimate global rigid poses for unordered RGB-D scans and align them in a metrically consistent coordinate frame. The conventional pairwise-then-global paradigm suffers from locally optimized pairwise registration, severe error propagation and high computational burden. In particular, existing methods typically treat RGB data as a mere auxiliary matching cue and overlook the holistic geometric priors (e.g., camera poses and 3D models) encoded across image sequences. This paper introduces PointVGGT, a zero-shot framework built upon a novel \emph{foundation-then-refinement} paradigm that systematically leverages visual geometry foundation models (e.g., VGGT) as the computational backbone for robust, training-free multiview RGB-D registration. In the foundation stage, we directly recover metrically consistent global poses (without any pairwise estimation) by grounding the scale-ambiguous pose predictions of the foundation model against metric depth observations. In the refinement stage, we introduce an efficient voxelized spatial hashing mechanism that exploits the globally coherent 3D reconstruction (induced by the foundation model) as a shared spatial anchor, enabling dense multiview correspondences in near-linear time. On top of this, an IRLS-based robust motion-only bundle adjustment is performed using a conjugate gradient solver to jointly minimize the correspondence and reprojection residuals for multiview pose refinement. Extensive experiments on indoor/object-centric/outdoor datasets verify the outstanding zero-shot registration accuracy and computational efficiency of our proposed method.
1 Introduction
PointVGGT addresses the cost and error propagation of pairwise-then-global registration by using visual geometry foundation priors for zero-shot multiview RGB-D alignment. It grounds foundation-model poses with sensor depth, builds correspondences through voxelized spatial hashing, and refines poses with robust bundle adjustment.
1 Introduction
- Pairwise-then-global methods face local optima and high pairwise-registration cost, error propagation during pose synchronization, and limited domain generalization or finetuning demands in some learning-based methods.Real-world low overlap, repetitive structures, and noisy measurements also make registration unreliable and computationally expensive.Source: §1 · p. 2
- The framework uses visual geometry priors encoded across RGB sequences, where foundation models jointly predict dense 3D pointmaps and camera poses in a feed-forward pass.This contrasts with treating RGB images merely as auxiliary pairwise-matching cues.Source: §1 · pp. 2–3
- PointVGGT initializes metrically grounded global poses by aligning scale-ambiguous foundation-model pose predictions with sensor depth, avoiding pairwise registration.It then uses voxelized spatial hashing for dense multiview correspondences and motion-only bundle adjustment with Geman-McClure IRLS and conjugate gradients to minimize correspondence and reprojection residuals.Source: §1 · p. 3
- PointVGGT reports strong zero-shot registration accuracy and reduces inference time from minutes to seconds while outperforming classical and learning-based approaches without task-specific training or finetuning.The pipeline eliminates pairwise registration and uses holistic pose and geometric priors from RGB inputs.Source: §1 · p. 3
- The foundation-then-refinement paradigm is plug-and-play across VGGT-style models, including VGGT, Pi3, and DepthAnything3, and can benefit as those models advance.The paper presents this as enabling sustained performance improvements as foundation models evolve.Source: §1 · p. 3
2 Related Work
Prior work spans pairwise geometric and RGB-D matching, pairwise-to-global multiview synchronization, and visual geometry foundation models. PointVGGT uses foundation-model global poses followed by holistic multiview refinement instead of pairwise registration and pose-graph synchronization.
Pairwise Point Cloud Registration
- Pairwise registration research progresses from handcrafted local geometric descriptors to learned feature matching and correspondence refinement.The cited approaches include local surface descriptors, coarse-to-fine matching, overlap-aware attention, and geometric embeddings.Source: §2 · pp. 3–4
Pairwise RGB-D Point Cloud Registration
- Pairwise RGB-D methods fuse color and geometry through optimization constraints, enriched correspondences, learned cross-modal features, or global scene models.The literature also includes pretrained visual representations for zero-shot RGB-D matching; multiview registration must reason across views jointly.Source: §2 · p. 4
Multiview Point Cloud Registration
- Multiview methods commonly estimate pairwise poses before global synchronization, while learned components improve edge weighting, graph construction, or pose refinement.Synchronization remains vulnerable to corrupted pairwise estimates and propagated errors.Source: §2 · p. 4
- FUSER (Jiang et al, 2025a) alleviates pairwise cost but remains geometry-only; PointVGGT instead predicts global poses in one feed-forward pass and refines them holistically.This avoids pose-graph establishment and synchronization error propagation.Source: §2 · p. 5
Visual Geometry Foundation Models
- Visual geometry foundation models increasingly predict camera geometry and 3D scene structure from image sets in feed-forward pipelines.The described family ranges from pairwise point-map prediction to joint many-view reconstruction and scene-level geometric attributes.Source: §2 · p. 5
3 Methodology
PointVGGT replaces pairwise-then-global registration with a foundation-then-refinement pipeline: it grounds global pose priors from RGB-D depth, then refines poses using cross-view correspondences and motion-only bundle adjustment.
3.1 Problem Statement
- Registration estimates globally consistent rigid poses for unordered, partially overlapping RGB-D scans; RGB-only foundation-model translations and scene geometry can remain scale-ambiguous or cross-view inconsistent.Conventional methods estimate pairwise poses and synchronize them globally, while RGB-only observations do not provide absolute metric depth cues.Source: §3.1 · pp. 5–6
3.2 Motivation
- PointVGGT addresses underused multiview visual geometry and pairwise-registration limitations with foundation-driven global initialization and joint pose refinement.The cited limitations include local optima, error propagation, and computational overhead.Source: §3.2 · pp. 6–7
3.3 Foundation-Driven Fast Pose Initialization
- The foundation model processes the RGB sequence jointly; rotations are adopted directly, while a median of pointmap-to-depth radial ratios estimates scale for metric translations without pairwise alignment.The scale estimator is described as robust to depth noise, prediction errors, and occasional mismatches.Source: §3.3 · p. 7
3.4.1 Voxelized Spatial Hashing for Fast Cross-view Correspondence Construction
- Pointmaps serve as 3D descriptors for voxel hashing; searching neighboring voxels addresses boundary artifacts and reduces matching complexity from O(S^2N^2) to O(SN).The method groups points by voxel and applies view constraints and a distance threshold when matching candidates.Source: §3.4.1 · pp. 7–8
3.4.2 Geometry-aware Robust Alignment Loss
- Motion-only bundle adjustment jointly fits matched-point and depth-reprojection residuals with confidence weighting and a Geman-McClure kernel, fixing the first pose as reference.The reprojection cue compares a transformed source point with a target point reconstructed from sampled target depth.Source: §3.4.2 · pp. 8–9
3.4.3 IRLS-based Motion-only Bundle Adjustment
- IRLS reformulates the robust objective as weighted least squares; Gauss-Newton linearizes residuals, and damped conjugate gradient solves updates until the relative residual falls below a preset threshold.The damping coefficient is used to improve numerical stability.Source: §3.4.3 · pp. 9–10
4 Experiments
Experiments evaluate PointVGGT on indoor, object-centric, and outdoor benchmarks, alongside ablations of its initialization, refinement, and matching components. The reported results show strong registration accuracy and efficiency across these settings.
4.1 Experimental Settings
- Evaluation covers 14 ScanNet scenes and 14 ScanNet++ sequences, using recall at varying error thresholds plus mean and median errors to assess precision and robustness.ScanNet uses 30 point clouds per scene sampled at 20-frame intervals; ScanNet++ retains up to 60 keyframes per sequence.Source: §4.1 · p. 10, §4.2.1 · pp. 10, 12, 13
4.2.1 Evaluation on Indoor Datasets
- On ScanNet, PointVGGT (Pi3) reaches 92.2%/99.3%/100.0% rotation recall at 3°/5°/10°, versus 57.4%/75.2%/88.5% for ZeroMatch+PGO.Its mean rotation error is 1.4° versus 6.2° for ZeroMatch+PGO.Source: Table 1 · p. 12, §4.2.1 · p. 12
- On ScanNet++, PointVGGT (DA3) achieves 100% recall at all evaluated thresholds; on ARKitScenes, DA3 leads at strict thresholds while Pi3 leads at relaxed thresholds.On ARKitScenes, DA3 reaches 75.2%/77.6% recall at 3°/5°, while Pi3 reaches 96.1%/99.2% at 30°/45°.Source: Table 1 · p. 12, §4.2.1 · p. 13
4.2.2 Evaluation on Object-centric Benchmark
- On WildRGB-D, PointVGGT (Pi3) reaches 88.7%/93.5%/98.9% rotation recall at 3°/5°/10°, versus 58.3%/76.0%/88.5% for UR&R+PGO.Pi3 also achieves 100% recall at 30°/45° and 0.01 m mean/median translation error.Source: Table 2 · p. 13, §4.2.2 · p. 14
4.2.3 Evaluation on Outdoor Benchmarks
- On KITTI, PointVGGT (Pi3) reduces mean translation error from 0.99 m for ZeroMatch+PGO to 0.41 m; on Waymo, PointVGGT (VGGT) reaches 0.07 m mean translation error.The reported results also show 100% rotation recall at 5°/10° on both benchmarks.Source: Table 3 · p. 13, §4.2.3 · p. 14
4.3 Ablation Studies
- On ScanNet, foundation-only initialization reaches 100.0% rotation recall at 10°; refinement raises 3° rotation recall from 90.9% to 92.2% and 0.05 m translation recall from 65.6% to 73.4%.Combining correspondence and reprojection losses gives the best 0.05 m translation recall, 73.4%.Source: §4.3 · pp. 14–15, Table 4 · p. 15
- On ScanNet, GN w/ PCG raises 0.05 m translation success from 70.8% with ADAM to 73.4%; voxel matching reduces correspondence construction time from 65.66 s to 0.07 s.Voxel matching has O(SN) complexity and yields 99.6%, 89.1%, and 68.7% inlier ratios on ScanNet, WildRGB-D, and KITTI, respectively.Source: Table 4 · p. 15, §4.3 · pp. 15–16, Table 5 · p. 16
- Motion BA accuracy gains plateau beyond 40 iterations: increasing to 80 yields only a 0.3% rotation gain while latency rises from 11.14 s to 14.62 s.The authors adopt 40 iterations as the default balance between precision and efficiency.Source: Table 7 · p. 16, §4.3 · p. 16
5 Conclusion
PointVGGT uses a foundation-then-refinement framework to recover metrically grounded global poses without costly pairwise matching and synchronization. Across six benchmarks, it reports strong zero-shot performance while reducing inference time from minutes to seconds.
5 Conclusion
- PointVGGT treats visual geometry priors as primary cues to recover metrically grounded global poses, avoiding costly pairwise matching and global synchronization.The framework follows a foundation-then-refinement paradigm.Source: §5 · p. 16
- Voxelized spatial hashing builds correspondences in near-linear time, while robust motion-only bundle adjustment refines the multiview poses.Together, these components form the refinement stage.Source: §5 · p. 16
- Across six benchmarks, PointVGGT delivers strong zero-shot performance and cuts inference time from minutes to seconds.The conclusion describes the framework as plug-and-play and able to benefit from continuing advances in 3D foundation models.Source: §5 · pp. 16–17