PaperlayerView exact revision on arXiv

Source-linked AI summary

VGGT-X: When VGGT Meets Dense Novel View Synthesis

Yang Liu, Chuanchen Luo, Zimo Tang, Junran Peng, Zhaoxiang Zhang

arXiv:2509.25191v2cs.CV

TL;DR

Dense novel view synthesis with 3D foundation models remains underexplored, and dense scaling brings memory costs and imperfect initialization. VGGT-X combines memory-efficient VGGT, adaptive global alignment, and robust 3DGS training; it substantially narrows the gap with COLMAP-initialized pipelines and achieves state-of-the-art results in COLMAP-free NVS and pose estimation.

  • Problem

    3D foundation models are mostly validated on sparse views, leaving their use for dense NVS open, while dense inference faces rising memory costs and noisy initialization.

    Source: §1 · p. 2

  • Method

    VGGT-X combines memory-efficient VGGT inference, adaptive global alignment, and robust 3DGS training strategies for dense COLMAP-free NVS.

    Source: §1 · p. 2

  • Results

    VGGT-X substantially narrows the gap with COLMAP-initialized counterparts and achieves state-of-the-art performance in COLMAP-free NVS and pose estimation.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The analysis of remaining limitations provides insights and directions for advancing 3D foundation models and NVS frameworks.

    Source: §5 · p. 10

    Rendering quality can exceed COLMAP on training data yet lag on test data, and joint optimization still struggles to correct poses with large deviations.

    Source: §4.4 · p. 9

Abstract

from arXiv · show

We study the problem of applying 3D Foundation Models (3DFMs) to dense Novel View Synthesis (NVS). Despite significant progress in Novel View Synthesis powered by NeRF and 3DGS, current approaches remain reliant on accurate 3D attributes (e.g., camera poses and point clouds) acquired from Structure-from-Motion (SfM), which is often slow and fragile in low-texture or low-overlap captures. Recent 3DFMs showcase orders of magnitude speedup over the traditional pipeline and great potential for online NVS. But most of the validation and conclusions are confined to sparse-view settings. Our study reveals that naively scaling 3DFMs to dense views encounters two fundamental barriers: dramatically increasing VRAM burden and imperfect outputs that degrade initialization-sensitive 3D training. To address these barriers, we introduce VGGT-X, incorporating a memory-efficient VGGT implementation that scales to 1,000+ images, an adaptive global alignment for VGGT output enhancement, and robust 3DGS training practices. Extensive experiments show that these measures substantially close the fidelity gap with COLMAP-initialized pipelines, achieving state-of-the-art results in dense COLMAP-free NVS and pose estimation. Additionally, we analyze the causes of remaining gaps with COLMAP-initialized rendering, providing insights for the future development of 3D foundation models and dense NVS. Our project page is available at https://dekuliutesla.github.io/vggt-x.github.io/

1 INTRODUCTION

The paper investigates whether 3D foundation models can support dense novel view synthesis, where their sparse-view validation leaves an open question. It identifies scaling and noisy initialization as barriers, then introduces VGGT-X to address them and reports state-of-the-art results in COLMAP-free NVS and pose estimation.

  • 3D foundation models are mostly demonstrated with tens of images, while dense inference faces sharply rising memory costs and noisy outputs that impair initialization-sensitive training.Source: §1 · p. 2
  • VGGT-X combines memory-efficient VGGT inference for 1,000+ images, adaptive global alignment, and robust 3DGS training practices to mitigate these barriers.Source: §1 · p. 2
  • The paper analyzes remaining discrepancies with COLMAP-initialized training and offers directions for stronger 3D foundation models and more robust NVS training.Source: §1 · p. 2
  • VGGT-X achieves state-of-the-art performance in both pose estimation and COLMAP-free NVS.Source: §1 · p. 2

2 RELATED WORKS

Related work traces how novel-view-synthesis methods developed while retaining dependencies on accurate scene initialization, surveys 3D foundation models and their scaling limits, and reviews pose-optimization approaches whose large-scale performance gap remains open.

2.1 NOVEL VIEW SYNTHESIS

  • NeRF and 3DGS advances improve rendering and efficiency, but NVS still depends on accurate camera parameters and 3DGS is sensitive to initial point-cloud quality.Source: §2.1 · pp. 2–3

2.2 3D FOUNDATION MODELS

  • 3D foundation models infer 3D attributes through varied architectures and input settings, yet many incur substantial computation and degrade at hundreds or thousands of images.Source: §2.2 · p. 3

2.3 3D RADIANCE FIELD LEARNING WITH POSE OPTIMIZATION

  • Pose optimization, correspondence losses, and robust 3DGS updates address imperfect initialization, but a COLMAP performance gap and limited large-collection evaluation remain.Source: §2.3 · p. 3

3 METHOD

The method combines a memory-efficient VGGT implementation with global camera alignment and robust 3DGS training. It uses these components to address dense-view inference costs and imperfect initialization.

3.1 PRELIMINARY

  • 3DGS represents scenes with learnable Gaussians optimized by photometric losses and adaptive density control; 3DGS-MCMC is used as the NVS baseline for noisy initialization.Source: §3.1 · p. 4

3.2 MEMORY-EFFICIENT VGGT IMPLEMENTATION

  • VGGT− discards unused intermediate outputs, raising throughput from 150 to 600 images; VGGT−− adds BFloat16 and chunked frame-wise processing, reducing peak GPU memory by up to 74% without noticeable degradation.Source: §3.2 · p. 4

3.3 CAMERA PARAMETERS GLOBAL ALIGNMENT (GA)

  • Global alignment refines VGGT camera parameters by minimizing weighted epipolar distances from image correspondences; XFeat supplies matches because VGGT tracking predictions and depth-confidence weights are insufficient.Source: §3.3 · p. 5
  • Adaptive weights down-weight large epipolar-distance outliers, while median epipolar distance selects learning rates across convergence regimes for robust camera optimization.Source: §3.3 · p. 5

3.4 3DGS TRAINING WITH IMPERFECT POSES

  • MCMC-3DGS offers robustness to noisy poses; joint photometric optimization of residual translations and rotations, initialized with reliable correspondence points, consistently improves performance.Source: §3.4 · pp. 5–6

4 EXPERIMENTS

The experiments compare rendering, pose estimation, and component ablations across multi-view reconstruction benchmarks. VGGT-X performs strongly against prior methods, while discussion identifies remaining gaps in rendering fidelity, pose correction, and generalization.

4.1 EXPERIMENTAL SETUP

  • Evaluation covers MipNeRF360, Tanks and Temple, and CO3Dv2, using AUC@30 and Chamfer Distance for 3D estimates and PSNR, SSIM, and LPIPS for rendering.The study reports runtime and VRAM on a 40G A100 GPU and compares COLMAP-free 3DGS methods alongside 3D-attribute prediction baselines.Source: §4.1 · pp. 6–7

4.2 COMPARISON WITH SOTA METHODS

  • The method achieves state-of-the-art rendering; global alignment and joint optimization improve pose estimation over prior methods jointly optimizing poses and 3DGS, with highest accuracy on MipNeRF360 and TnT and second on CO3Dv2.Table 1 includes COLMAP initialization as an upper-bound reference for rendering comparisons.Source: §4.2 · pp. 7–8

4.3 ABLATION

  • Redundant-feature elimination and precision adjustment reduce VRAM use by 83% on MipNeRF360; batched attention saves over 1 GB beyond 800 images, enabling inference on 1,000+ images with negligible accuracy impact.These are ablations of key-attribute estimation components.Source: §4.3 · p. 8, Table 3 · p. 8
  • Replacing XFeat with VGGT tracking lowers AUC@30 by 6.8 points and nearly quadruples Chamfer Distance; depth-confidence weighting also degrades performance, while adaptive learning rates improve accuracy.Scaling the VGG-SfM bundle-adjustment strategy to hundreds of images takes over two hours and does not improve NVS quality.Source: §4.3 · pp. 8–9
  • Under imperfect initialization, MCMC outperforms vanilla 3DGS, pose optimization is essential for stable convergence and high rendering quality, and high-confidence-correspondence point clouds perform best.The reconstruction ablations are conducted on MipNeRF360.Source: §4.3 · p. 9, Table 4 · p. 9

4.4 DISCUSSION

  • A noticeable test-rendering gap to COLMAP remains; VGGT-X surpasses COLMAP-initialized 3DGS on training scenes but trails it on test scenes, indicating overfitting, and depth supervision adds little improvement.The authors point to the ill-posed problem and local minima in the highly non-convex loss landscape as relevant challenges.Source: §4.4 · p. 9
  • Most pose residuals cluster near zero, but large deviations remain difficult to correct; Figure 5 relates residual distributions to blurry artifacts from inaccurate poses.Even after joint optimization, pose accuracy remains below COLMAP.Source: §4.4 · p. 9, Figure 5 · p. 9
  • VGGT substantially outperforms π3 on MipNeRF360 but is considerably worse on CO3Dv2, leaving 3D-foundation-model generalization an open challenge.Source: §4.4 · p. 10

5 CONCLUSIONS

VGGT-X addresses dense NVS challenges in computational scalability and prediction accuracy with a memory-efficient VGGT implementation, adaptive global alignment, and robust 3DGS training. The approach narrows the performance gap with COLMAP-initialized counterparts, while the analysis examines remaining limitations and future directions.

5 CONCLUSIONS

  • VGGT-X combines memory-efficient VGGT, adaptive global alignment, and robust 3DGS training to address dense NVS scalability and prediction-accuracy challenges.The challenges concern computational overhead and accuracy for subsequent radiance-field fitting.Source: §5 · p. 10
  • VGGT-X substantially narrows the performance gap with COLMAP-initialized counterparts.Source: §5 · p. 10
  • The paper analyzes remaining limitations and outlines directions for advancing 3DFMs and NVS frameworks.Source: §5 · p. 10

A ADDITIONAL ABLATIONS

Additional ablations on MipNeRF360 report that several tested design changes do not clearly improve rendering quality, while combining global alignment with GS training is suboptimal. The authors therefore use global alignment as an independent component.

A ADDITIONAL ABLATIONS

  • Table 5 reports independent model-component modifications on MipNeRF360, using MCMC-3DGS equipped with pose embedding as the baseline.Source: Table 5 · p. 15
  • Integrating global alignment into GS training yields suboptimal results, so the authors adopt it as an independent component.Source: A ADDITIONAL ABLATIONS · p. 15
  • MLP-based pose embedding learning, epipolar loss, and depth supervision bring no clear rendering-quality benefits in the reported ablations.Source: A ADDITIONAL ABLATIONS · p. 15
  • Doubling the learning rate and encouraging a broader learned distribution aggregates the overfitting phenomenon.Source: A ADDITIONAL ABLATIONS · p. 15

B PERMUTATION-EQUIVARIANT AUC@30

The conventional AUC@30 metric can depend on input image order because relative translation angles change with the ordering. Including both directions of each relative-pose pair preserves rotation error and makes translation error and AUC@30 permutation-equivariant.

B PERMUTATION-EQUIVARIANT AUC@30

Related papers

Loading 2509.25191v2…