PaperlayerView exact revision on arXiv

Source-linked AI summary

VGGT-Diff: Visual Geometry Meets Diffusion for Sparse-View Novel View Synthesis

Kangjie Chen, Xiangyu Li, Dongbin Zhang, Chaoda Zheng, Shijia Chen, Jinhao Deng, Hongbin Lin, Choo Sin Wai, Minqi Wang, Minghao Yang, Dake Zhong, Guorui Song, Yu Zhang, Xianming Liu, Boyang Wang

arXiv:2609.33253v1cs.CVcs.AI

TL;DR

Sparse-view NVS must preserve observed scene evidence while completing unseen regions, but reconstruction and diffusion approaches have complementary limitations. VGGT-Diff routes visual geometry into pretrained multi-view diffusion and regularizes residuals along 3D tracks. It ranks first in three reported metrics on DL3DV targets and reports competitive or state-of-the-art performance across pose difficulties.

  • Problem

    Reconstruction-oriented methods struggle to determine unseen regions from sparse views, while diffusion methods offer strong generative priors but often lack explicit geometric conditioning.

    Source: §1 · p. 2

  • Method

    VGGT-Diff routes confidence-associated VGGT-Ω visual geometry into query-aligned conditions for pretrained multi-view diffusion and aligns predicted-clean residuals along reliable 3D tracks.

    Source: §1 · p. 2

  • Results

    VGGT-Diff ranks first in PSNR, LPIPS, and DreamSim and second in SSIM on 6,188 DL3DV targets, with competitive or state-of-the-art performance across pose difficulties.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    The jointly generated views show improved geometric reconstructability, while the method combines geometry-grounded scene structure with diffusion’s capacity to complete unseen content.

    Source: §1 · p. 2, §5 · p. 10

Abstract

from arXiv · show

We present VGGT-Diff, a geometry-routed multi-view diffusion model for sparse-view novel view synthesis. Existing novel view synthesis (NVS) methods face a fundamental trade-off: reconstruction-based approaches preserve observed geometry but struggle to synthesize unseen regions, while diffusion-based methods provide strong generative priors yet rely on implicit source-to-query correspondence. VGGT-Diff bridges these regimes by routing visual geometry latents from VGGT-Ω into a pretrained video diffusion model. Each visual token is associated with a 3D point and confidence, then transformed into query-aligned latent conditions through a confidence-aware Visual Geometry Router (VGR) that preserves front and back surface evidence. These conditions guide joint target-view denoising, while Point-Track Residual Consistency (PTRC) regularizes predicted-clean residuals along reliable 3D tracks, improving multi-view stability. We further introduce robust geometry conditioning, combining training-time regularization with inference-time guidance for improved robustness. Experiments show competitive or state-of-the-art performance across interpolation and extrapolation under different viewpoint difficulties. Our code is available at https://github.com/chenkangjie1123/VGGT-Diff.

1 INTRODUCTION

Sparse-view NVS must balance preserving observed geometry with completing unseen regions. VGGT-Diff routes confidence-aware visual geometry into multi-view diffusion and uses geometry-based consistency and regularization to guide generation.

1 INTRODUCTION

  • Reconstruction methods preserve observed geometry but become under-constrained in unseen regions, while diffusion priors can hallucinate content inconsistent with scene geometry.The paper identifies structural drift, unstable occlusions, and cross-view inconsistency under wide-baseline interpolation and extrapolation as risks of implicit correspondence.Source: §1 · p. 2
  • VGGT-Diff routes VGGT-Ω visual tokens, associated with 3D locations and confidence, into query-aligned conditions through a confidence-aware Visual Geometry Router.The router preserves front- and back-surface evidence and visibility uncertainty rather than treating reconstructed geometry as a complete scene explanation.Source: §1 · p. 2
  • Point-Track Residual Consistency aligns denoising residual errors for the same physical point across generated views, avoiding direct matching of view-dependent features or VAE latents.The method also randomly retains, attenuates, or removes routed geometry during training, with optional Geometry-Prior CFG at inference.Source: §1 · p. 2
  • VGGT-Diff combines a pretrained video diffusion prior with geometry-routed generation, residual consistency, and geometry-condition regularization for sparse-view synthesis.The paper reports competitive or state-of-the-art performance across pose difficulties and improved geometric reconstructability of jointly generated views.Source: §1 · p. 2

2 RELATED WORK

Related work spans reconstruction-oriented and feed-forward view synthesis as well as generative NVS. VGGT-Diff combines explicit visual geometry with diffusion, using geometry as uncertainty-aware evidence rather than a complete scene representation.

Generalizable and Feed-Forward View Synthesis

  • Reconstruction and feed-forward methods provide geometric fidelity, but sparse observations leave distant query views under-constrained.The section traces approaches from scene-specific radiance fields and Gaussian splatting to generalizable models, Transformers, and feed-forward 3D representations.Source: §2 · p. 3

Generative and Geometry-Guided Novel View Synthesis

  • Generative NVS uses image or video priors to complete unobserved regions, while implicit source-to-query correspondence can limit wide-baseline robustness and cross-view consistency.Geometry foundation models encode multi-view geometry and correspondence, and prior work has combined such evidence with reconstruction and generative approaches.Source: §2 · p. 3
  • VGGT-Diff routes visual geometry features, 3D points, and confidence into query-aligned diffusion conditions and reuses correspondences to regularize jointly generated views.Unlike approaches that construct complete Gaussian or radiance-field scenes, it uses geometry to guide and constrain the generative prior.Source: §2 · p. 3

3 METHOD

VGGT-Diff treats sparse-view synthesis as camera-conditioned multi-view diffusion: source and query views share a diffusion sequence, allowing information exchange while retaining view-specific camera controls.

3.1 CAMERA-CONDITIONED MULTI-VIEW DIFFUSION

  • The method uses sparse posed source images and prescribed query cameras to synthesize requested views.The available passage introduces the source and query setup but does not provide further detail about the camera representation.Source: §3.1 · p. 3
  • Source and query views are placed in one diffusion sequence so they can exchange information across slots while retaining view-specific camera controls.A frozen VAE encodes the training views into clean latents.Source: §3.1 · p. 3
  • The method follows Wan’s linear flow formulation, sampling a flow time and noise to define the diffusion state and training target.The supplied passage gives the formulation’s equations but only partial surrounding text.Source: §3.1 · p. 3

2. Point-track residual consistency 𝑍'&,% = 𝑋',% −𝜎𝑉#,%

VGGT-Diff combines jointly processed multi-view diffusion with geometry-routed conditions and point-track residual regularization. Its training and inference strategy weakens or selectively strengthens geometry conditioning to account for imperfect geometry.

2. Point-track residual consistency 𝑍'&,% = 𝑋',% −𝜎𝑉#,%

  • Source appearance, camera rays, noisy query latents, and routed geometry conditions are embedded per view and jointly processed by a Wan-initialized multi-view DiT.Flow matching supervises target fidelity, while PTRC supervises cross-view consistency.Source: Figure 2 · p. 4, §2 · p. 4

3.2 GEOMETRY-ROUTED VISUAL CONDITIONING

  • VGGT-Ω associates source features with 3D points and confidence, letting the router transfer appearance-bearing features to query locations instead of routing pre-rendered RGB.The hard z-buffered anchor retains near-front features; confidence-weighted front and secondary hypotheses provide residual cues for valid evidence near occlusions or geometric errors.Source: §3.2 · pp. 4–5, Figure 3 · p. 5

3.3 POINT-TRACK RESIDUAL CONSISTENCY

  • PTRC couples errors for the same physical point across jointly generated views rather than directly matching RGB, latents, or velocities.Tracks must project inside both views, in front of both cameras, and pass per-view z-buffer visibility tests; confidence weighting limits unreliable tracks while preserving valid appearance changes.Source: §3.3 · p. 6

3.4 TRAINING AND INFERENCE

  • Training combines query-wise flow matching with point-track regularization, and randomly retains, attenuates, or removes geometry conditions to handle imperfect projections.The number of jointly generated query views varies during training; geometry condition dropout is intended to improve robustness when routed evidence is sparse, uncertain, or locally missing.Source: §3.4 · p. 6
  • At inference, Geometry-Prior CFG strengthens query geometry during early denoising by contrasting fully conditioned predictions with a reference that removes only query-routed geometry.The two branches retain the same diffusion state, source appearance, source-aligned visual features, camera conditions, timestep, and empty textual context.Source: §3.4 · p. 6

4 EXPERIMENTS

The experiments compare VGGT-Diff with prior methods on DL3DV and zero-shot Mip-NeRF 360, assess results across pose difficulties and cross-view consistency, and ablate its design and geometry conditioning. VGGT-Diff performs strongly across these evaluations, while the ablations support the roles of visual geometry, PTRC, and geometry-conditioning strategies.

4.2 COMPARISON WITH BASELINES

  • On 6,188 DL3DV targets, VGGT-Diff ranks first in PSNR, LPIPS, and DreamSim and second in SSIM; versus FrameCrafter it gains 0.924 dB PSNR and 0.063 SSIM.On zero-shot Mip-NeRF 360, it ranks first in LPIPS and DreamSim and second in PSNR.Source: Table 1 · p. 7, §4.2 · p. 7

4.1 EXPERIMENTAL SETUP

  • The common evaluation uses six source views, tests 6,188 DL3DV targets and zero-shot Mip-NeRF 360, and reports PSNR, SSIM, LPIPS, and DreamSim.Training uses 980 DL3DV scenes after exclusions; the model is initialized from Wan2.1-I2V-14B and evaluated with 50 flow-sampling steps.Source: §4.1 · pp. 6–7

4.2 COMPARISON WITH BASELINES

  • Qualitative DL3DV comparisons show better preservation of thin structures, scene geometry, signage, target layout, and occlusion order than competing methods.The figure comparison includes diverse scenes and contrasts against blur, structural distortion, and content drift.Source: §4.2 · p. 7, Figure 4 · p. 8
  • VGGT-Diff leads five of six pose-difficulty bins and trails SEVA by 0.020 dB in interpolation-near; its margin over FrameCrafter is 0.582–1.228 dB.The margin exceeds 0.9 dB in five bins.Source: §4.2 · p. 7, Table 2 · p. 8
  • Across 52 DL3DV scenes, VGGT-Diff ranks first under both reconstructors and reduces Chamfer-L1 versus FrameCrafter by 26.7% with VGGT-Ω and 16.4% with Pi3.These metrics measure relative consistency and reconstructability, not absolute 3D accuracy, because references come from target RGB rather than physical scans.Source: §4.2 · p. 8, Table 4 · p. 9

4.3 ABLATION STUDIES

  • Removing all VGGT-Ω-dependent components causes the largest reported degradation: PSNR falls by 1.730 dB, SSIM by 0.056, and LPIPS rises by 0.056.Replacing routed features with point-rendered RGB costs 0.471 dB PSNR; removing PTRC costs 0.438 dB PSNR and 0.027 SSIM.Source: Table 3 · p. 8, §4.3 · p. 9
  • Removing condition regularization costs 0.302 dB PSNR and 0.019 SSIM while increasing LPIPS by 0.008; Geometry-Prior CFG improves the full checkpoint by 0.115 dB PSNR and 0.006 SSIM while reducing LPIPS by 0.005.The authors identify regularization as the primary mechanism and matched CFG as a modest inference-time complement.Source: §4.3 · pp. 9–10

5 CONCLUSION

The conclusion presents VGGT-Diff as a geometry-routed multi-view framework combining visual geometry with pretrained video diffusion for sparse-view NVS. It reports strong target-view fidelity and cross-view coherence, and the paper describes its AI assistance and reproducibility materials.

5 CONCLUSION

  • VGGT-Diff combines visual geometry with pretrained video diffusion, using confidence-aware VGR conditions and PTRC along reliable correspondences to support sparse-view NVS.Stochastic training-time weakening and selective inference-time strengthening are used to improve robustness to imperfect geometry.Source: §5 · p. 10
  • The paper reports strong target-view fidelity and cross-view coherence across diverse pose difficulties, combining scene structure with the generative prior's ability to complete unseen content.Source: §5 · p. 10
  • The authors state that AI tools assisted language polishing, formatting, and literature discovery, while experiments and scientific claims were not determined by AI.Source: AI USE STATEMENT · p. 10
  • The paper says it will release training and evaluation code, checkpoints, and configurations, and that detailed protocols and controlled experiments appear in the Appendix.Source: REPRODUCIBILITY STATEMENT · p. 10

A APPENDIX

The appendix specifies how LPIPS differs between the cross-method comparison and controlled ablations, and outlines the supplementary protocol and analysis topics.

A APPENDIX

  • Table 1 cross-method comparisons use LPIPS-AlexNet, while controlled ablations use LPIPS-VGG within the fixed 192-case diagnostic protocol.Absolute LPIPS values are not compared across these protocols.Source: A APPENDIX · p. 16
  • The appendix covers view sampling, camera gauge, pose difficulty, training and conditioning settings, data and optimization scaling, joint multi-target generation, and VGGT-Ω feature analyses.Source: A APPENDIX · p. 16
  • The supplementary material also includes complete pose-difficulty results, two-reconstructor consistency evaluation, PTRC residual analyses, source-view robustness, and VAE protocols.It concludes with directions for extending geometry-routed multi-view generation.Source: A APPENDIX · p. 16

A.1 VIEW SAMPLING AND POSE-DIFFICULTY PROTOCOL

The appendix describes reproducible six-view source sampling, target selection, and a source-defined camera gauge. Its pose-difficulty protocol separates trajectory interpolation or extrapolation from the target's translation-and-rotation novelty.

A.1 VIEW SAMPLING AND POSE-DIFFICULTY PROTOCOL

  • Each training scene has 16 reproducible six-view bundles: 13 use global trajectory stratification and three sample within a local window, with selected views shuffled.Source: §A.1 · p. 16
  • Targets exclude active source frames; the curriculum varies target counts, uses ordered trajectory sampling for sets of eight or more, and does not explicitly balance evaluation pose-difficulty bins.Source: §A.1 · p. 16
  • Camera normalization uses only the six sources: the source nearest their centroid anchors rotation and origin, and translation is scaled by mean source-to-anchor distance.Query cameras do not affect this transformation, so single-target and joint inference share a frame for a fixed source set.Source: §A.1 · p. 16
  • Interpolation means a target frame lies between the minimum and maximum source frame indices; extrapolation means it lies outside that interval, not necessarily outside the source cameras' 3D convex hull.Source: §A.1 · p. 16
  • Pose novelty combines scene-normalized translation and viewing-direction change, with near, mid, and far bins set by separate tertiles for interpolation and extrapolation.Figure 6 uses rows for trajectory position and columns for novelty; a target can be extrapolative but pose-near, or interpolative yet pose-far.Source: §A.1 · p. 17, Figure 6 · p. 17
  • The frozen diagnostic selects 32 targets from 32 different scenes in each of six bins, yielding 192 unique targets from 109 scenes.Each method uses the same six sources and predicts targets independently with fixed per-case noise and no rollout or prediction feedback.Source: §A.1 · p. 17

A.2 FIXED HYPERPARAMETERS AND CONDITIONING REGULARIZATION

The fixed conditioning setup routes confidence-normalized geometry into query cells while retaining front and secondary surface evidence. Training regularizes geometry conditions with a shared scale and ramps PTRC during each stage; inference guidance uses a target-only reference branch.

A.2 FIXED HYPERPARAMETERS AND CONDITIONING REGULARIZATION

  • The shared router thresholds use relative depth, while PTRC is evaluated directly in the Wan latent space.The fixed hyperparameter table collects constants shared by the main experiments.Source: §A.2 · p. 17, Table 5 · p. 18
  • The Visual Geometry Router uses the nearest splatted depth as each query cell’s front anchor and retains a secondary hypothesis beyond a relative-depth threshold.Its thresholds are dimensionless, and the back-surface attenuation uses τ = 0.02.Source: §A.2 · p. 17
  • PTRC is disabled for the first 10% of each training stage, ramps linearly over the next 20%, and is fully weighted thereafter.The latent-space implementation uses a Smooth-L1 penalty with β = 0.1 and no VAE decoding.Source: §A.2 · p. 18
  • Geometry-condition regularization applies one shared sampled scale to global Ω context and dense routed geometry across source and query slots.Source-image latents, Plücker rays, diffusion state, and timestep remain unchanged; the operation is not text dropout.Source: §A.2 · p. 18
  • Geometry-Prior CFG compares full routed geometry with a reference branch that removes query-slot dense geometry while preserving source-aligned conditions and global context.Guidance decays from 2 to 1 over the first 60% of the 50-step trajectory, then is inactive.Source: §A.2 · p. 18

A.3 SCALING WITH DATA AND OPTIMIZATION BUDGET

On DL3DV clean-10K, extending optimization beyond the original 18K budget improves all three image metrics, although checkpoint quality fluctuates locally. Flow and PTRC losses also continue declining through 100K steps.

A.3 SCALING WITH DATA AND OPTIMIZATION BUDGET

  • The clean-10K model uses 9,510 available scenes and is evaluated at fixed 192 targets, six source views, and 50 sampling steps.The diagnostic reports LPIPS-VGG, which should not be compared directly with the LPIPS-AlexNet results in Table 1; logged losses average the preceding 1K effective steps.Source: §A.3 · p. 18
  • At 100K, PSNR rises from 18.7861 to 20.2780 dB, SSIM from 0.5272 to 0.6166, and LPIPS-VGG falls from 0.2793 to 0.2335.These are gains of 1.4918 dB PSNR and 0.0894 SSIM, with a 0.0458 reduction in LPIPS-VGG; the 100K checkpoint is best on all three image metrics.Source: §A.3 · pp. 18–19, Table 6 · p. 19, Figure 7 · p. 19
  • Checkpoint quality is not strictly monotonic: 42K, 60K, and 88K regress locally, while the 100K PSNR gain over 96K narrows to 0.0135 dB.Continued SSIM and LPIPS-VGG improvements mean performance has not fully saturated.Source: §A.3 · p. 19
  • From 18K to 100K, preceding-1K Flow loss falls from 0.074406 to 0.050209 (32.5%), and raw unweighted PTRC loss falls from 0.087436 to 0.073912 (15.5%).PTRC rises early as it is activated and the target curriculum becomes more difficult; after full activation, its decline accompanies continued image-quality improvement.Source: §A.3 · pp. 19–20, Figure 8 · p. 19

A.4 BENEFITS OF JOINT MULTI-TARGET GENERATION

Jointly generating multiple target views outperforms independent runs across image-quality and geometry metrics in the reported eight-view comparison. The comparison holds the checkpoint, sources, target cameras, noise, and sampler fixed.

A.4 BENEFITS OF JOINT MULTI-TARGET GENERATION

  • The comparison covers 52 DL3DV scenes and 416 targets, using the same checkpoint, six sources, eight target cameras, seed, per-view noise, and 50-step sampler.Independent runs cannot access other target cameras, latents, or predictions.Source: §A.4 · p. 20, Table 7 · p. 20
  • Joint generation improves PSNR by 0.462 dB and SSIM by 0.0334, reduces LPIPS-VGG by 0.0176 and normalized Chamfer-L1 by 0.0259, and raises F-score@1% and F-score@2% by 0.1006 and 0.1294.Scene-bootstrap 95% confidence intervals exclude zero for all three image metrics and all three geometry metrics.Source: Table 7 · p. 20, §A.4 · p. 20

A.5 VGGT-ΩDESIGN CHOICES FOR GEOMETRY CONDITIONING

VGGT-Diff’s geometry-conditioning and consistency choices are tested across pose difficulty, reconstruction coherence, reduced source coverage, and implementation settings. Results favor intermediate frozen VGGT-Ω features and residual-based consistency, while the paper identifies boundaries and future extensions for these findings.

A.5 VGGT-Ω DESIGN CHOICES FOR GEOMETRY CONDITIONING

  • Frozen Layer 11 performs best across metrics, improving over frozen Layer 23 by 0.153 dB PSNR and 0.0098 SSIM while reducing LPIPS-VGG by 0.0040.Jointly optimizing Layer 23 instead reduces PSNR by 0.647 dB and SSIM by 0.0352, while increasing LPIPS-VGG by 0.0218 relative to frozen Layer 23. These half-resolution results are not directly comparable with the full-resolution main results.Source: Table 8 · p. 20, §A.5 · p. 21

A.6 COMPLETE POSE-DIFFICULTY RESULTS

  • VGGT-Diff achieves the best LPIPS in all six DL3DV pose bins and on every Mip-NeRF 360 bin, with best PSNR in five and four bins, respectively.On DL3DV, its PSNR margin over FrameCrafter ranges from 0.582 to 1.228 dB; it trails SEVA by 0.020 dB in interpolation-near.Source: §A.6 · p. 21, Table 9 · p. 21, Table 10 · p. 22

A.7 POINT-CLOUD RECONSTRUCTION

  • For jointly generated VGGT-Diff views, both VGGT-Ω and Pi3 reconstructions show fewer displaced points and more coherent geometry than SEVA and FrameCrafter, indicating stronger cross-view consistency.Pi3 is an independent reconstructor not used by VGGT-Diff for training or conditioning.Source: Figure 10 · p. 22, §A.7 · p. 22

A.8 RESIDUAL VERSUS RAW-VELOCITY CONSISTENCY

  • PTRC raises PSNR from 17.769 to 18.581 dB and SSIM from 0.460 to 0.516, while reducing LPIPS from 0.330 to 0.280 versus direct velocity matching.The controlled comparison holds correspondences and weighting fixed; gains occur across all six pose bins and in both interpolation and extrapolation.Source: §A.8 · p. 23, Table 11 · p. 23

A.9 ADDITIONAL OPTIMIZATION AND STATISTICAL DIAGNOSTICS

  • Full DiT fine-tuning improves PSNR by 1.324 dB and SSIM by 0.0478, while reducing LPIPS-VGG by 0.0751 versus rank-32 LoRA.Separately, the scene-mean PSNR gain over FrameCrafter is 0.7893 dB, with a 92.14% scene win rate and a scene-bootstrap 95% confidence interval of [0.7003, 0.8769] dB.Source: §A.9 · pp. 23–24, Table 12 · p. 23

A.10 ROBUSTNESS TO SPARSE INPUT VIEWS

  • With three inputs, VGGT-Diff improves over FrameCrafter by 0.781 dB PSNR and 0.0319 SSIM, while reducing LPIPS-VGG by 0.0318 and DreamSim by 0.0127.With four inputs, the corresponding gains are 0.584 dB, 0.0231, 0.0255, and 0.0093; the six-source-trained checkpoint is used without adaptation.Source: §A.10 · p. 24, Table 13 · p. 24

A.11 VAE ENCODING PROTOCOLS

  • Quantitative experiments independently encode each source and target view, while the 80-frame demo uses temporally compressed target slots and aggregates camera maps within VAE windows.The demo reduces DiT view-time slots from 86 to 27; it is fine-tuned separately and does not affect reported quantitative results.Source: §A.11 · pp. 24–25

A.12 FUTURE DIRECTIONS

  • Future directions include multi-level feature fusion, geometry-preserving encoder adaptation, richer visibility modeling, longer joint generation, and spatiotemporal point tracks.These are proposed extensions, not evaluated findings.Source: §A.12 · p. 25
Loading 2609.33253v1…