PaperlayerView exact revision on arXiv

Source-linked AI summary

GeoVerse: World-Consistent Novel View Synthesis in Geometric Latent Space

Kerui Ren, Tao Lu, Linning Xu, Changjian Jiang, Mu Huang, Chunhua Shen, Mulin Yu, Bo Dai

arXiv:2609.35734v1cs.CV

TL;DR

Sparse-view novel view synthesis must balance faithful reconstruction of observed content with plausible completion of unseen regions and consistency across viewpoints. GeoVerse injects video appearance priors into geometric latent diffusion and anchors successive predictions with spatial memory; it reports 2.23 dB higher PSNR than GLD on DL3DV.

  • Problem

    Sparse-view novel view synthesis must balance faithful reconstruction of observed regions with plausible completion of unseen content while maintaining cross-view consistency.

    Source: §1 · p. 2

  • Method

    GeoVerse augments geometric latent diffusion with video generative priors and uses persistent spatial memory to provide target-aligned guidance across successive views.

    Source: §1 · p. 2

  • Results

    2.23 dB higher PSNR than GLD on DL3DV; GeoVerse also reports state-of-the-art visual quality and pose accuracy.

    Source: §4.2 · p. 8, §6 · p. 10

  • Takeaways & Limitations

    GeoVerse combines video-learned priors with geometric latent spaces to generate high-fidelity views while maintaining rigid spatial structure.

    Source: §6 · p. 10

    Peak visual fidelity is constrained by DA3 appearance capacity, while textureless or complex regions can accumulate depth and appearance errors through memory updates.

    Source: §5 · pp. 9–10

Abstract

from arXiv · show

Novel view synthesis from sparse images must reconcile faithful reconstruction of observed regions with plausible completion of unseen content, while maintaining world consistency across viewpoints. Existing geometry-based methods preserve observed scene structure but often struggle to complete unseen regions, whereas video generative models offer rich appearance priors but accumulate inconsistencies during sequential view generation. We propose GeoVerse, a framework that synthesizes world-consistent novel views by performing generation within the geometric latent space of a pretrained 3D foundation model and injecting appearance priors from a video generative model. Specifically, GeoVerse extracts multilevel features from Wan2.2 VACE and injects them into the geometric latent diffusion model via a ControlNet-style adapter, incorporating video-learned appearance priors to enhance structural completion. To enforce cross-view coherence, a global spatial memory continuously aggregates observed and synthesized content, reprojecting target-aligned guidance to anchor subsequent predictions to a shared scene representation. Extensive experiments across diverse datasets demonstrate improved visual quality and geometric consistency, with a 2.23 dB higher PSNR on DL3DV and 32.4% lower ATE on Mip-NeRF360 compared to GLD.

1 INTRODUCTION

Sparse-view novel view synthesis must balance faithful reconstruction of observed content with plausible completion of unseen regions across viewpoints. GeoVerse combines video appearance priors with geometric latent diffusion and persistent spatial memory to address this challenge.

1 INTRODUCTION

  • GeoVerse bridges geometric latent diffusion with video generative priors to synthesize views with structural consistency and plausible completion.The framework incorporates Wan2.2 VACE appearance priors into the geometric latent space.Source: §1 · p. 2
  • A global spatial memory accumulates scene content and reprojects target-aligned guidance to anchor multi-step predictions and support unseen-region completion.The memory is a colored point cloud that retains historical content across successive views.Source: §1 · p. 2
  • Training scales from 4 to 15 diverse real and synthetic datasets to broaden scene coverage and enhance cross-domain generalization.The expanded training corpus is described as supporting broader scene coverage.Source: §1 · pp. 2–3
  • Reflow distillation reduces denoising to 4 steps, cutting per-inference latency from over 100 seconds to under 10 seconds.Source: §1 · pp. 2–3

2 RELATED WORK

Reconstructive methods progress from per-scene optimization toward feed-forward sparse-view prediction, while generative methods use video priors to extend scene coverage. GeoVerse instead generates directly in geometric latent space, augmented with video priors and persistent 3D memory.

2.1 RECONSTRUCTIVE NOVEL VIEW SYNTHESIS

2.2 GENERATIVE NOVEL VIEW SYNTHESIS

  • Video-based methods expand scene content, while spatial-memory approaches store or retrieve prior content to improve long-horizon consistency.Examples include geometry-grounded memory, Mirage’s persistent 3D cache, and Lyra 2.0’s geometric history retrieval.Source: §2.2 · p. 3
  • Unlike video-sequence generation, GeoVerse adopts geometric-latent synthesis at specified target cameras, augmented by video priors and persistent 3D memory.GLD repurposes DA3 geometric features as the latent space for direct multi-view generation (Lin et al., 2025; Jang et al., 2026).Source: §2.2 · p. 3

3 METHOD

GeoVerse couples geometric-latent view synthesis with video appearance features and persistent spatial memory. It also distills the primary denoiser for few-step generation.

3 METHOD

  • GeoVerse uses context observations and target-aligned RGB-D hints to synthesize target views, then updates shared memory with decoded RGB and geometry.The framework predicts target images and geometry for subsequent view expansion.Source: Figure 2 · p. 4, §3 · p. 4

3.1 ENHANCING GEOMETRIC LATENT DIFFUSION WITH VIDEO PRIORS

  • Generation occurs in GLD’s multilevel geometry latent, combining depth and raymap structure with RGB texture features and frozen Wan2.2 VACE features extracted once.The VACE features are reused throughout geometric denoising rather than generating a complete video.Source: §3.1 · pp. 4–5
  • A ControlNet-style branch aligns VACE features and adds learned residuals under matching timestep and camera conditions, adapting appearance priors while preserving camera control.Zero-initialized projections initially leave the pretrained denoiser unmodified.Source: §3.1 · p. 5
  • Training supervises latent generation and decoded target predictions, weighting target views 1 and context views 0.25.Decoded supervision includes RGB reconstruction, high-gradient regions, and scale-aligned depth.Source: §3.1 · p. 5

3.2 PERSISTENT SPATIAL MEMORY FOR LONG-SEQUENCE CONSISTENCY

  • Context observations initialize colored point-cloud memory; aligned predictions are fused in shared coordinates, and target-aligned RGB-D projections anchor known regions while masked areas remain for generation.Fusion mitigates scale drift; projection validity accounts for point confidence and z-buffer visibility.Source: §3.2 · pp. 5–6

3.3 FEW-STEP INFERENCE VIA REFLOW DISTILLATION

  • Inference allocates three steps to Level-1 and one to Level-0; reflow distillation shortcuts teacher evaluations, with one cascade step matching 49-step quality.Distillation is applied only to the Level-1 denoiser; the cascade and decoding heads remain unchanged.Source: §3.3 · pp. 6–7

4 EXPERIMENTS

The evaluation combines benchmark comparisons with denoising-budget and component ablations. GeoVerse reports gains in visual quality, geometric consistency, and inference speed, while the analyses examine budget and component trade-offs.

4.1 EXPERIMENTAL SETUP

  • Evaluation covers RealEstate10K and DL3DV in-domain, Mip-NeRF 360 and ScanNetv2 out-of-domain, with two context and six target views per round.Metrics include image quality, pose fidelity, cross-view consistency, and inference time; baselines include ViewCrafter, NeoVerse, GEN3C, MVGenMaster, Matrix3D, CAMEO, and GLD.Source: §4.1 · pp. 7–8

4.2 COMPARISON

  • 2.23 dB higher PSNR on DL3DV and 2.45 dB higher PSNR on Mip-NeRF 360 than GLD; GeoVerse also leads all four benchmarks in visual quality.On three-round ScanNetv2 testing, PSNR rises from 15.33 dB for GLD to 19.65 dB for GeoVerse.Source: §4.2 · p. 8
  • 32.4% lower ATE on Mip-NeRF 360 (0.071 versus GLD's 0.105) accompanies stronger global scene coherence, despite some metric-specific baseline wins.The passage notes that some baselines have lower reprojection or translation errors on individual datasets.Source: §4.2 · p. 9
  • Roughly 9 seconds for six target views across three main benchmarks is about 18× faster than GLD.GeoVerse uses four geometric denoising steps and single-pass video feature extraction.Source: §4.2 · p. 9

4.3 FEW-STEP ANALYSIS

  • 9.24 to 6.07 s runtime reduction from three to one L1 step raises LPIPS from 0.339 to 0.367, while one Cascade L0 step retains 49-step detail.The table caption identifies three L1 updates plus one Cascade L0 update as the best overall quality–speed trade-off among the compared configurations.Source: Figure 4 · p. 8, Table 2 · p. 8, §4.3 · p. 9

4.4 ABLATION STUDIES

  • Removing the Wan2.2 video prior causes the most severe visual degradation, while disabling spatial memory degrades visual fidelity and cross-view alignment.The ablation table evaluates components, training losses, and alternative video-prior injection architectures on ScanNetv2.Source: §4.4 · p. 9, Table 3 · p. 9
  • Removing RGB supervision moderately reduces visual quality, while removing depth supervision weakens overall geometric consistency.The passage also notes non-uniform metric variations for geometric outcomes.Source: §4.4 · p. 9

5 LIMITATIONS

GeoVerse's peak visual fidelity is constrained by the appearance capacity of the DA3 feature space, and long-sequence consistency depends on backbone geometry and synthesized RGB coherence.

5 LIMITATIONS

  • Peak visual fidelity is constrained by DA3 feature-space appearance capacity, limiting fine-texture richness relative to computationally heavy, full-scale video diffusion models.GeoVerse prioritizes speed through compact feature conditioning.Source: §5 · pp. 9–10
  • In textureless or complex regions, initial depth errors and generated appearance drift can accumulate through spatial-memory updates and occasionally compromise long-trajectory cross-view alignment.Multi-round consistency depends on both the geometry backbone and synthesized RGB coherence.Source: §5 · p. 10

6 CONCLUSION

GeoVerse combines video-learned appearance priors with geometric latent diffusion and spatial memory to synthesize high-fidelity, structurally consistent views. The paper reports state-of-the-art visual quality and pose accuracy with over 17 times speedup compared to GLD.

6 CONCLUSION

  • GeoVerse embeds pretrained video appearance priors in a 3D geometric latent diffusion model and uses global spatial memory to anchor extended trajectories to accumulated scene context.A few-step reflow distillation scheme accelerates inference.Source: §6 · p. 10

AI USE STATEMENT

The authors report using generative AI tools for translation, literature summarization, and manuscript editing, with translated or polished text manually checked sentence by sentence.

AI USE STATEMENT

  • Generative AI assisted with translation, literature summarization, and editing for readability; translated or polished text was manually cross-checked sentence by sentence.The authors state that these tools were not used to design methods or experiments, implement methodologies, or interpret results.Source: AI USE STATEMENT · p. 10

A WAN2.2 VACE DETAILS

GeoVerse extracts multilevel Wan2.2 VACE features as video-appearance cues for geometric generation. Its feature-extraction path encodes a shared RGB proxy and reuses frozen features in the geometric denoiser.

A WAN2.2 VACE DETAILS

  • Wan2.2 combines a causal video VAE, text encoder, and diffusion Transformer; its A14B version uses separate high- and low-noise experts.The Transformer denoises video latents under text and optional visual conditions, and the VAE decoder converts the final latent into RGB frames.Source: A WAN2.2 VACE DETAILS · p. 15
  • VACE organizes reference images, videos, and spatiotemporal masks into conditions for tasks including inpainting, outpainting, and temporal extension.Its context adapters introduce encoded conditions into the video backbone, providing completion cues for GeoVerse's geometric representation.Source: A WAN2.2 VACE DETAILS · p. 15
  • GeoVerse extracts 5,120-dimensional features at Wan blocks 0, 5, 10, and 15 from the low-noise Wan2.2-VACE-Fun-A14B component.The frozen extractor runs once per multi-view prediction, and the geometric denoiser reuses its features across denoising steps.Source: A WAN2.2 VACE DETAILS · p. 15
  • The shared RGB proxy combines observed context images with projected target images; only valid target projections receive RGB noise, while invalid regions are zero-filled.For the Wan backbone, the proxy is VAE-encoded and perturbed in latent space at t = 500; the VACE condition separately encodes inactive and reactive inputs with a packed visibility mask.Source: A WAN2.2 VACE DETAILS · p. 15
  • Wan2.2-VACE features are extracted in a single pass without a full video-sampling trajectory or VAE decoding, then reused by ControlNet-style adapters.The available model card does not give an absolute training-set size for the particular checkpoint; Wan2.2's reported increases over Wan2.1 are relative.Source: A WAN2.2 VACE DETAILS · pp. 15–16

B LOSS DEFINITIONS AND COMPUTATION

The loss definitions combine multi-view latent flow matching with target-view RGB, high-gradient RGB, and scale-aligned depth supervision. A shared-attention variant provides an alternative way to inject video priors.

B LOSS DEFINITIONS AND COMPUTATION

  • Latent flow matching supervises context and target views, while decoded RGB and depth losses supervise target views only.Context views receive weight 0.25 and target views weight 1 in the described flow objective.Source: B LOSS DEFINITIONS AND COMPUTATION · p. 16
  • The two-level flow-matching objective interpolates clean features with Gaussian noise and trains predicted velocities against the noise-minus-feature target.The squared error is summed over feature-grid and channel dimensions for each view.Source: B LOSS DEFINITIONS AND COMPUTATION · p. 16
  • Target RGB supervision uses valid reference pixels, with a separate gradient-weighted term emphasizing high-gradient regions.The gradient weights come from reference-image Sobel magnitudes; valid supervision pixels remain included even when absent from the memory projection.Source: B LOSS DEFINITIONS AND COMPUTATION · p. 16
  • Scale-aligned depth supervision fits one camera-center similarity transform per sample, then applies its scale to camera-space depth.The same scale is shared across target views; rotation and translation align camera coordinates but do not enter the depth residual.Source: B LOSS DEFINITIONS AND COMPUTATION · pp. 16–17

C MOT VARIANT FOR VIDEO-PRIOR INJECTION

  • The Mixture of Transformers variant couples geometric and video tokens through shared attention, with geometric queries attending to video keys and values.Video keys and values are cached across denoising updates, unlike the ControlNet-style design's auxiliary residual injection.Source: C MOT VARIANT FOR VIDEO-PRIOR INJECTION · p. 17

D TRAINING DATA AND STATISTICS

GeoVerse's training mixture draws on 15 real and synthetic multi-view datasets spanning indoor, outdoor, driving, and synthetic environments. Source sampling balances coverage against domination by the largest collections.

D TRAINING DATA AND STATISTICS

  • The training mixture contains 15 datasets spanning real indoor captures, outdoor videos, driving sequences, and synthetic environments.TartanAir and TartanAir v2 are listed as distinct sampling sources, and MegaDepth is excluded.Source: D TRAINING DATA AND STATISTICS · pp. 17–18
  • Dataset sources are sampled in proportion to the square root of their RGB frame counts, balancing coverage against dominance by the largest collections.For most sources, available dataset geometry and Pi3-estimated geometry are selected at a nominal 2:1 ratio; RealEstate10K and Waymo use Pi3-estimated poses and depth.Source: D TRAINING DATA AND STATISTICS · p. 18
  • The reported inventory counts describe the available training pool, not the distinct images consumed by a completed run.Source: D TRAINING DATA AND STATISTICS · p. 18

E ADDITIONAL QUALITATIVE RESULTS

The additional qualitative material shows novel-view comparisons on RE10K and DL3DV and successive view expansion on ScanNetv2. It presents selected target views alongside context or ground-truth frames.

E.1 NOVEL-VIEW SYNTHESIS

  • Figure 5 shows two target views for each of three scenes from RE10K and DL3DV, alongside context images and ground truth.Source: Figure 5 · p. 18, §E.1 · p. 18

E. ADDITIONAL QUALITATIVE RESULTS

  • The ScanNetv2 sequence is arranged to show how scene appearance evolves as viewpoints expand, while the RE10K and DL3DV comparisons pair targets with context and ground truth.Source: §E.1 · p. 18, §E.2 · p. 19

E.2 SUCCESSIVE VIEW EXPANSION

  • Figure 6 presents three successive generation rounds on two ScanNetv2 scenes, showing target frames 1, 3, and 5 from each round.Source: Figure 6 · p. 19, §E.2 · p. 19

Related papers

Loading 2609.35734v1…