PaperlayerView exact revision on arXiv

Source-linked AI summary

NAMVIS: Next-Scale Autoregressive Multi-View Image Synthesis

Ramil Khafizov, Ilya Statsenko, Ruslan Rakhimov, Artem Komarichev, Peter Wonka, Evgeny Burnaev

arXiv:2610.04722v1cs.CV

TL;DR

Sparse-view novel view synthesis must generate views from a few posed images, while iterative diffusion can make multi-view inference expensive. NAMVIS uses geometry-conditioned next-scale autoregression, improving reported metrics across three datasets and running about 3.3× faster than the fastest evaluated diffusion baseline in the 1-to-1 setting.

  • Problem

    Sparse-view novel view synthesis seeks target images from one or a few posed images, a challenging under-constrained task, while diffusion-based multi-view inference can be expensive.

    Source: §1 · pp. 1–2

  • Method

    NAMVIS predicts target views through denoising-free coarse-to-fine next-scale autoregression conditioned on source images and camera poses and desired target poses.

    Source: §1 · p. 2

  • Results

    Across Objaverse, GSO, and OmniObject3D, NAMVIS improves PSNR, SSIM, and LPIPS over representative public diffusion baselines; in 1-to-1, it is approximately 3.3× faster than the fastest evaluated baseline.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The results suggest geometry-conditioned next-scale autoregression is a promising direction for efficient sparse-view novel view synthesis.

    Source: §5 · p. 10

    The current checkpoint is trained and evaluated at 256 × 256; direct 512 × 512 inference requires additional training or fine-tuning.

    Source: §4.5 · p. 9

Abstract

from arXiv · show

Sparse-view novel view synthesis is a central problem in 3D content creation, but diffusion-based approaches remain limited by iterative denoising, making multi-view generation expensive at inference time. We introduce NAMVIS, a diffusion-free framework that reformulates multi-view image synthesis as geometry-conditioned next-scale autoregression. Instead of generating target views through repeated denoising, NAMVIS predicts discrete visual tokens through a small number of coarse-to-fine scale steps, while sampling all tokens within each scale and across target views in parallel. To anchor this generation process to explicit camera geometry, we propose Multi-scale Projective Pose Encoding, which injects source and target camera transformations into both target-view self-attention and source-to-target cross-attention at every resolution. NAMVIS further combines global conditioning with dense geometry-aware cross-attention, enabling the model to preserve source-view appearance while maintaining target-view consistency. Across Objaverse, GSO, and OmniObject3D, NAMVIS outperforms diffusion-based baselines in PSNR, SSIM, and LPIPS, while running over 3 times faster than the evaluated diffusion baselines under the same evaluation setting. These results suggest that geometry-conditioned next-scale autoregression is a promising and efficient alternative to diffusion for sparse-view multi-view synthesis. Additional qualitative results, videos, and resources are available at https://corl-team.github.io/namvis/

1 Introduction

Sparse-view synthesis must infer views from a few posed images despite ambiguity in geometry, appearance, and illumination, while diffusion-based generation can make multi-view inference expensive. NAMVIS replaces iterative denoising with geometry-conditioned next-scale autoregression and combines camera-aware attention with global source conditioning.

1 Introduction

  • Sparse-view synthesis generates target images from one or more posed source images without per-scene optimization or explicit 3D reconstruction, despite under-constrained scene geometry.Diffusion-based methods can make inference expensive when generating several target views.Source: §1 · pp. 1–2
  • NAMVIS predicts discrete visual tokens through denoising-free, coarse-to-fine next-scale autoregression, avoiding iterative denoising.The model is conditioned on source images and camera poses and on desired target poses.Source: §1 · p. 2
  • Multi-scale Projective Pose Encoding injects camera geometry across varying spatial resolutions into target-view self-attention and source-to-target cross-attention.Source: §1 · p. 2
  • A dual-path mechanism combines global source-view conditioning with dense geometry-aware cross-attention to preserve semantic appearance and fine spatial detail.Source: §1 · p. 2
  • NAMVIS improves PSNR, SSIM, and LPIPS over representative diffusion-based baselines across Objaverse, GSO, and OmniObject3D, with faster inference under the same evaluation setting.The paper also releases preprocessing resources and data for approximately 200K objects.Source: §1 · p. 2
  • The authors release filtered scene lists, rendered images, camera poses, masks, point maps, captions, and preprocessing scripts for approximately 200K objects.Source: §1 · p. 2

2 Related work

Related work spans methods that reconstruct 3D representations, diffusion-based generative synthesis, and autoregressive image generation. NAMVIS directly synthesizes posed target images by extending next-scale autoregression with camera geometry.

Optimization-based and feed-forward novel view synthesis

  • Classical novel-view methods render from intermediate 3D representations and excel with dozens of views, but require per-scene optimization.Source: §2 · p. 3
  • Feed-forward reconstruction methods predict explicit 3D representations from sparse inputs, whereas NAMVIS directly synthesizes posed target images without producing one.The paper focuses its main experimental comparison on generative methods that directly predict target views.Source: §2 · p. 3

Diffusion-based generative NVS

  • Diffusion-based generative NVS improves multi-view consistency through attention and synchronized generation, but iterative denoising increases latency when generating multiple target views.Source: §2 · p. 3

Autoregressive visual generation

  • Next-scale autoregression generates coarse-to-fine rather than token-by-token; NAMVIS extends this paradigm to geometry-aware multi-view synthesis without iterative denoising.CausNVS still uses iterative denoising to generate each individual view.Source: §2 · p. 3

3 Method

NAMVIS adapts next-scale autoregression for sparse-view synthesis, conditioning target generation on source images and camera geometry. Its design combines multi-scale projective encoding, target-view communication, and dense source-to-target attention.

3.1 Background: Autoregressive Next-Scale Models

  • Next-scale models generate images coarse-to-fine through residual token maps, with later-scale residuals accumulated into a continuous feature map.NAMVIS builds its next-scale autoregressive transformer on Infinity, which extends Visual Autoregressive modeling.Source: §3.1 · pp. 3–4, §3 · p. 3

3.2 NAMVIS

  • NAMVIS replaces iterative denoising with coarse-to-fine prediction of target residuals, parallelizing tokens within each scale and across target views.Later scales condition on earlier scales, with a block-causal mask allowing communication among views at the current scale.Source: §3.2 · pp. 4–5
  • Multi-scale Projective Pose Encoding adapts projective camera encoding to next-scale generation and injects geometry into target self-attention and source-to-target cross-attention.Separate coordinate grids and rotary coefficients are constructed for each target scale; projective components encode camera geometry while local rotary components preserve 2D layout.Source: §3.2 · p. 5
  • Target self-attention supports consistency among generated views, while source-to-target cross-attention anchors generation to posed source observations.In cross-attention, target tokens are queries and source features are keys and values; the query/output branches use target cameras and key/value branches use source cameras.Source: §3.2 · p. 5
  • A global path pools source features into an [SOS] token for transformer conditioning, while dense cross-attention supplies unpooled source features for spatial detail.Dense source features remain at full VQ-VAE resolution, so even coarse target scales can attend to high-fidelity source evidence.Source: §3.2 · p. 6

4 Experiments

NAMVIS is evaluated against diffusion-based novel-view-synthesis baselines across three datasets, with image-quality, inference-efficiency, scaling, and geometric-consistency results. The experiments also describe the training and evaluation setup, qualitative comparisons, and limitations concerning resolution and transfer beyond object-centric data.

4.1 Implementation Details

  • The experiments use a filtered dataset of approximately 200K objects, a frozen pretrained VQ-VAE, and a trained autoregressive transformer; evaluation covers fidelity, perceptual similarity, and COLMAP reconstructability.The implementation caches ProPE projections across transformer layers to reduce geometry-aware attention cost without changing outputs.Source: §4.1 · p. 6

4.3 Quantitative Results

  • Across Objaverse, GSO, and OmniObject3D, NAMVIS outperforms diffusion-based baselines in PSNR, SSIM, and LPIPS, and degrades more gracefully across datasets.The reported comparison includes single-view and multi-view generation methods.Source: Table 1 · p. 7, §4.3 · pp. 7–8

4.2 Baselines

  • Baselines include single-view and multi-view diffusion methods evaluated with official pretrained weights, using supported camera-elevation protocols and nine source–target view configurations.Reported quantitative metrics are averaged across the nine configurations at 256 × 256 resolution.Source: §4.2 · p. 7, §4.3 · p. 7

4.3 Quantitative Results

  • NAMVIS takes 0.6 seconds per target view versus Wonder3D’s 2.0 seconds, and supports 6→6 generation with 1.56 s inference time and 5.0 GB peak VRAM.The 6→6 Objaverse setting achieves 26.12 PSNR, 0.9015 SSIM, and 0.0473 LPIPS.Source: §4.3 · p. 7, Table 2 · p. 8
  • NAMVIS reconstructs 221 sparse points, versus 214 for EscherNet, 172 for Wonder3D, and 151 for SyncDreamer; it registers 162 of 240 views (67.5%), versus EscherNet’s 145 of 240 (60.4%).The registration result approaches the 70.0% coverage obtained using ground-truth images.Source: Table 3 · p. 8, §4.3 · p. 8, Table 4 · p. 8

4.4 Qualitative Results

  • Figures 3 and 4 compare qualitative results on OO3D and GSO; NAMVIS better preserves source appearance, follows target poses, and produces sharper boundaries and more consistent geometry.Project-page trajectory videos show stable geometry and appearance across smooth viewpoint changes.Source: §4.4 · p. 8, §4.3 · p. 8, Figure 3 · p. 9

4.5 Limitations and future work

  • The current checkpoint is trained at 256 × 256, while training on object-centric Objaverse-XL means its learned geometric priors may not transfer directly to complex real-world scenes.Direct 512 × 512 inference would require additional training or fine-tuning.Source: §4.5 · pp. 9–10
  • A proposed efficiency direction replaces full causal attention over previous target scales with scale-local or windowed attention while preserving geometry-aware source-to-target cross-attention.The authors suggest this could reduce memory and KV-cache cost at high resolutions.Source: §4.5 · pp. 9–10

5 Conclusion

NAMVIS replaces iterative diffusion denoising with geometry-conditioned, coarse-to-fine autoregression that predicts tokens within each scale and across target views in parallel. Across three datasets, it improves on diffusion-based baselines in image-quality metrics and runs about 3.3× faster than the fastest evaluated baseline in the 1-to-1 setting.

5 Conclusion

  • NAMVIS predicts visual tokens through geometry-conditioned, coarse-to-fine autoregression, with tokens within each scale and across target views generated in parallel.This provides a diffusion-free alternative for sparse-view multi-view synthesis.Source: §5 · p. 10
  • Across Objaverse, GSO, and OmniObject3D, NAMVIS improves PSNR, SSIM, and LPIPS over representative public diffusion-based baselines.In the 1-to-1 setting, it runs in 0.6 s per target view, approximately 3.3× faster than the fastest evaluated diffusion baseline at 256 × 256.Source: §5 · p. 10
  • Future work targets sequence-length bottlenecks at higher resolutions and denser target-view generation, pose-noise robustness, and training beyond object-centric data.The conclusion identifies more complex scene-level multi-view settings as an extension beyond the current training scope.Source: §5 · p. 10

A Architectural Details

  • The model uses a 1B-parameter transformer initialized from Infinity’s 2B checkpoint, retaining every other block to form a 16-block model.The final model has hidden dimension 2048 and 16 attention heads.Source: A Architectural Details · p. 12

B Runtime and View-Count Scaling

NAMVIS is evaluated across source–target view configurations, with inference time measured at 256 × 256 on one NVIDIA A40 GPU. More source views generally improve reconstruction quality, while generating more target views increases runtime and leaves image quality relatively stable for a fixed source-view count.

B Runtime and View-Count Scaling

  • More source views generally improve reconstruction quality, while additional target views increase runtime and leave image quality relatively stable at a fixed source-view count.The evaluation measures inference time and image-quality metrics across configurations at 256 × 256 on a single NVIDIA A40 GPU.Source: B Runtime and View-Count Scaling · p. 13
  • Overall, NAMVIS scales efficiently with target-view count while benefiting from additional source-view evidence.Source: B Runtime and View-Count Scaling · p. 13
  • Removing ProPE caching raises 3-to-3 inference time to 1.28 s, compared with 0.96 s with caching; removing KV caching raises it to 1.06 s.The reported results indicate that caching geometry-dependent ProPE features is particularly important for efficient multi-view inference.Source: B Runtime and View-Count Scaling · p. 14, Table 7 · p. 14

C Effect of the Number of Source Views

  • As source views cover previously unseen areas, NAMVIS shifts from hallucinating those regions to rendering object parts directly supported by the inputs.The accompanying figures show image quality improving when views covering previously unseen areas become available.Source: C Effect of the Number of Source Views · p. 14, Figure 5 · p. 14

D Attention Maps Analysis

Attention visualizations suggest that NAMVIS learns view-correspondence patterns aligned with camera geometry. The maps show different emphases across attention types and scales.

D Attention Maps Analysis

  • For a queried target token, self-attention often assigns high weights to geometrically and visually corresponding regions in other generated views.Source: D Attention Maps Analysis · p. 14
  • At finer scales, self-attention tends to emphasize consistency between target views.Figure 7 illustrates high self-attention weights on corresponding regions in both generated target views.Source: D Attention Maps Analysis · p. 14
  • At coarser scales, cross-attention tends to focus on source-to-target alignment, assigning high weights to visually aligned source regions.Figure 8 shows this pattern for a target-to-source cross-attention map at layer 14.Source: D Attention Maps Analysis · p. 14

E Ablation Details

The ablations compare camera-pose conditioning and reference-image conditioning choices. They favor multiscale ProPE, VQ-VAE pooled global features, dense cross-attention, dual-attention injection, and frozen VQ-VAE dense features in the tested setting.

E Ablation Details

  • The ablation variants are trained on a randomly sampled 10K subset of filtered Objaverse for 200 epochs and evaluated on a held-out subset under a fixed compute budget.Source: E Ablation Details · pp. 14–15
  • Attention maps show target-target correspondence in self-attention and source-region alignment in cross-attention, consistent with camera geometry.The visualized self-attention example is between two generated views; the cross-attention example is from a target image to two source images at layer 14.Source: Figure 7 · p. 15, Figure 8 · p. 16

E.1 Camera Pose Parameterization

  • Multiscale ProPE outperforms Plücker raymap conditioning across all metrics in the ablation comparison.The authors suggest that injecting geometry into attention is more effective in their setting than concatenating explicit ray features.Source: §E.1 · p. 15

E.2 Reference Image Conditioning

  • Replacing VQ-VAE attentive pooling with pretrained CLIP features yields inferior perceptual quality for global semantic conditioning.The authors attribute the VQ-VAE pooled features’ advantage to stylistic alignment with target tokens and avoiding a domain gap.Source: §E.2 · p. 16
  • Removing dense cross-attention and relying only on the [SOS] token substantially degrades quality, suggesting that global conditioning alone lacks fine-grained guidance capacity.Source: §E.2 · p. 16
  • Replacing self-attention followed by cross-attention with standard prefix conditioning leads to a large performance drop.The authors hypothesize that prefixing dense source tokens interferes with the scale-structured sequence, whereas cross-attention keeps source features in a separate key-value space.Source: §E.2 · p. 16
  • Frozen VQ-VAE dense features perform marginally better than separately learned ConvNeXt features while avoiding an auxiliary encoder’s computational overhead.Source: §E.2 · p. 16

F Additional Qualitative Results

NAMVIS produces qualitative generations across diverse object categories, shapes, poses, and textures on OO3D and GSO. Its images develop from coarse structure to finer detail over successive scales.

F Additional Qualitative Results

G Attention Maps Analysis

NAMVIS visualizations cover self- and cross-attention across multiple layers and scales, and suggest learned correspondence patterns consistent with camera geometry. Additional qualitative results are shown on OO3D.

G Attention Maps Analysis

  • Self- and cross-attention visualizations across layers and scales suggest correspondence patterns consistent with underlying camera geometry.The analysis covers patterns across several layers and scales.Source: G Attention Maps Analysis · p. 16
  • The attention analysis includes both self-attention and cross-attention patterns.The visualizations span several layers and scales.Source: G Attention Maps Analysis · p. 16
  • Figure 9 presents qualitative NAMVIS results on OO3D (Wu et al. [2023]).The passage identifies the dataset and figure, without specifying particular visual outcomes.Source: Figure 9 · p. 17

H Evaluation Scenes

The evaluation appendix lists scenes for Objaverse, GSO, and OO3D and states that methods use the same scene list, with camera configurations fixed whenever supported. It also describes potential benefits and risks of NAMVIS’s efficient multi-view generation.

H Evaluation Scenes

  • All methods are evaluated on the same scene list, with source and target camera configurations fixed whenever supported.For methods with camera-protocol constraints, the evaluation follows the protocol described in the main text.Source: H Evaluation Scenes · p. 17

I Broader Impacts

  • NAMVIS is described as potentially lowering the computational barrier and energy footprint for 3D content creation, while rapid realistic generation from sparse inputs carries risks of deceptive media, unauthorized replication, and harmful assets.The stated risks motivate ongoing efforts on watermarking, provenance tracking, and responsible deployment guidelines.Source: I Broader Impacts · p. 17

H Evaluation Scenes

Related papers

Loading 2610.04722v1…