PaperlayerView exact revision on arXiv

Source-linked AI summary

LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias

Haian Jin, Hanwen Jiang, Hao Tan, Kai Zhang, Sai Bi, Tianyuan Zhang, Fujun Luan, Noah Snavely, Zexiang Xu

arXiv:2410.17242v2cs.CVcs.GRcs.LG

TL;DR

Novel view synthesis must generalize from sparse posed inputs without depending on restrictive 3D representations or handcrafted architectural priors. LVSM uses transformer architectures that either learn a fixed latent scene representation or directly map input views to targets, with decoder-only LVSM delivering the strongest quality and scalability while encoder-decoder LVSM is faster at inference. Both variants achieve state-of-the-art results, including a 1.5 to 3.5 dB PSNR advantage over prior methods, but deterministic LVSM struggles in unseen regions and with out-of-distribution image aspect ratios and resolutions.

  • Problem

    Existing novel view synthesis methods rely on 3D representations or architectural inductive biases that limit flexibility, generalization, and scalability.

  • Method

    LVSM uses transformer architectures that either encode input image tokens into fixed-length learned latent tokens or directly map input-view and target-pose tokens to target-view tokens.

  • Results

    1.5 to 3.5 dB PSNR: decoder-only LVSM surpasses previous state-of-the-art methods, while both variants achieve state-of-the-art quality across object-level and scene-level benchmarks.

  • Takeaways & Limitations

    Decoder-only LVSM provides superior quality, scalability, and zero-shot generalization, whereas encoder-decoder LVSM offers faster inference through a fixed-length latent representation.

  • Takeaways & Limitations

    Deterministic LVSM struggles with high-quality synthesis in unseen regions and degrades on image aspect ratios and resolutions outside those used during training.

Abstract

from arXiv · show

We propose the Large View Synthesis Model (LVSM), a novel transformer-based approach for scalable and generalizable novel view synthesis from sparse-view inputs. We introduce two architectures: (1) an encoder-decoder LVSM, which encodes input image tokens into a fixed number of 1D latent tokens, functioning as a fully learned scene representation, and decodes novel-view images from them; and (2) a decoder-only LVSM, which directly maps input images to novel-view outputs, completely eliminating intermediate scene representations. Both models bypass the 3D inductive biases used in previous methods -- from 3D representations (e.g., NeRF, 3DGS) to network designs (e.g., epipolar projections, plane sweeps) -- addressing novel view synthesis with a fully data-driven approach. While the encoder-decoder model offers faster inference due to its independent latent representation, the decoder-only LVSM achieves superior quality, scalability, and zero-shot generalization, outperforming previous state-of-the-art methods by 1.5 to 3.5 dB PSNR. Comprehensive evaluations across multiple datasets demonstrate that both LVSM variants achieve state-of-the-art novel view synthesis quality. Notably, our models surpass all previous methods even with reduced computational resources (1-2 GPUs). Please see our website for more details: https://haian-jin.github.io/projects/LVSM/ .

1 INTRODUCTION

LVSM addresses novel view synthesis by minimizing 3D inductive biases through a fully data-driven transformer framework. It introduces encoder-decoder and decoder-only variants that improve quality, scalability, and generalization while trading off inference speed and representation compression.

  • Motivation: Novel view synthesis methods commonly rely on 3D representations, rendering equations, or architectural structures such as epipolar projections and plane-sweep volumes.These approaches include NeRF, 3D Gaussian Splatting, and related generalizable networks.
  • LVSM: LVSM removes predefined rendering equations and 3D structures to synthesize photo-realistic novel views from posed sparse-view inputs.The framework targets accurate, training-efficient, and scalable feed-forward synthesis.
  • Encoder-decoder LVSM: The encoder-decoder LVSM maps patchified multi-view image tokens into a fixed number of learned 1D latent tokens before decoding target-view image tokens.Its latent representation is independent of the number of input views and uses target-view Plücker rays as positional embeddings.
  • Decoder-only LVSM: The decoder-only LVSM directly converts input-view and target-pose tokens into target-view tokens, bypassing intermediate scene representations.This design performs scene reconstruction and rendering jointly and implicitly with minimal 3D inductive bias.
  • Evaluation: 1.5 to 3.5 dB PSNR separates decoder-only LVSM from previous state-of-the-art methods such as GS-LRM.Both variants achieve state-of-the-art quality across object-level and scene-level benchmarks, while the decoder-only model leads in quality, scalability, and zero-shot capability.
  • Evaluation: The encoder-decoder variant provides faster inference through its fixed-length latent scene representation, whereas decoder-only LVSM scales better with varying numbers of input views.The evaluation covers models trained with 2 or 4 input views and tested zero-shot from a single input to more than 10 views.

2 RELATED WORK

Prior novel view synthesis methods range from image-based rendering and light fields to optimized or predicted 3D representations. LVSM follows recent transformer-based work that removes some architectural biases while retaining a deterministic, token-based synthesis design.

  • Image-based and light-field methods: Image-based rendering blends reference images using proxy geometry, while light-field methods construct slices of the 4D plenoptic function from dense views.Learning-based image-based rendering predicts blending weights or depth maps, but typically remains constrained near the input views.
  • Optimizing 3D representations: NeRF introduced differentiable volumetric 3D representations optimized with rendering losses, establishing a foundation for neural scene reconstruction.Subsequent work targeted rendering quality, optimization speed, rendering speed, and reduced requirements.
  • Generalizable view synthesis: Generalizable view synthesis networks predict novel views or underlying 3D representations feed-forward across scenes, often using epipolar lines or plane-sweep cost volumes.These methods include PixelNeRF, MVSNeRF, and IBRNet.
  • Large reconstruction models: Large reconstruction models use scalable transformers to learn generic 3D priors but still rely on representations such as tri-plane NeRFs, meshes, or 3DGS.LVSM instead minimizes these representation-level 3D biases.
  • Generative novel view synthesis: Recent generative novel view synthesis models use image or video diffusion, whereas LVSM is deterministic.This distinction separates LVSM from diffusion-based approaches in the related-work landscape.

3 METHOD

LVSM uses an end-to-end transformer pipeline that tokenizes posed sparse-view inputs and target rays, then predicts target-view pixels with minimal 3D inductive bias. It provides encoder-decoder and decoder-only variants, trading faster fixed-latent inference for direct implicit synthesis.

  • 3.1 OVERVIEW: LVSM patchifies each input image and its Plücker ray embeddings, concatenates corresponding patches, and projects them into input tokens.Target camera extrinsics and intrinsics are likewise converted into patched Plücker ray tokens.
  • 3.1 OVERVIEW: A full transformer updates target query tokens conditioned on flattened input tokens, producing one output token for each target-view patch.Each output token contains information for predicting the pixel values of its corresponding target patch.
  • 3.1 OVERVIEW: A linear layer followed by a sigmoid regresses RGB values from each target token, after which patches are reshaped and assembled into the synthesized image.Training uses a photometric loss combining MSE and perceptual loss.
  • 3.2 TRANSFORMER-BASED MODEL ARCHITECTURE: The encoder-decoder variant maps input tokens to a fixed-length learned latent scene representation before decoding target-view outputs.The encoder aggregates input information into learnable latent tokens, while the decoder conditions on those tokens and target pose queries.
  • 3.2 TRANSFORMER-BASED MODEL ARCHITECTURE: The decoder-only variant concatenates input and target tokens in one transformer, discards updated input tokens, and directly outputs target tokens without an intermediate representation.Both encoder and decoder architectures use dense full self-attention across their tokens.
  • 3.2 TRANSFORMER-BASED MODEL ARCHITECTURE: Table 1 compares object-level results at 256 and 512 resolution and scene-level results at 256 resolution under matched baseline settings.The object-level comparison matches GS-LRM in input and rendering settings, while the scene-level comparison uses the pixelSplat validation dataset.

4 EXPERIMENTS

LVSM is evaluated on object- and scene-level novel view synthesis, with comparisons, visual analyses, zero-shot view-count tests, and architecture/model-size ablations. The decoder-only variant delivers stronger quality and scalability, while the encoder-decoder variant provides faster inference through fixed-length latent tokens.

  • Experimental setup: LVSM is trained and evaluated separately on object-level and scene-level datasets, using Objaverse for training and GSO and ABO for object-level testing.The object-level training setup renders 32 random views of 730K Objaverse objects; GSO and ABO contain 1,099 and 1,000 objects, respectively.
  • Object-level results: 3.5 dB and 2.2 dB PSNR gains over GS-LRM are reported for decoder-only LVSM on ABO and GSO, respectively, at 256 resolution.The encoder-decoder LVSM performs slightly better than GS-LRM in the same comparison.
  • Scene-level results: 1.6 dB PSNR gain over GS-LRM is reported for decoder-only LVSM on scene-level inputs, while encoder-decoder LVSM achieves comparable results.Qualitative comparisons report fewer floaters and better performance on thin structures and specular materials.
  • Zero-shot generalization: Decoder-only LVSM handles more input views increasingly well after training with four views, whereas encoder-decoder LVSM drops in performance beyond eight input views.Single-input results are competitive and sometimes outperform baselines using four images.
  • Efficiency and trade-offs: The encoder-decoder variant renders efficiently because its decoder always receives a fixed number of latent tokens, independent of input-image count.Its fixed-length latent representation can lose information because it compresses longer image-token sequences.
  • Efficiency and trade-offs: Decoder-only LVSM scales with additional input information but incurs quadratic computational growth as input-image tokens increase, limiting rendering speed.The model directly maps input images to target novel views without an intermediate representation.

5 CONCLUSION

LVSM minimizes 3D inductive biases through encoder-decoder and decoder-only transformer architectures for scalable and generalizable novel view synthesis. The decoder-only model prioritizes quality, scalability, and zero-shot generalization, while the encoder-decoder model provides faster inference.

  • LVSM uses encoder-decoder and decoder-only architectures to bypass physical-rendering-based 3D representations and learn priors directly from data.
  • The decoder-only LVSM excels in scalability, zero-shot generalization, and rendering quality.
  • The encoder-decoder LVSM achieves faster inference through its fully learned latent scene representation.
  • Both LVSM architectures demonstrate superior performance across diverse benchmarks for novel view synthesis.

A.1 NAMING CLARIFICATION

LVSM names its encoder and decoder according to their outputs rather than strictly according to transformer architecture conventions.

  • The encoder is named for producing latent tokens, while the decoder is named for producing the target view.
  • In the encoder-decoder model, the decoder uses transformer layers with self-attention despite being called a decoder.

A.2 ADDITIONAL IMPLEMENTATION DETAILS

LVSM training uses large-scale transformer configurations, staged resolutions, and optimization choices shared across the reported experiments.

  • LVSM is trained with 64 A100 GPUs, batch size 8 per GPU, a peak learning rate of 4e-4, and 80k or 100k iterations depending on data type.
  • The model uses image patch size p = 8, token dimension d = 768, 24 transformer layers, and 3072 latent tokens in the encoder-decoder variant.
  • Object-level training uses 4 input and 8 target views, while scene-level training uses 2 input and 6 target views per example.
  • Training proceeds from resolution 256 to 512, with 10k fine-tuning iterations at a learning rate of 4e-5 and batch size 128.
  • LVSM omits bias terms and initializes weights with a layer-index-dependent normal distribution.
  • AdamW uses β1 = 0.9, β2 = 0.95, and weight decay 0.05 except for LayerNorm weights.

A.3 ADDITIONAL VISUAL RESULTS

Additional visual results show that LVSM outperforms baselines on challenging appearance and geometry details at 256 resolution.

  • At 256 resolution, LVSM performs better than baselines on texture details, specular material, and concave geometry.

A.4 DETAILED MODEL ARCHITECTURE

Figure 8 provides the detailed model architecture for LVSM.

  • Figure 8 presents the detailed model architecture referenced for LVSM.

A.5 ATTENTION MASK ILLUSTRATION FOR DIFFERENT DESIGN CHOICES

The section visualizes attention-mask choices and compares LVSM architectures with baselines at 256 resolution. The models reduce artifacts and improve view-dependent effects, while Decoder-Only better preserves texture details.

  • At 256 resolution, both LVSM variants show fewer floater artifacts and more accurate view-dependent effects than LGM and GS-LRM.
  • The Decoder-Only model better preserves texture details than the compared baselines in the first two examples.
  • Figure 8 contrasts encoder-decoder LVSM, which uses fixed latent tokens, with decoder-only LVSM, which removes intermediate scene representations.
  • Both LVSM architectures use bidirectional self-attention modules, with Figure 9 visualizing masks for different design choices.

A.6 DISCUSSION OF DIFFERENCES WITH PRIOR GENERATIVE NVS MODELS

LVSM is deterministic, unlike prior probabilistic generative NVS models, supporting independently generated views with precise camera control and view consistency. Its main limitations are difficulty with unseen regions and degraded performance under unfamiliar aspect ratios or resolutions.

  • LVSM’s deterministic models avoid the inherent multi-view inconsistency of probabilistic generative models.
  • After training on large-scale multi-view data, LVSM can independently generate target images with precise camera control while maintaining view consistency.
  • Unlike prior generative models, LVSM can generate an unlimited number of consistent views for observed regions of reconstructed scenes.
  • LVSM struggles to produce high-quality results in unseen regions, often producing noisy and flickering artifacts with fixed patterns.
  • Performance degrades on aspect ratios and resolutions different from training, including blurred horizontal boundaries for 512×512-trained models tested on 512 × 960 inputs.
Loading 2410.17242v2…