PaperlayerView exact revision on arXiv

Source-linked AI summary

LagerNVS: Latent Geometry for Fully Neural Real-time Novel View Synthesis

Stanislaw Szymanowicz, Minghao Chen, Jianyuan Wang, Christian Rupprecht, Andrea Vedaldi

arXiv:2603.20176v3cs.CV

TL;DR

NVS needs fast, effective view synthesis without relying on explicit per-scene reconstruction, while preserving useful 3D structure. LagerNVS addresses this with an encoder initialized from an explicitly supervised 3D reconstruction model and a lightweight decoder, achieving state-of-the-art deterministic feed-forward NVS with broad generalization and real-time decoding. Its decoder also supports a preliminary generative diffusion extension.

  • Problem

    Reconstruction-free NVS removes explicit 3D reconstruction, but the paper asks whether strong 3D inductive biases remain useful for designing such networks.

  • Method

    LagerNVS uses an encoder initialized from a 3D reconstruction network to produce 3D-aware latent features, paired with a lightweight viewpoint-conditioned decoder.

  • Results

    LagerNVS achieves state-of-the-art deterministic feed-forward NVS, including a +1.7dB PSNR margin over LVSM, while supporting real-time decoding and broad generalization.

  • Takeaways & Limitations

    3D-aware latent features can improve reconstruction-free NVS, while the resulting model remains efficient, robust across input settings, and applicable to generative NVS.

  • Takeaways & Limitations

    The camera setup assumes equal horizontal and vertical focal lengths, and unknown-camera inputs use a canonical target field of view while training retains the unknown source focal length.

Abstract

from arXiv · show

Recent work has shown that neural networks can perform 3D tasks such as Novel View Synthesis (NVS) without explicit 3D reconstruction. Even so, we argue that strong 3D inductive biases are still helpful in the design of such networks. We show this point by introducing LagerNVS, an encoder-decoder neural network for NVS that builds on `3D-aware' latent features. The encoder is initialized from a 3D reconstruction network pre-trained using explicit 3D supervision. This is paired with a lightweight decoder, and trained end-to-end with photometric losses. LagerNVS achieves state-of-the-art deterministic feed-forward Novel View Synthesis (including 31.4 PSNR on Re10k), with and without known cameras, renders in real time, generalizes to in-the-wild data, and can be paired with a diffusion decoder for generative extrapolation.

1. Introduction

LagerNVS argues that reconstruction-free NVS still benefits from 3D-aware features and efficient encoder-decoder design. It achieves strong deterministic results, broad generalization, real-time rendering, and a preliminary generative extension.

  • Motivation: NVS renders new scene views from other views, while conventional optimization-based reconstruction can be slow and overfit with few source views.Feed-forward methods avoid per-scene optimization, and some directly output new views without explicit reconstruction.
  • Core idea: LagerNVS uses 3D-aware latent features from an encoder initialized with explicit-3D-supervised reconstruction weights, while bypassing explicit reconstruction at inference.The design tests whether strong 3D inductive biases remain useful in reconstruction-free NVS.
  • Architecture: Highway encoder-decoder architectures provide an excellent quality-speed trade-off because encoding runs once per scene and decoding is conditioned on each target viewpoint.The paper compares decoder-only, bottleneck encoder-decoder, and highway encoder-decoder designs.
  • Results: +1.7dB PSNR separates LagerNVS from LVSM on RealEstate10k, while the model also outperforms feed-forward 3D reconstruction networks.This establishes the reported state-of-the-art result for deterministic NVS.
  • Generalization and extension: Training on a mixture of datasets supports generalization across ego-centric, 360°, non-square, and in-the-wild images with unknown source poses.A diffusion-denoising decoder additionally enables promising generative results when source views leave scene regions ambiguous.
  • Efficiency: 30FPS+ decoding at 512×512 with up to nine source images runs on one H100 GPU, while encoding takes seconds.The renderer uses a standard neural network without explicit 3D representations, custom kernels, or JIT compilation.

2. Related work

Related NVS methods range from optimized explicit 3D reconstruction to feed-forward latent representations and direct decoding. These approaches differ in camera assumptions, intermediate representations, and rendering efficiency.

  • Explicit 3D representations: Explicit NeRF or 3D Gaussian representations map 3D locations to local properties but require optimization and can be slow or overfit with limited views.Feed-forward networks were introduced to extract such representations more quickly.
  • Camera assumptions: Some reconstruction methods relax the assumption of known source cameras, often using pretrained multi-view reconstruction models.LagerNVS instead uses those models’ latent features rather than their explicit 3D outputs.
  • Latent representations: Latent-representation methods encode scenes into features decoded directly into new views without necessarily representing explicit 3D properties.SRT introduced feed-forward encoder-decoder extraction of such representations.
  • Decoder-only methods: Decoder-only NVS maps source images and a target camera directly to the target image, requiring the full model to run for every rendered view.This design limits rendering speed compared with architectures that reuse a scene encoding.
  • Generative NVS: Generative NVS uses diffusion decoders to produce plausible completions when target views contain regions absent from the source images.LagerNVS focuses on deterministic NVS but includes a preliminary generative experiment.

3. Method

LagerNVS separates source-image encoding from target-view decoding, using latent features with implicit 3D bias rather than explicit 3D reconstruction. Its highway design preserves source-view information while camera-conditioned transformer decoding produces target images efficiently.

  • NVS formulation: NVS maps source images and an optional set of source cameras to a target image specified by camera parameters g.The target camera is expressed relative to the first source camera.
  • Encoder-decoder design: An encoder-decoder first computes a target-camera-independent representation z, then decodes each requested view as I = h(z; g).This amortizes the cost of computing z across multiple target views.
  • Encoder-decoder design: Highway encoder-decoders retain separate feature vectors for each source image, unlike bottleneck models whose latent token count is independent of the number of views.The highway design is intended to maximize information flow from source images to the latent representation.
  • Implicit 3D bias: LagerNVS uses neural encoder and decoder networks while initializing the encoder from a network trained for explicit 3D reconstruction, producing 3D-aware latent features without explicit reconstruction.Its encoder builds on VGGT features rather than directly using VGGT’s predicted cameras or depth maps.
  • Decoder: The decoder represents the target camera as a dense Plücker ray map and uses transformer attention from camera tokens to encoded source-image tokens.The decoder outputs image patches that are reshaped to the target image size.
  • Training: Training minimizes a combination of mean-squared and perceptual losses, with end-to-end fine-tuning of the pretrained encoder found essential for good results.The loss is L = λ2L2 + λpLp.

4. Experiments

The experiments evaluate LagerNVS against prior NVS methods, across datasets, architectures, attention variants, ablations, and generative extensions. Results show strong deterministic quality, real-time rendering, generalization, and benefits from 3D-aware pre-training and end-to-end optimization.

  • Experimental setup: LagerNVS is evaluated on RealEstate10k, DL3DV, and CO3D using PSNR, SSIM, and LPIPS.The experiments match baseline setups where applicable and treat decoder transformer blocks as the rendering-complexity control variable.
  • Comparison to LVSM: LagerNVS outperforms LVSM by up to +1.7dB PSNR across both small and large training scales.The comparison uses batch sizes 64 and 512, with full- and cross-attention variants evaluated against LVSM models.
  • Generalization: LagerNVS generalizes to in-the-wild data and different scene types, works without source camera poses, and supports single-view NVS.Single-view translation requires training with point-based scaling, while the full model supports small camera motion.
  • Analysis and ablations: 3D pre-training improves performance by +2.9dB PSNR over training from scratch.Generic 2D pre-training provides only a marginal improvement, while 3D pre-training especially benefits foreground depth estimation.
  • Analysis and ablations: The highway encoder-decoder achieves the best quality for a fixed decoding budget, while cross-attention enables 56–30 FPS rendering with 1–9 source views.The highway design avoids a bottleneck and shifts view-independent computation into the encoder; full attention is slightly better but supports only 6 source images at 30FPS+ versus 9 for cross-attention.
  • Generative NVS: The diffusion variant produces plausible completions for occlusion and extrapolation cases where deterministic regression yields blurry outputs.The pre-trained model adapts after 60k fine-tuning iterations using pixel-space diffusion with 12 transformer blocks.

5. Conclusion

LagerNVS shows that strong 3D inductive biases remain beneficial for NVS without explicit 3D representations. Its efficient, robust model achieves strong deterministic results and also supports generative NVS.

  • LagerNVS demonstrates benefits from strong 3D inductive biases in NVS models that do not use explicit 3D representations.
  • The model uses a pre-trained 3D reconstruction network and a lightweight decoder to achieve state-of-the-art NVS across implicit and feed-forward 3DGS models.
  • LagerNVS renders 512 × 512 views in real time on a single GPU, while remaining robust, efficient, and usable without camera poses.
  • The model can also be applied to generative NVS through a diffusion decoder, with further benefits reported by the authors.

A. Additional experimental results

Additional experiments compare LagerNVS with neural and feed-forward 3DGS baselines across benchmarks and qualitative settings. The results emphasize challenging geometry, occlusions, and the trade-off between model quality and capacity.

  • Quantitative evaluation is expanded to standard benchmarks at 512×512 resolution, with and without known camera poses.
  • LagerNVS performs better than LVSM on challenging cases involving low overlap, repeated patterns, and geometry close to the camera.
  • LagerNVS outperforms SVSM due to greater model capacity and 3D pre-training, although its network is larger.
  • Qualitative comparisons show improved handling of reflective surfaces, thin structures, and occluded regions relative to feed-forward 3DGS methods.
  • In difficult occlusion settings, deterministic completions become blurry, although they remain reasonable in the reported examples.

B. Diffusion

The diffusion extension adapts LagerNVS’s decoder to generate plausible completions for ambiguous or occluded regions. The method modifies the decoder minimally but remains limited for consistent video generation.

  • The diffusion decoder adds timestep conditioning and expands the input patch embedding to accept noisy-image channels.
  • The occlusion examples show successful simple completions, while harder cases produce blurry but still reasonable outputs.
  • The diffusion model is trained with a learning rate of 1 × 10−5 for 60k iterations.
  • The diffusion decoder generates individual novel views rather than temporally consistent videos, so video output would necessarily flicker.

C.1. Architecture: encoder

The encoder adapts VGGT to produce a latent 3D representation from image and camera tokens. It combines per-image and global sequence processing before projecting features for the decoder.

  • Source images are resized so their longer side is 518, matching the input dimension expected by VGGT.
  • Camera tokens are projected to dimension 1024 and use a reference-frame convention based on the first camera.
  • The VGGT-initialized backbone combines a 24-layer per-image transformer with a 48-layer aggregator using local and global attention.
  • Two VGGT features are projected from channel dimension 1024 to 768 and concatenated across images to form the latent 3D representation.

C.2. Architecture: decoder

The decoder combines latent scene tokens with target-camera tokens in a Vision Transformer, using attention variants that trade rendering quality against speed. The model uses bidirectional cross-attention as a practical middle ground, while a camera-intrinsics update addresses focal-length leakage and train-test mismatch.

  • Decoder design: The encoder passes scene tokens to the decoder after concatenating VGGT local- and global-attention tokens and projecting them to decoder channels.Camera tokens are discarded before projection to the decoder representation.
  • Attention variants: Bidirectional cross-attention updates camera and scene tokens through cross-attention while allowing camera-token self-attention but no scene-token self-attention.The main model selects this variant as a middle ground between speed and quality.
  • Attention variants: Unidirectional cross-attention is excluded because its performance gap relative to full attention grows from −0.8 PSNR with two source views to −1.9 PSNR as source views increase.The passage reports this comparison in the supplementary attention-variant table.
  • Camera conditioning: The updated model avoids leaking focal-length information when source cameras are absent by preventing target intrinsics from encoding the shared training focal length.The change addresses a train-test mismatch caused by differing target intrinsics and slightly lowers benchmark performance while retaining state-of-the-art status.
  • Decoder design: The decoder tokenizes the target camera as a Plücker ray map and combines its tokens with latent scene tokens in a Vision Transformer.After processing, register and scene tokens are discarded, and novel views are read from the remaining tokens.

C.4. Training details

Training uses diverse multi-view data, variable source-view and camera conditioning, and scale normalization to support different inference settings. Point-based normalization is especially important for single-view translation, while the model remains constrained to static scenes and supported visual conditions.

  • Data: Training uses a rich mixture of 13 multi-view datasets and approximately 45k Internet-style videos to improve dataset breadth and generalization.The data include synthetic, benchmark, egocentric, and in-the-wild sources.
  • Efficiency: The decoder processes one target view while the encoder processes all source images, so training predicts multiple target views per scene encoding to rebalance computation.The target views are rendered in parallel but independently.
  • Augmentation: The model samples 1–10 source views during training, drops camera-pose tokens for 40% of examples, and predicts eight target views per source set.These augmentations enable variable numbers of source images and camera-availability conditions at test time.
  • Scene scale: Scale normalization relates target-camera motion to scene scale, which is otherwise ambiguous under common rescaling of cameras and 3D geometry.With multiple known-camera views, triangulation can infer this relationship; single-view or unposed settings require additional information or assumptions.
  • Scene scale: When camera-based normalization is unavailable for a single or overlapping-view setup, the model uses point-based normalization from average distances between the reference camera and visible scene points.The network receives both normalization factors as camera parameters and can train with either factor missing.
  • Scene scale: Adding point-based scale training enabled successful camera-translation renders from a single source image, whereas camera-only scaling supported rotation but not translation.This comparison motivates training with both normalization methods.

D. Limitations

The model assumes matching camera intrinsics across source and target images and has limited quality when hallucinating unseen or high-frequency regions.

  • Camera intrinsics: Providing source images with different focal lengths or target intrinsics can deteriorate performance because training uses identical intrinsics throughout.The authors suggest focal-length randomization as a route toward a more general model.
  • Unseen regions: Unseen-region renderings can be blurry and contain block artifacts, sometimes appearing as flicker in video outputs.High-frequency patterns such as grass or trees are also systematically poorly represented.
Loading 2603.20176v3…