Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations

Mehdi S. M. Sajjadi; Henning Meyer; Etienne Pot; Urs Bergmann; Klaus Greff; Noha Radwan; Suhani Vora; Mario Lucic; Daniel Duckworth; Alexey Dosovitskiy; Jakob Uszkoreit; Thomas Funkhouser; Andrea Tagliasacchi

Source-linked AI summary

Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations

Mehdi S. M. Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lucic, Daniel Duckworth, Alexey Dosovitskiy, Jakob Uszkoreit, Thomas Funkhouser, Andrea Tagliasacchi

arXiv:2111.13152v3Nov 25, 2021cs.CVcs.AIcs.GRcs.LGcs.RO

TL;DR

Interactive novel view synthesis from few images is difficult because prior methods often require precise poses or expensive per-scene processing. SRT uses transformer-based set-latent scene representations and ray-conditioned decoding to render novel views in one feed-forward pipeline. It outperforms prior art on reported benchmarks while supporting efficient real-world visualization and semantic segmentation, with weaker performance expected on very small datasets.

Problem
Interactive novel view synthesis seeks to render unseen views from few RGB images without expensive processing for each scene, while recovering parallax and occlusions.
Method
SRT uses an encoder transformer to build a set-latent representation from posed or unposed images and a decoder transformer that attends to it using target rays.
Results
SRT surpasses prior art on standard benchmarks and a more challenging dataset, while operating orders of magnitudes faster for realistic novel-scene video generation.
Takeaways & Limitations
SRT’s learned scene representation supports interactive real-world visualization and a proof-of-concept semantic-segmentation application.
Takeaways & Limitations
SRT does not work as well on very small datasets as methods with explicit geometric inductive biases, and better training protocols remain to be studied.

Abstract

from arXiv · show

A classical problem in computer vision is to infer a 3D scene representation from few images that can be used to render novel views at interactive rates. Previous work focuses on reconstructing pre-defined 3D representations, e.g. textured meshes, or implicit representations, e.g. radiance fields, and often requires input images with precise camera poses and long processing times for each novel scene. In this work, we propose the Scene Representation Transformer (SRT), a method which processes posed or unposed RGB images of a new area, infers a "set-latent scene representation", and synthesises novel views, all in a single feed-forward pass. To calculate the scene representation, we propose a generalization of the Vision Transformer to sets of images, enabling global information integration, and hence 3D reasoning. An efficient decoder transformer parameterizes the light field by attending into the scene representation to render novel views. Learning is supervised end-to-end by minimizing a novel-view reconstruction error. We show that this method outperforms recent baselines in terms of PSNR and speed on synthetic datasets, including a new dataset created for the paper. Further, we demonstrate that SRT scales to support interactive visualization and semantic segmentation of real-world outdoor environments using Street View imagery.

1. Introduction

Interactive novel view synthesis requires inferring enough 3D information from few images to render correct parallax and occlusions without expensive per-scene processing. SRT addresses this with transformer-based global reasoning, aiming for scalability, robustness, and efficiency across increasingly complex scenes.

Motivation: Interactive novel view synthesis seeks to render unseen views from few RGB images without expensive processing for each scene.The target application includes virtual exploration, mapping, visualization, and AR/VR.
Prior limitations: Traditional explicit representations enable interactive rendering but usually require expensive and fragile reconstruction processes.Examples include colored point clouds, meshes, voxels, octrees, and multi-plane images.
Prior limitations: NeRF-style implicit methods typically require accurate camera poses, hundreds of MLP evaluations per ray, and expensive optimization for every novel scene.Their lack of parameter sharing across scenes contributes to the per-scene training requirement.
Prior limitations: Existing follow-up methods address some shortcomings but generally fail to scale to complex real-world datasets, leaving interactive synthesis challenging.Re-projection methods depend on accurate poses, while latent models can reason globally but remain difficult to scale.
Approach: SRT uses transformer encoder-decoder components and learned attention instead of explicit geometric operations to support global reasoning and instant inference on new scenes.The encoder processes global scene information once, while the decoder remains efficient for rendering novel pixels.
Evidence: Across datasets, SRT balances scalability to complex scenes, robustness to noisy or absent camera poses, and efficiency, while also supporting semantic segmentation as a downstream task.The evaluation includes datasets of increasing complexity and a real-world semantic-segmentation proof of concept.

2. Related Work

Prior novel-view methods span explicit projections, latent scene codes, and transformer-based representations, each trading off geometry, pose requirements, scalability, or rendering speed. SRT combines global set-based scene encoding with attention-driven rendering to target these limitations.

Neural representations: NeRF-based methods produce detailed novel views but generally require expensive per-scene MLP optimization, limiting large-scale deployment.Extensions improve rendering speed, color handling, pose optimization, or few-shot performance without removing the core deployment challenge.
Comparison framework: Table 1 compares methods across real-world evaluation, view coherence, real-time inference, pose availability, sparse inputs, novel-scene generalization, feed-forward capture, and global latent representations.These criteria organize the design trade-offs addressed by SRT and prior approaches.
Re-projection methods: Explicit 3D-to-2D projection methods can be pretrained for novel scenes but require precise camera poses and aggregate features locally or through voxel grids.Their geometry-dependent projections gather CNN features along rays or into volumetric structures.
Light fields: Light-field methods use latent vectors to synthesize views efficiently, but LFN relies on costly auto-decoder inference and does not scale beyond simple controlled settings.LFN queries its radiance network once per pixel rather than hundreds of times per ray, yet still requires latent-code optimization.
Transformers: Transformers use multi-head attention to gather information from relevant input portions and have been applied to geometric and geometry-free vision tasks.The cited applications include depth estimation, point-cloud processing, and generative geometry-free view synthesis.

3. Method

SRT encodes an unordered set of optionally posed RGB images into a variable-size scene representation, then renders ray-conditioned pixel colors through a transformer decoder. It is trained end-to-end from novel-view image reconstruction without explicit 3D supervision, with scene encoding performed once for efficient multi-view rendering.

Input and encoding: The model accepts an unordered collection of same-scene RGB images with optional extrinsic and intrinsic camera information.The extrinsic camera matrix is c_i=[R_i|t_i], while K_i denotes the intrinsic matrix.
Training: Inputs and outputs remain 2D images during training, so the model learns without explicit 3D supervision through an end-to-end novel-view reconstruction loss.Training uses an L2 reconstruction loss over novel views.
Evaluation: Table 2 evaluates novel-view synthesis on simple and challenging datasets using multiple image-quality metrics, with SRT best across datasets and metrics except LPIPS on MSN.The table compares new-scene synthesis against baselines; LPIPS for LFN is unavailable on NMR.
Input and encoding: A shared CNN extracts patch features, which are flattened into unsorted patch-embedding sets before transformer processing.The CNN scales spatial dimensions by K=16, mapping H×W inputs to h=H/K and w=W/K patches.
Encoder transformer: The encoder transformer attends across patches and input images to infer a global 3D scene representation whose size scales with the amount of input information.The resulting set-latent representation fully encodes the scene as observed by the input image set rather than compressing it into a fixed-size latent.
Evaluation: Table 3 measures model size, scene-encoding time, 100-frame rendering rate, and total video-generation time, reporting SRT as orders of magnitudes faster for real-world use.The application metric combines encoding a novel scene with rendering a 100-frame video.
Decoder transformer: The decoder transformer uses a target pixel’s ray as its query and attends to scene-representation keys and values to compute the output color.A two-layer MLP finally converts the decoder output into pixel color.
Inference: At inference, the encoder computes the scene representation once, after which an arbitrary number of novel views can be rendered.This places the scene-level 3D reasoning in a single encoder pass independent of the number of rendered frames.

4. Experimental Results

Experiments show SRT balances image quality, scalability, robustness to camera-pose noise, and interactive rendering across increasingly challenging datasets. Ablations and applications further examine its attention-based scene representation and usefulness for semantic reasoning.

Comparisons to Baselines: SRT achieves the best image quality on NMR and outperforms LFN and PixelNeRF.Both PixelNeRF and SRT encode scenes quickly, but SRT is the only method fast for both scene encoding and novel-view generation.
Comparisons to Baselines: On MSN, SRT outperforms LFN and PixelNeRF in PSNR and SSIM, while PixelNeRF achieves better LPIPS.SRT’s blurring of uncertain regions is offered as an explanation for the LPIPS result.
Comparisons to Baselines: SRT maintains more consistent quality across target views, whereas PixelNeRF and LFN degrade farther from the input views.On MSN, SRT provides the only good reconstructions for far-away target views in the cited qualitative comparison.
Model Inspection: The encoder learns global and spatially corresponding attention, while the decoder first attends globally and then to relevant scene positions.A two-layer decoder appears crucial: one layer performs significantly worse, while additional layers provide no significant gains.
Ablation Studies: Removing the encoder transformer reduces PSNR from 23.41 to 21.64, while replacing the set-latent representation with a flat latent reduces it to 20.88.These ablations support moving computation into the encoder and retaining a large set representation with an attention decoder.
Robustness Study and Applications: SRT handles noisy poses gracefully and also works with fully un-posed imagery, while Street View experiments demonstrate view interpolation and semantic segmentation.The learned representation supports novel views far from inputs and contains enough information for semantic reasoning.

5. Limitations

SRT has limitations on complex data, small datasets, and settings where nearby views and perfect camera poses are available. Its image outputs can also become blurry because pixel-accurate light-ray transformations are uncertain under L2 loss.

Complex datasets produce some blurriness, even for views close to the input cameras.The authors attribute this to uncertainty in learning pixel-accurate light-ray transformations under an L2 loss.
SRT may underperform methods with explicit geometric inductive biases on very small datasets.The authors describe it as a geometry-free learning-based method and note that further training often improves performance.
SRT is strongest with sparse inputs or noisy or missing poses, whereas explicit geometric methods can perform better for nearby views with perfect poses.Those explicit methods may provide better results in this setting, but with much longer inference times.

6. Conclusion

SRT learns scalable neural scene representations from color images and renders novel views without explicit geometry, optionally without 3D poses. The authors report strong quality, substantially faster inference, and suitability for large-scale real-world applications.

SRT learns scalable neural scene representations from color images using a novel encoder-decoder transformer.The architecture renders novel views without explicit geometric reasoning and can optionally operate without 3D image poses.
SRT surpasses prior-art quality on standard benchmarks and a more challenging dataset while being orders of magnitudes faster in realistic settings.The reported combination targets novel-scene video generation and other large-scale applications.

A. Appendix

The appendix provides further details and results, including video results hosted on the project website.

Further details and results are included beyond the main paper.
Video results are available on the project website at srt-paper.github.io.

A.1. Potential Negative Societal Impact

The paper discusses risks from synthesized imagery, privacy concerns, dataset bias, and training energy use. It describes mitigations for image misuse and privacy, while noting systematic sampling and high training compute.

Fakes: Novel-view synthesis could produce misleading or fake images, especially when an adversary selects the input views.The authors plan secure-server deployment for real-world datasets and clear labeling of synthesized imagery.
Privacy: Street View imagery may expose privacy-sensitive information from public outdoor scenes.The stated mitigations include regional-law compliance, anonymization, and secure data storage.
Fairness: The Street View dataset may encode demographic or economic biases because road coverage is city- and route-dependent.Images are captured systematically rather than selected by a human photographer, which might mitigate some other dataset biases.
Energy consumption: Training the transformer requires substantial compute and has a negative environmental impact through energy consumption.The authors report lower inference costs than volumetric-rendering competitors when deployed at scale.

Dataset Details

The experiments use NMR and MultiShapeNet datasets to evaluate novel-view synthesis across object and scene settings.

NMR: NMR contains ShapeNet objects from 13 classes, each rendered from 24 fixed 64×64 views.The SoftRas split includes 31 k training, 4 k validation, and 9 k test objects; one input image is used to render the other 23 views.
MultiShapeNet: MultiShapeNet uses 1 M scenes for training and 10 k scenes for testing.

SRT Model Details

SRT uses convolutional features with ViT-like encoder and decoder transformers, trained end-to-end for rendering; its representation also shows coherent 3D behavior in EPI analysis.

Architecture: The CNN has four two-layer blocks, downsamples through strided convolutions, and maps final activations to 768 channels.It begins with 96 channels and doubles channels at each strided convolution.
Architecture: Both transformers use 12 attention heads with 64 channels, while their MLPs use 1536 hidden channels.The decoder output MLP instead uses 128 hidden channels and a final sigmoid for RGB prediction.
Training: Training uses Adam with a learning rate decayed from 1 × 10^-4 to 1.6 × 10^-5 over 4 M steps.Each batch has 256 examples, with 8192 points sampled across target views for rendering and loss computation.
Representation analysis: SRT’s EPI for dolly motion indicates a coherent 3D scene representation whose consistency may support depth extraction.
Training: Validation metrics can continue improving beyond 4 M steps and up to the 10 M-step training maximum.Later qualitative improvements become increasingly subtle.

Baseline Details

The comparisons cover optimization-based, appearance-aware, unposed, semantic, volumetric, and robustness variants, emphasizing speed, pose handling, depth, and real-world deployment constraints.

Efficiency: LFN requires per-scene latent-code optimization, whereas PixelNeRF and SRT encode a novel scene in 5–10 ms.The timing is measured on a single NVIDIA V100 during scene onboarding.
Appearance handling: An appearance encoder addresses exposure and white-balance variation by modifying rendered colors without changing scene structure.At test time, an input image supplies the appearance embedding.
Semantic segmentation: The semantic decoder is trained with cross-entropy against one-hot class predictions generated by a pretrained 2D segmentation model.
UpSRT Model Inspection: UpSRT can use unposed imagery: attention patterns show it integrates all input images, while its decoder need not attend strongly to the reference camera.The reference camera supplies the coordinate frame, but the model is reported to use other views for reconstruction.
UpSRT Model Inspection: 22.2 / 0.65 / 0.41 are UpSRT’s PSNR / SSIM / LPIPS values on MSN, compared with 19.2 / 0.53 / 0.52 for one-image SRT.UpSRT also reconstructs an object visible only in other unposed input views at the correct 3D position.
Volumetric rendering: V-SRT offers depth maps with render quality comparable to SRT, while its volumetric rendering is slower at inference.Street View depth can be inaccurate under lighting variation, such as a metallic car changing color across views.
Robustness and baselines: Camera noise can severely degrade geometry-based methods even when the perturbation is visually subtle.SRT’s EPI analysis instead reports a coherent 3D scene, and NeRF-based methods fail on the challenging Street View dataset with only five images.