PaperlayerView exact revision on arXiv

Source-linked AI summary

Geometry-Free View Synthesis: Transformers and no 3D Priors

Robin Rombach, Patrick Esser, Björn Ommer

arXiv:2104.07652v2cs.CV

TL;DR

Single-image view synthesis must handle large viewpoint changes and multiple plausible target views. The paper uses probabilistic transformers to model this task, finding that they learn view relationships without explicit geometric priors and benefit from probabilistic modeling.

  • Problem

    Single-image synthesis of large viewpoint changes is an underdetermined extrapolation task, with many target images consistent with the source and camera pose.

    Source: §1 · p. 1, §3 · p. 3

  • Method

    The paper models target views probabilistically with an autoregressive transformer, comparing explicit depth-and-warp conditioning with implicit conditioning.

    Source: §3 · p. 3, §3.2 · p. 4

  • Results

    Transformers learn the required transformations without built-in priors, and the probabilistic approach brings significant benefits over previous state-of-the-art methods for large camera transformations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Explicitly coding 3D transformations is not necessary for the transformer to learn them, and explicit depth information offers relatively small benefits.

    Source: §1 · p. 2

    Without camera-motion priors or warping layers, the model needs a compressed representation; compression artifacts dominate error for small viewpoint changes, while avoiding them costs more computation.

    Source: §5 · p. 8

Abstract

from arXiv · show

Is a geometric model required to synthesize novel views from a single image? Being bound to local convolutions, CNNs need explicit 3D biases to model geometric transformations. In contrast, we demonstrate that a transformer-based model can synthesize entirely novel views without any hand-engineered 3D biases. This is achieved by (i) a global attention mechanism for implicitly learning long-range 3D correspondences between source and target views, and (ii) a probabilistic formulation necessary to capture the ambiguity inherent in predicting novel views from a single image, thereby overcoming the limitations of previous approaches that are restricted to relatively small viewpoint changes. We evaluate various ways to integrate 3D priors into a transformer architecture. However, our experiments show that no such geometric priors are required and that the transformer is capable of implicitly learning 3D relationships between images. Furthermore, this approach outperforms the state of the art in terms of visual quality while covering the full distribution of possible realizations. Code is available at https://git.io/JOnwn

1. Introduction

The paper targets single-image view synthesis under large camera changes, where many target views may be plausible. It proposes probabilistic transformer modeling and finds that transformers can learn the required 3D relationships without explicit geometric priors.

  • The task extrapolates from a single image to large viewpoint changes, unlike multi-view synthesis that is closer to view interpolation.Source: §1 · p. 1
  • Because large viewpoint changes admit many consistent target images, reconstruction can blur alternatives and adversarial training can drop modes; the paper instead maximizes target-data likelihood with an autoregressive transformer.Source: §1 · p. 2
  • The transformer learns the view transformation without built-in priors, and can predict depth without supervision; adding explicit depth provides relatively small benefits.Source: §1 · p. 2
  • The probabilistic model brings significant benefits over prior state-of-the-art approaches for large camera transformations, while the transformer learns 3D transformations without built-in priors.Source: §1 · p. 2

2. Related Work

Prior novel-view-synthesis methods span different input requirements and degrees of geometric modeling, while single-image methods often use predicted geometry. The paper situates its transformer among attention-based and two-stage generative modeling approaches.

  • Earlier multi-view systems trade input structure against geometric modeling: needs many structured views, a coarse volumetric estimate, and an accurate photogrammetric model.Source: §2 · p. 2
  • Single-image methods include CNN view or appearance-flow prediction and geometry-based warping or 3D representations; prior results degrade for large changes when uncertainty is not modeled.Source: §2 · pp. 2–3
  • Figure 2 summarizes conditioning variants, contrasting explicit depth estimation and feature warping with implicit conditioning that learns the relationship without warping.Source: Figure 2 · p. 3

Self-Attention and Transformers

  • Self-Attention and Transformers: Attention weighs sequence interactions regardless of relative position, avoiding the locality bias associated with CNNs; the model uses GPT-2 architecture.Source: §2 · p. 3

Generative Two Stage Approaches

  • Generative Two Stage Approaches: VQGANs provide compressed, high-fidelity discrete representations on which autoregressive likelihood models can be trained.Source: §2 · p. 3

3. Approach

The approach models plausible target views probabilistically with a transformer, compares explicit and implicit geometry encodings, and probes whether the transformer represents depth.

3.1. Probabilistic View Synthesis in Latent Space

  • A VQGAN supplies discrete image latents, then an autoregressive transformer models target-token likelihood conditioned on the source image and camera transformation.Source: §3.1 · p. 4

3.2. Encoding Inductive Biases

  • Explicit variants warp source information using camera geometry, while implicit variants provide the source image and camera parameters for the transformer to learn their relationship; tested variants include depth-free and hybrid conditioning.Source: §3.2 · pp. 4–5

3.3. Depth Readout

  • To test whether impl.-nodepth encodes geometry, a position-wise linear classifier predicts frozen VQGAN depth codes from layer-specific source-frame representations.Source: §3.3 · p. 5
  • With four samples, impl.-depth reaches state-of-the-art in two of three metrics; with 16, it reaches state-of-the-art in all three.Source: Figure 3 · p. 6

4. Experiments

The experiments compare explicit and implicit geometric biases in transformers, evaluate novel-view quality against prior approaches, and probe whether the least-biased model represents scene geometry. The results show strong performance from probabilistic implicit variants and depth information that becomes linearly recoverable in deeper transformer representations.

4.1. Comparing Implicit and Explicit Transformers

  • On ACID, implicit variants and the hybrid achieve the best NLL, while explicit warping adds no significant benefit; on RealEstate, impl.-nodepth is slightly worse than expl.-img.Supervised monocular depth is more beneficial in the indoor RealEstate setting than in outdoor ACID.Source: §4.1 · p. 6

4.2. Comparison to Previous Approaches

  • With 32 samples, probabilistic variants lead PSIM, SSIM, and PSNR on RealEstate and rank among the top three on ACID; implicit variants have the best and second-best FID scores on both datasets.With four samples, impl.-depth beats all other approaches except 3DPhoto on SSIM; it overtakes 3DPhoto on SSIM with 16 samples.Source: Table 2 · p. 7, Table 3 · p. 7, §4.2 · p. 7

4.1. Comparing Implicit and Explicit Transformers

  • Implicit variants with depth outperform explicit variants; impl.-nodepth matches their FID but trails in reconstruction fidelity while remaining on par with explicit variants without depth supervision.For impl.-nodepth, lower predictive entropy in source-visible regions indicates geometric source–target relations without 3D bias.Source: §4.1 · pp. 6–7, Figure 4 · p. 6

4.2. Comparison to Previous Approaches

  • Probabilistic implicit variants produce plausible views, while compared deterministic baselines show artifacts, blur, or uniform regions.The displayed target is one of many possible realizations.Source: §4.2 · p. 7, Figure 5 · p. 7

4.3. Probing for Geometry

  • Linear depth probing peaks at l = 4, after the third self-attention block, and the recovered maps resemble ground-truth depth; such estimates are unavailable from the VQGAN encoder representation at l = 0.The authors hypothesize that the transformer develops an implicit 3D representation to perform view mapping.Source: §4.3 · p. 8

5. Discussion

The paper presents probabilistic transformer-based novel view synthesis for strong viewpoint changes. It reports that explicit 3D transformations do not significantly improve performance, while compression artifacts and computational cost remain trade-offs.

5. Discussion

  • The proposed approach combines probabilistic modeling and transformers for novel view synthesis from a single image with strong viewpoint changes.The discussion describes this as an important step, not a final solution.Source: §5 · p. 8
  • Depth probing is strongest at transformer layer 4, while the right-column reconstructions provide an upper bound on achievable quality.The caption says this pattern mirrors the curve in Fig. 7.Source: Figure 8 · p. 8
  • Explicitly adding a 3D transformation does not significantly improve transformer performance, but removing priors requires compressed representations and can make compression artifacts dominate for small viewpoint changes.Avoiding those artifacts increases computational costs.Source: §5 · p. 8

Supplementary Material

The supplementary material collects additional results and documents the models, data, evaluations, and compression trade-offs used in the study.

Supplementary Material

  • Section A presents additional results, while Section B summarizes model architectures and hyperparameters used in the main paper.The supplementary also includes training and test-data details in Section C.Source: Supplementary Material · p. 9
  • Section D describes uncertainty evaluations via entropy.The passage identifies this as the supplementary’s uncertainty-evaluation section.Source: Supplementary Material · p. 9
  • Section E discusses compression artifacts introduced by using VQGAN as the compression model.Source: Supplementary Material · p. 9

A. Additional Results

The supplementary reports viewpoint-change comparisons, continuous-trajectory samples, and training-variant results, alongside an interactive exploration interface and further qualitative examples.

Interactive Scene Exploration Interface

  • The interface lets users move a camera freely from a single image, previewing views warped with MiDaS depth before autoregressively sampling a novel view.The supplementary videos use test-set trajectories from RealEstate10K and ACID with the implicit-depth model.Source: A. Additional Results · p. 9, Figure 9 · p. 9

Small Viewpoint Changes & Continuous Trajectories

  • The approach outperforms previous approaches averaged across small, medium, and large viewpoint changes, with the gap increasing on harder examples.For very small viewpoint changes, compression distortions dominate its error.Source: A. Additional Results · p. 9, Figure 10 · p. 10
  • The model can generate continuous trajectories, but sampling can flicker; deterministic sampling can alleviate this.The supplementary also notes that previous methods' ability to synthesize small changes does not ensure high-quality moderately long trajectories.Source: A. Additional Results · p. 9, Figure 11 · p. 10

Transformer Variants Over the Course of Training

  • On ACID, the models overfit to the training split early, enabling faster training and multiple runs with different initializations.Figure 12 plots negative log-likelihood on RealEstate and ACID; its solid lines and shaded areas show means and standard deviations over three runs.Source: A. Additional Results · pp. 9–10, Figure 12 · p. 11

Additional Qualitative Results

  • Additional figures provide qualitative comparisons, sample diversity and consistency visualizations, depth-probing results, and entropy visualizations.Source: A. Additional Results · p. 10

B. Architectures & Hyperparameters

The supplement specifies the transformer and VQGAN architectures, training settings, and pretrained models used for comparisons.

Transformer

  • The transformer follows GPT-2, with L blocks applying layer normalization, multi-head self-attention, and a position-wise MLP.The MLP is applied position-wise, unlike global attention.Source: B. Architectures & Hyperparameters · p. 11
  • The model concatenates source-image and camera conditioning with embedded target tokens, then projects transformer outputs to categorical logits over sequence elements.Non-conditioning elements are masked autoregressively.Source: B. Architectures & Hyperparameters · p. 11
  • The transformer uses 32 blocks, 16 attention heads, embedding dimensionality 1024, and 437M parameters, and trains for 500k steps with AdamW.The learning rate warms up for 5k steps before cosine decay [42].Source: B. Architectures & Hyperparameters · p. 11

VQGAN

  • The VQGAN uses a downsampling factor of 24 and a 16,384-entry codebook, producing 13 × 23 latent codes for 208 × 368 RealEstate and ACID images.Dataset-specific VQGANs are fine-tuned for 50,000 steps; the architecture and training procedure are adopted from [17].Source: B. Architectures & Hyperparameters · p. 11

Other models

C. Training and Testing Data

Training pairs use source and target frames with their relative camera transformation; the data preparation resolves the arbitrary scale of SfM translations. Evaluation uses a motion-balanced test subset and a protocol focused on medium and large viewpoint changes.

C. Training and Testing Data

  • Training pairs combine source and target frames with their relative camera transformation, derived from their SfM poses.The relative rotation is computed from the two frame rotations, and the relative translation from their translations.Source: C. Training and Testing Data · p. 12
  • COLMAP points and MiDaS depth estimates align depth to camera-pose scale; depth and translation are then normalized by minimum depth.Source: C. Training and Testing Data · p. 12
  • Evaluation uses 564 RealEstate10K and ACID test pairs, balanced across medium-forward, large-forward, medium-backward, and large-backward motions.The selected split is slated for release with the code.Source: C. Training and Testing Data · p. 12

A Note on Reported Metrics

  • The evaluation removes SynSin's small-change bias by using all test pairs, excluding unchanged views, and adding 50% larger viewpoint changes.The main paper reports results at 100% medium and large viewpoint changes; Table 4 summarizes effects of removing small-change biases.Source: A Note on Reported Metrics · p. 12, Table 4 · p. 12

D. Details on Entropy Evaluation

The entropy evaluation measures the transformer's uncertainty over target codes conditioned on the source image and camera transformation. It visualizes this uncertainty spatially, including regions visible in the source image.

D. Details on Entropy Evaluation

  • Darker, higher-confidence regions in the entropy visualizations correspond to areas visible in the source and indicate geometric source-target relations without 3D bias.The visualization quantifies uncertainty from a single example rather than requiring multiple samples.Source: Figure 13 · p. 13, D. Details on Entropy Evaluation · p. 13
  • For each target-sequence position, the transformer assigns a conditional probability given the source frame and camera transformation, from which entropy is computed.The target, camera, and source are encoded before conditioning the transformer.Source: D. Details on Entropy Evaluation · p. 13
  • The evaluated target sequence has 299 positions, and reshaping and upsampling their entropy values produces image-sized visualizations.The latent map is resized to the input dimensions using bicubic upsampling.Source: D. Details on Entropy Evaluation · p. 13

E. Faithful Reconstructions/Compression Artifacts

The section examines reconstruction quality across compression rates and illustrates outputs and depth probes. VQGAN compression reduces transformer compute but introduces artifacts and a fidelity-cost trade-off.

E. Faithful Reconstructions/Compression Artifacts

  • 4× larger discrete codes improve fidelity but incur approximately 16× training costs; sampling currently takes 4.32 ± 0.04 seconds.The authors attribute the cost increase to the transformer's O(n^2) attention and note that reducing artifacts requires more hardware or training time.Source: E. Faithful… · p. 14
  • At 208 × 368 input resolution, VQGAN f8 yields 26 × 46 codes and f16 yields 13 × 23; Table 5 compares reconstruction metrics across compressions.Figure 14 provides qualitative visualizations of different compression rates.Source: E. Faithful… · p. 14, Table 5 · p. 14, Figure 14 · p. 14
  • Additional samples and comparisons cover RealEstate10K and ACID; sample figures show pixel-wise standard deviation from n = 32 samples.Source: Figure 15 · p. 15, Figure 16 · p. 16, Figure 17 · p. 17
  • Additional linearly probed depth maps compare transformer layers, with visualizations labeling the input, layers #0, #1, #4, and #5, and depth reconstruction.Source: Figure 19 · p. 19, E. Faithful… · p. 19
Loading 2104.07652v2…