PaperlayerView exact revision on arXiv

Source-linked AI summary

Viewpoint Textual Inversion: Discovering Scene Representations and 3D View Control in 2D Diffusion Models

James Burgess, Kuan-Chieh Wang, Serena Yeung-Levy

arXiv:2309.07986v2cs.CVcs.AIcs.LG

TL;DR

Whether 2D-trained diffusion models encode 3D scene structure remains an open question. ViewNeTI learns text-space view tokens from camera parameters and uses them to control generated viewpoints. The experiments find scene-specific and cross-scene view-control manifolds, supporting the presence of 3D scene representations in frozen diffusion models.

  • Problem

    Whether diffusion models trained on 2D images represent 3D scene structure remains an open question.

    Source: §1 · p. 2

  • Method

    ViewNeTI maps camera parameters to a word-embedding view token that conditions diffusion generation through text encoding and cross-attention.

    Source: §1 · p. 2

  • Results

    Experiments find a continuous scene-specific view-control manifold and evidence for a semantically disentangled manifold that transfers viewpoint control across scenes.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors argue that view tokens controlling rendering perspective provide evidence for 3D scene representations in Stable Diffusion.

    Source: §7 · p. 14

    The learned view tokens use DTU camera coordinates covering only part of a sphere; broader camera-parameter spaces remain for future investigation.

    Source: §6 · p. 14

Abstract

from arXiv · show

Text-to-image diffusion models generate impressive and realistic images, but do they learn to represent the 3D world from only 2D supervision? We demonstrate that yes, certain 3D scene representations are encoded in the text embedding space of models like Stable Diffusion. Our approach, Viewpoint Neural Textual Inversion (ViewNeTI), is to discover 3D view tokens; these tokens control the 3D viewpoint - the rendering pose in a scene - of generated images. Specifically, we train a small neural mapper to take continuous camera viewpoint parameters and predict a view token (a word embedding). This token conditions diffusion generation via cross-attention to produce images with the desired camera viewpoint. Using ViewNeTI as an evaluation tool, we report two findings: first, the text latent space has a continuous view-control manifold for particular 3D scenes; second, we find evidence for a generalized view-control manifold for all scenes. We conclude that since the view token controls the 3D `rendering' viewpoint, there is likely a scene representation embedded in frozen 2D diffusion models. Finally, we exploit the 3D scene representations for 3D vision tasks, namely, view-controlled text-to-image generation, and novel view synthesis from a single image, where our approach sets state-of-the-art for LPIPS. Code available at https://github.com/jmhb0/view_neti

1 Introduction

The introduction asks whether 2D-trained diffusion models encode 3D scene structure, citing 3D-consistent car infill as motivation. ViewNeTI probes this through text-space view control.

1 Introduction

  • ViewNeTI maps camera parameters to a word-embedding token that conditions diffusion through text encoding and cross-attention.The mapper is trained with textual inversion on small datasets of posed images.Source: §1 · p. 2
  • With 3–9 posed images and a 140k-parameter mapper, a scene-specific token generalizes to new viewpoints without updating diffusion weights.The authors interpret this as evidence of a pre-existing control mechanism in the text embedding space.Source: §1 · p. 3
  • Single-scene control interpolates but cannot extrapolate, and its token remains entangled with training-scene semantics rather than transferring to new scenes.Source: §1 · p. 3
  • Across fewer than 100 scenes, separate scene tokens and a shared view token provide evidence for a semantically disentangled manifold that transfers viewpoint control to new scenes.The paper further argues that this supports a general view-control manifold acting on 3D scene representations.Source: §1 · pp. 3–4, Figure 4 · p. 3
  • Applications include viewpoint-controlled text-to-image and single-image novel-view synthesis, with photorealistic real-object views and state-of-the-art LPIPS on DTU.Source: §1 · pp. 3–4

2 Related Work

Prior work probes 3D structure in diffusion models, while this paper locates controllable 3D viewpoint in the text embedding space. Related work also establishes textual inversion and motivates view control and sparse-view novel view synthesis as applications.

2.1 3D Representations in 2D Diffusion Models

  • Prior studies probe implicit 3D structure in Stable Diffusion through linear probing of UNet features, whereas this work controls viewpoint in the word embedding space.The paper describes its approach as a “proof by construction” and relates cross-attention view control to Prompt2Prompt’s control of 2D layout.Source: §2.1 · pp. 4–5

2.2 Textual Inversion of Diffusion Models

  • Textual inversion learns word embeddings for new concepts from a few images, and NeTI predicts embeddings conditioned on the UNet layer and noise step.ViewNeTI extends NeTI’s mapper architecture.Source: §2.2 · p. 5
  • This work applies textual inversion to 3D viewpoint learning and novel view synthesis, unlike concurrent work on continuous 3D words that fine-tunes model weights and omits NVS.The paper also notes that optimizing multiple TI tokens in one prompt had previously been used by Break-a-Scene [3].Source: §2.2 · p. 5

2.3 3D Control Applications

  • Viewpoint control is a downstream image-generation application, and personalization-based weight fine-tuning could improve it but falls outside this work’s focus.The paper notes that TI has low storage cost and can be composed with fine-tuned model checkpoints [26].Source: §2.3 · p. 5
  • Prior sparse-view NVS methods use explicit 3D representations, regularizers, modified training, or pretrained image features; some single-image approaches use diffusion priors.The paper positions ViewNeTI as using no explicit 3D representation, geometric priors, regularizers, or large 3D training datasets.Source: §2.3 · pp. 5–6

3 Background

Latent diffusion generates images by modeling compressed image latents conditioned on text. Textual inversion adapts this conditioning vocabulary with learned embeddings while keeping diffusion weights frozen.

3.1 Text-to-image Latent Diffusion Models

  • Latent diffusion compresses images into VAE latents and trains a conditional diffusion model to generate that latent distribution from text prompts.The model adds noise over steps and trains a network to predict it given the noising step and text.Source: §3.1 · p. 6
  • At inference, the reverse diffusion process samples a latent that is decoded into an image, while CLIP text embeddings condition UNet layers through cross-attention.The text conditioning contains one vector per token.Source: §3.1 · p. 6

3.2 Textual Inversion

  • Textual inversion learns a new word embedding for a concept from a small image-and-caption dataset by optimizing the LDM loss while freezing diffusion weights.Captions use the pseudo-word representing the concept.Source: §3.2 · pp. 6–7
  • NeTI predicts embeddings conditioned on each UNet cross-attention layer and noise step, optimizing the mapper network rather than a single embedding.ViewNeTI extends the NeTI architecture.Source: §3.2 · p. 7

4 Method

ViewNeTI learns a camera-conditioned word embedding that controls generated viewpoint while leaving the diffusion model frozen. It trains either on one scene or jointly across scenes with separate scene tokens and a shared view mapper.

4 Method

  • ViewNeTI learns a 3D view token to control generated viewpoint and evaluates latent 3D representations in Stable Diffusion.Its two training modes are single-scene optimization and multi-scene pretraining.Source: §4 · p. 7

4.1 Single-scene Optimization and Viewpoint Generalization

  • For single-scene optimization, a mapper predicts the view-token embedding from camera pose, diffusion timestep, and UNet layer, using Fourier-feature encoded inputs.Camera pose may use extrinsics and intrinsics; experiments use normalized camera-to-world projection matrices, and the method is also verified with spherical coordinates.Source: §4.1 · pp. 7–8
  • The single-scene mapper is trained on fewer than 10 posed images by replacing each view token’s prompt embedding and optimizing the mapper with the diffusion loss.The output embedding has the text encoder’s dimension and the mapper has 140,000 parameters; image augmentations help textual-inversion accuracy on small datasets.Source: §4.1 · pp. 8–9
  • At inference, new continuous camera parameters produce a view token that conditions diffusion generation for the requested viewpoint.The method uses the same prompt and runs DPMSolver for 50 steps.Source: §4.1 · p. 9

4.2 Multi-Scene Optimization and Scene Generalization

  • Multi-scene pretraining jointly optimizes one shared view mapper and scene-specific mappers, with separate scene tokens intended to capture semantics apart from viewpoint.The training data uses multiple scenes with known camera poses and dense view-space coverage.Source: §4.2 · pp. 9–10
  • The pretrained view token controls viewpoint in text-to-image prompts for new objects and supports NVS by fitting a new scene mapper to the target scene’s images.The reported NVS experiments include single-image input.Source: §4.2 · p. 10

5 Results

The experiments find continuous viewpoint control for a single scene and evidence of a semantically disentangled control manifold across scenes. ViewNeTI also performs strongly on single-image novel view synthesis, while the scope of generalization remains qualified.

5.1 Single-Scene Optimization and Viewpoint Generalization

  • With six posed images, single-scene ViewNeTI predicts interpolated views correctly, supporting a continuous view-control manifold in Stable Diffusion’s text space.Predictions do not extrapolate, and this manifold is specific to the training scene.Source: §5.1 · pp. 10–11, Figure 6 · p. 11

5.2 Multi-scene Optimization and Scene Generalization

  • View-controlled generation preserves object semantics across viewpoints for objects outside the DTU training distribution, though generated backgrounds show some training-style entanglement.The conditioning adds negligible generation runtime.Source: §5.2 · p. 11
  • Single-image NVS on different object classes supports cross-scene viewpoint generalization, but test scenes share the training scenes’ style.The authors describe the view token as only somewhat general and not tested for complete generality.Source: §5.2 · p. 12, Figure 7 · p. 12
  • Across multiple scenes, the view token changes rendering viewpoint, while scene tokens capture the semantics of their respective scenes.This provides further evidence for an implicit 3D scene representation in Stable Diffusion.Source: §5 · p. 10, §5.2 · p. 12
  • Against NVS baselines, ViewNeTI produces photorealistic images with plausible semantics and little blurriness, but can hallucinate object details.The compared NeRF methods show blur, artifacts, and semantic-detail errors, while Zero-1-to-3 does not make reasonable predictions on DTU.Source: §5.2 · p. 12
  • ViewNeTI achieves state-of-the-art LPIPS and near-state-of-the-art SSIM on single-image NVS metrics on DTU.The approach uses no large multi-view datasets, explicit 3D representation, or task-specific 3D regularizations.Source: §5.2 · p. 13, Table 1 · p. 13

5.3 Ablations

  • Ablations identify positional-encoding frequency and text-embedding norm scaling as key design choices, while image augmentation helps avoid degenerate solutions.The paper reports qualitative ablation results in its appendix.Source: §5.3 · p. 13

6 Limitations and Future Work

The reported view-control manifolds are qualitative and have limits in generalization, camera coverage, and novel-view synthesis. The authors identify broader training and camera-parameter coverage, and improved reconstruction methods, as future directions.

6 Limitations and Future Work

  • The single-scene view-control result is qualitative, and the paper cannot quantify representation quality across diffusion models.View-synthesis reconstruction accuracy may not provide a convincing comparison if ViewNeTI’s optimal design parameters differ by architecture.Source: §6 · pp. 13–14
  • The shared view token is not fully general: generations retain pretraining-scene style entanglement, while view synthesis includes errors and hallucinations.The authors suggest larger-corpus pretraining or staged training that learns the object token before the view token, similar to [10].Source: §6 · p. 14
  • View-control tokens were learned on DTU camera coordinates covering only part of a sphere; broader camera-parameter spaces remain future work.The authors give buildings and outdoor scenes as examples for further investigation.Source: §6 · p. 14
  • Novel-view synthesis makes errors in object details and precise pose, and its PSNR is below SOTA.The authors suggest better disentanglement, broader camera coverage, reconstruction advances, or full-model finetuning as possible ways to improve applications.Source: §6 · p. 14

7 Conclusions

The paper presents evidence that Stable Diffusion’s text space can control rendering viewpoint and may encode a 3D scene representation. It connects this finding to 3D applications and questions about what 2D-only models learn.

7 Conclusions

  • A 3D view-control token in Stable Diffusion controls rendering perspective, suggesting an embedded 3D scene representation.The paper calls this a constructive proof and the first evidence that text-space cross-attention can do 3D control, similarly to 2D layout control [22].Source: §7 · p. 14
  • The authors note that similar questions arise for text-to-video models: whether 2D-only models can learn 3D geometry and physics.They suggest approaches like theirs can help investigate these questions [23].Source: §7 · p. 14
  • The authors propose that methods like ViewNeTI could help improve 3D-method data efficiency by using 3D understanding learned by 2D-only models.Their single-image novel-view-synthesis application used 3D pre-training on fewer than 100 scenes, compared with over 100,000 for Zero-1-to-3.Source: §7 · p. 14

Supplementary overview

The supplement collects additional evidence, ablations, implementation and evaluation details, and novel-view-synthesis results. It also documents the Stable Diffusion image-outfilling test used to examine 3D reasoning.

Supplementary overview

  • The overview identifies supplementary figures extending single-scene results and showing three-view and single-view novel-view-synthesis predictions for every DTU test scene.It highlights Figures 9–10 for additional scenes and views, and Figures 11–12 for the synthesis predictions.Source: Supplementary overview · p. 21
  • The supplement covers outfilling tests, qualitative single-scene ablations, augmentation, optimization results, and DTU camera-position visualizations.These topics are listed in supplementary sections A–F.Source: Supplementary overview · p. 21
  • Other supplementary topics include mapper validation, spherical camera parameterization, a Zero-1-to-3 comparison, and implementation details.The overview lists these in sections G–L.Source: Supplementary overview · p. 21

A Evidence for 3D capabilities in diffusion models with image outfilling

B Ablations

Qualitative ablations show that camera-parameter encoding and training choices affect view coverage and the learning of scene semantics.

B Ablations

  • Too-low-frequency camera-parameter encoding makes it difficult to cover all camera views.The supplement presents qualitative ablations for single-scene ViewNeTI training.Source: B Ablations · p. 21
  • Without augmentations or with too-high-frequency encoding, learning scene semantics is worse.Source: B Ablations · p. 22
  • Omitting norm scaling also worsens the learning of scene semantics.Source: B Ablations · p. 22

C Data Augmentation

Image augmentations help ViewNeTI learn robust scene tokens from small datasets, while prompt augmentations vary the textual templates used during training.

C Data Augmentation

  • Image augmentations support accurate textual inversion on small training sets, and horizontal flips are omitted because they would prevent learning the view-mapper.The authors also omit grayscaling to avoid gray generations and use higher-probability Gaussian blur.Source: C Data Augmentation · p. 22
  • For three-view novel view synthesis, the authors reduce RandomResizedCrop’s size range because strong augmentations were unnecessary.The size parameter is changed to (0.950, 1.05).Source: C Data Augmentation · p. 22
  • Prompt augmentation uses varied text templates around the view and scene tokens, following standard textual-inversion practice.Examples include “SRi. a rendition of the Ssj.” and “SRi. a photo of a small Ssj.”Source: C Data Augmentation · p. 22

D Single-Scene Optimization Results

Single-scene results support continuous viewpoint control for novel test views, while the camera visualizations clarify which DTU views count as interpolation or extrapolation.

D Single-Scene Optimization Results

  • Test views unseen during training support a continuous 3D view-control manifold in word space.With nine training views, the test views are all interpolations; with six views, extrapolated predictions are bad.Source: D Single-Scene Optimization Results · p. 22

E Visualizing the Pretraining Camera Positions

  • Pretraining cameras lie on a sphere around the object, spanning about 160° in azimuth and 75° in polar angle.The cameras approximately point toward the origin, but the object need not be centered in every image.Source: E Visualizing the Pretraining Camera… · p. 23

F Visualizing the Camera Positions for DTU Sparse Dataset

  • DTU extrapolation means test views outside the convex hull of training views in spherical angle space, especially beyond the training polar-angle range.The six-view split spans most azimuth angles but only about half the polar-angle range.Source: F Visualizing the Camera Positions for… · p. 23

G Pretraining Results

Multi-scene pretraining shares a viewpoint mapper while assigning scene-specific tokens, and the reported applications test this learned control across parameterizations and datasets.

G Pretraining Results

  • Reconstructions after 100k training steps support the shared view-mapper’s learning of relative camera viewpoints, while scene-only generations approximately capture scene semantics with varying success.For some scenes, disentanglement worsens with training time; one example becomes only a circular pattern by 100k steps.Source: G Pretraining Results · pp. 23–24

H Validation of Spherical Coordinate Parameterization

  • ViewNeTI is agnostic to camera parameterization; spherical coordinates use fixed-radius cameras with variable polar and azimuth angles for single-scene ShapeNet-car view synthesis.The middle columns show predictions interpolating between polar and azimuth angles of the edge training views.Source: H Validation of Spherical Coordinate… · p. 24, Figure 16 · p. 33

J ViewNeTI Implementation Details

  • Experiments use frozen Stable Diffusion 2.1 weights and 140,000-parameter view- and scene-mappers; Stable Diffusion 1 was not tested.Camera, timestep, and UNet-layer inputs receive 64-dimensional random Fourier feature encodings.Source: J ViewNeTI Implementation Details · p. 25

K DTU Evaluation Details

  • DTU test scenes are separated from pretraining scenes to create a scene-semantic domain shift, and sparse-view splits select the first views from a standard ordered list.The test set comprises 15 scenes; the 6-, 3-, and 1-view splits use the first six, three, or one listed views.Source: K DTU Evaluation Details · p. 26

L Output Bypass Architecture for Scene Tokens

  • The scene-token output bypass adds a normalized perturbation scaled by α = 0.2 after text encoding, intended to capture fine details without changing coarse semantics.The bypass vector is an additional output of the scene-mapper.Source: L Output Bypass Architecture for Scene… · pp. 26–27

M Pretraining ViewNeTI on Objaverse with inference on DTU

  • 0.37 vs 0.40 LPIPS and 12.2 vs 12.0 PSNR favor ViewNeTI over Zero-1-to-3 pretrained on 800,000 scenes in Objaverse-to-DTU synthesis.With 50 Objaverse scenes, ViewNeTI learns camera control while Zero-1-to-3 trained on the same 50 scenes fails completely.Source: M Pretraining ViewNeTI on Objaverse… · p. 27

I Qualitative Comparison to Zero-1-to-3

Loading 2309.07986v2…