PaperlayerView exact revision on arXiv

Source-linked AI summary

SPAD : Spatially Aware Multiview Diffusers

Yash Kant, Ziyi Wu, Michael Vasilkovsky, Guocheng Qian, Jian Ren, Riza Alp Guler, Bernard Ghanem, Sergey Tulyakov, Igor Gilitschenski, Aliaksandr Siarohin

arXiv:2402.05235v1cs.CV

TL;DR

Single-view priors and unconstrained multi-view attention can leave geometric inconsistencies or copy content across views. SPAD adds epipolar constraints and Plücker ray encodings to pretrained diffusion models for text- or image-conditioned view generation. Its qualitative comparisons report improved text alignment and detail, and better structural preservation and 3D consistency than the cited baselines.

  • Problem

    Single-view diffusion priors ignore geometric relationships across views, while unconstrained multi-view attention can copy content between nearby views.

    Source: §1 · p. 2

  • Method

    SPAD fine-tunes a pretrained diffusion model for text- or image-conditioned multi-view generation, using epipolar-constrained attention and Plücker ray embeddings.

    Source: §1 · pp. 2–3

  • Results

    SPAD produces higher-quality details and better text alignment than MVDream, and preserves object structure and perceptual details with better 3D consistency than Zero123.

    Source: §B.1 · p. 15

  • Takeaways & Limitations

    SPAD generates high-quality, 3D-consistent images in both text-conditioned multi-view generation and image-conditioned novel-view synthesis.

    Source: §1 · p. 3

    The authors identify scope to improve performance, including with a larger Stable Diffusion model, and note image-conditioning compression as a concern.

    Source: §5 · p. 9

Abstract

from arXiv · show

We present SPAD, a novel approach for creating consistent multi-view images from text prompts or single images. To enable multi-view generation, we repurpose a pretrained 2D diffusion model by extending its self-attention layers with cross-view interactions, and fine-tune it on a high quality subset of Objaverse. We find that a naive extension of the self-attention proposed in prior work (e.g. MVDream) leads to content copying between views. Therefore, we explicitly constrain the cross-view attention based on epipolar geometry. To further enhance 3D consistency, we utilize Plucker coordinates derived from camera rays and inject them as positional encoding. This enables SPAD to reason over spatial proximity in 3D well. In contrast to recent works that can only generate views at fixed azimuth and elevation, SPAD offers full camera control and achieves state-of-the-art results in novel view synthesis on unseen objects from the Objaverse and Google Scanned Objects datasets. Finally, we demonstrate that text-to-3D generation using SPAD prevents the multi-face Janus issue. See more details at our webpage: https://yashkant.github.io/spad

A DSLR photo of a pair of tan cowboy boots, studio lighting, product photography

The supplied material pairs two object descriptions with a caption about SPAD generating multiple consistent views from text.

A DSLR photo of a pair of tan cowboy boots, studio lighting, product photography

  • The supplied object descriptions name a cute steampunk elephant and a knight's armored metal helmet with gold trim and holes.Source: A DSLR photo of a pair of tan cowboy… · p. 1
  • SPAD is described as generating multiple 3D-consistent images of one prompted object, from everyday objects to complex machines.The caption says the model can generate arbitrary camera viewpoints despite training on four views.Source: Figure 1 · p. 1
  • The illustrated example uses eight views sampled uniformly at a fixed elevation.Source: Figure 1 · p. 1

1. Introduction

The introduction motivates multi-view generation as a way to address geometric inconsistency in 3D synthesis. SPAD extends pretrained 2D diffusion models with geometry-aware cross-view attention and ray-based positional encodings, and evaluates text- and image-conditioned generation.

1. Introduction

  • Single-view diffusion priors ignore geometric relationships across views, and text-to-3D methods can produce inconsistencies such as the multi-faced Janus problem [59, 90].Source: §1 · p. 2
  • Zero-1-to-3 concatenates the input view without geometric priors, while iNVS can produce high-quality views with accurate depth maps but depends on their precision [45] [39].Source: §1 · p. 2
  • Multi-view self-attention over concatenated features can still lack precise camera control and copy content between nearby, overlapping views [73].Source: §1 · p. 2
  • Epipolar Attention restricts cross-view attention to epipolar lines, improving camera control and distinct views at nearby viewpoints, though epipolar lines alone leave ray direction ambiguous [79] [39].Source: §1 · p. 2
  • Plücker coordinates encode camera rays as positional embeddings, biasing attention against opposite object sides and toward nearby rays [89].Source: §1 · p. 3
  • SPAD supports text-conditioned multi-view generation and image-conditioned novel-view synthesis, evaluated on GSO and an unseen Objaverse subset [22] [63].The paper also describes text-to-3D generation using a feed-forward multi-view-to-3D triplane generator or multi-view Score Distillation Sampling [73].Source: §1 · p. 3

2. Related Works

Related work spans 3D generative modeling and diffusion-based novel-view synthesis, with prior approaches facing limits in generalization, camera control, or dependence on depth. SPAD adapts pretrained 2D diffusion models with epipolar cross-view interactions and Plücker positional encodings.

2. Related Works

  • Earlier 3D VAEs and GANs were limited by small datasets, realism, diversity, generalizability, or training stability, while some 3D diffusion models trained from scratch on specific objects had limited generalization [41] [34] [78].Source: §2 · p. 3
  • Diffusion-based novel-view methods include SparseFusion's epipolar feature fusion and Zero-1-to-3's fine-tuning on Objaverse multi-view images [79] [64] [45] [18].Source: §2 · p. 3
  • MVDream jointly denoises four views but uses fixed camera positions spaced 90 degrees apart, limiting view count and requiring synthetic data rendered at those positions [73] [18].Source: §2 · p. 3
  • iNVS relies on accurate depth maps and can suffer flipped predictions from epipolar-line depth ambiguity, while SyncDreamer requires explicit pixel-depth understanding for reprojection [39] [46].Source: §2 · p. 3
  • SPAD enriches pretrained text-to-image self-attention with epipolar cross-view interactions and uses Plücker coordinates as positional encodings for camera conditioning and distinguishing object sides.Source: §2 · p. 3
  • The pipeline fine-tunes a pretrained text-to-image model on multi-view 3D renders, jointly denoises views, and applies epipolar constraints and Plücker embeddings in attention.The illustrated two-view setup is described as extensible to more views.Source: Figure 2 · p. 4
  • Text-to-3D distillation methods still face visual-quality and view-consistency issues, including Janus and content drifting; SPAD's multi-view images can mitigate these issues in multi-view distillation [73].Source: §2 · p. 4

3. Method

SPAD adapts a pretrained 2D diffusion model for generating consistent object views from text or images, conditioned on relative camera poses. It combines cross-view attention constrained by epipolar geometry with Plücker ray embeddings to address content copying and geometric ambiguity.

3.2. Multi-View Diffusion Models

  • SPAD trains a multi-view diffusion model to generate consistent novel views from text or images and relative camera poses, extending the approach to arbitrary view counts.It fine-tunes a pretrained 2D diffusion model on multi-view rendered images.Source: §3 · p. 4, §3.2 · p. 4

3.1. Preliminary: Text-to-Image Diffusion Models

  • Diffusion models learn to denoise Gaussian-noised data, with a U-Net denoiser predicting the added noise at each timestep.The denoiser includes residual blocks and self-attention layers; its feature maps are flattened for attention.Source: §3.1 · p. 4

3.2.1 Multi-View Self-attention

  • Unconstrained attention over concatenated view features permits cross-view interaction but can copy content between views when camera poses are close.The authors report that similar images may be generated despite a small pose difference, ignoring underlying 3D geometry.Source: §3.2.1 · p. 5

3.2.2 Multi-View Epipolar Attention

  • Epipolar Attention restricts cross-view access to epipolar-line points while retaining access to all points in the source view.The lines are dilated with a 3×3 filter for robustness; relying on them alone can still cause flipped predictions, especially with large viewpoint changes and no precise surface depth.Source: §3.2.2 · p. 5, Figure 3 · p. 5

3.2.3 Pl¨ucker Ray Embedding

  • Plücker ray embeddings encode each ray as (o × d, d), giving equivalent embeddings to rays with camera origins shifted along the same ray.The embeddings encourage attention between spatially nearby rays and reduce attention between opposite-camera rays through the same 3D location, addressing flipped predictions.Source: §3.2.3 · pp. 5–6

4. Experiments

The experiments evaluate text-conditioned multi-view generation, image-conditioned novel-view synthesis, qualitative consistency, and text-to-3D generation. SPAD shows stronger novel-view results across Objaverse and GSO, while its attention design addresses copying and flipping and supports two text-to-3D pipelines.

4.1. Experimental Setup

  • Experiments test text-conditioned generation and image-conditioned synthesis using Objaverse and Google Scanned Objects, with image-conditioned testing on 1,000 unseen Objaverse objects.Text-conditioned image quality is measured with CLIP-score, Inception Score, and FID; novel-view synthesis uses PSNR, SSIM, and LPIPS.Source: §4 · p. 6, §4.1 · p. 6

4.2. Text-conditioned Multi-View Generation

  • SPAD matches or outperforms same-base Stable Diffusion v1.5 methods on 2D image-quality metrics without compromising text alignment or image quality.Qualitative examples also show diverse subjects, from daily objects to complex machines.Source: §4.2 · p. 7, Table 1 · p. 7

4.3. Image-conditioned Novel View Synthesis

  • SPAD outperforms all baselines on LPIPS across Objaverse and GSO while matching SyncDreamer on SSIM; adding Epipolar Attention and Plücker embeddings progressively improves scores.The tables report PSNR, SSIM, and LPIPS for 1,000 unseen Objaverse objects and GSO objects.Source: §4.3 · p. 7, Table 2 · p. 7, Table 3 · p. 7

4.4. Qualitative Analysis

  • Epipolar Attention reduces content copying by restricting cross-view interactions to spatially related pixels, while Plücker embeddings help prevent flipped views by favoring spatially closer views.The qualitative comparison prompts models trained on two views to generate four views at 90-degree intervals.Source: §4.4 · p. 8, Figure 5 · p. 8

4.5. Text-to-3D Generation

  • With multi-view SDS, SPAD reconstructs consistent NeRF geometry without the Janus problem while maintaining visual quality.The pipeline uses four SPAD views for text-to-3D generation.Source: §4.5 · p. 8
  • The triplane-generator pipeline uses four orthogonal SPAD views to produce a NeRF in one feed-forward pass, taking roughly 10 seconds per asset—over two orders of magnitude faster than SDS optimization.The authors describe SPAD as a faithful base model for this generation approach.Source: §4.5 · p. 8, Figure 6 · p. 9

5. Conclusion

The conclusion presents SPAD as a multi-view generation framework that modifies pretrained diffusion attention with epipolar constraints and Plücker positional encodings. It reports state-of-the-art image-conditioned novel-view synthesis results and identifies several directions for improvement.

5. Conclusion

  • SPAD transforms pretrained text-to-image diffusion self-attention into Epipolar Attention and adds Plücker positional encodings to improve camera control and prevent flipped predictions.The paper reports state-of-the-art results for image-conditioned novel-view synthesis.Source: §5 · p. 9
  • The authors identify larger Stable Diffusion models, depth-estimator-assisted correspondences, dynamic 4D assets, multi-object scenes, and video data or generators as future work.They note that lossy compression of image conditioning remains a concern that a larger model such as SDXL could address.Source: §5 · p. 9

A. More Implementation Details

The implementation details note a sampling initialization and point to a table of SPAD hyperparameter choices.

A. More Implementation Details

B.1. Qualitative Results and Baseline Comparisons

Qualitative comparisons report text alignment, structural and perceptual detail, and behavior at closely spaced viewpoints.

B.1. Qualitative Results and Baseline Comparisons

  • SPAD produces continuous views at 10-degree azimuth increments without content copying, while preserving structural and perceptual details in image-conditioned novel-view synthesis.The qualitative comparisons also report higher image detail and better text alignment than MVDream.Source: §B.1 · p. 15

B.2. User Study comparing SPAD with MVDream

In a user study comparing four-view generations, participants preferred SPAD over MVDream when judging consistency, visual quality, and text alignment.

B.2. User Study comparing SPAD with MVDream

  • Participants preferred SPAD over MVDream by 59% to 41% in a study evaluating consistency, visual quality, and text alignment.Participants compared sets of four views showing front, left, right, and back views of an object.Source: §B.2 · p. 15

B.3. Training with Stable Diffusion v2.1 Weight

The appendix reports stronger text alignment with SD v2.1 initialization, examines classifier-free guidance, and describes joint inference for generating more views than the model was trained on.

B.3. Training with Stable Diffusion v2.1 Weight

  • 30.39±3.30 CLIP-score for SPAD initialized from SD v2.1 exceeds both SPAD with SD v1.5 (29.87±3.33) and MVDream with SD v2.1 (30.22±3.83).The authors also report better text alignment, especially for longer and more complicated prompts.Source: §B.3 · p. 15

B.4. Classifier-free Guidance

  • Classifier-free guidance beyond text conditioning does not provide additional benefits and leads to over-saturated generations.The guidance setup extends to text, camera, Epipolar Attention, and Plücker Embedding conditions.Source: §B.4 · pp. 15–16

B.5. Joint Multi-View Inference

  • Joint multi-view inference enables generating an infinite number of views with a model trained on fewer views, avoiding quadratically increasing training costs from adding views.The paper motivates this because high-quality NeRF assets usually require more than ten views.Source: §B.5 · p. 16
  • The procedure initializes noise maps for all views, then iteratively denoises pairs sampled without replacement and uniformly across combinations.The model is trained on two views at the same noise level.Source: §B.5 · p. 16
  • Joint inference trades off 3D consistency because only two views can communicate across views at any given denoising timestep.Source: §B.5 · p. 16

B.6. Fr´echet Inception Distance (FID) Results

SPAD improves FID over Vanilla MV-DM, while the authors caution that its FID cannot be directly compared with MVDream because their pose distributions differ.

B.6. Fréchet Inception Distance (FID) Results

  • 52.77 FID for SPAD is better than Vanilla MV-DM's 55.25, indicating improved 2D generation quality.Source: §B.6 · p. 16
  • SPAD's FID cannot be directly compared with MVDream's reported 32.06 because SPAD's random-view pose distribution differs from MVDream's orthogonal 90-degree views.Source: §B.6 · p. 16
Loading 2402.05235v1…