PaperlayerView exact revision on arXiv

Source-linked AI summary

MVDream: Multi-view Diffusion for 3D Generation

Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, Xiao Yang

arXiv:2308.16512v4cs.CV

TL;DR

Existing 2D-lifting methods lack multiview knowledge and 3D awareness, causing inconsistent 3D generation. MVDream learns from 2D and 3D data, then serves as a multiview 3D prior for SDS-based generation. It improves stability and quality while retaining generalizability and supporting personalized 3D generation.

  • Problem

    2D-lifting methods lack comprehensive multiview knowledge and 3D awareness, leading to regenerated content and content drift across views.

  • Method

    MVDream jointly trains a multi-view diffusion model on 3D renderings and 2D image-text data, using cross-view 3D attention and camera embeddings, then applies it as a 3D prior through SDS.

  • Results

    MVDream provides better stability and quality than current open-sourced 2D-lifting methods while maintaining the base model's generalizability and generating multiview-consistent images.

  • Takeaways & Limitations

    The model supports robust 3D generation and few-shot personalized 3D generation without depending on a specific 3D representation.

  • Takeaways & Limitations

    The model is limited to 256×256 images, its generalizability is constrained by the base model, and its generated styles are affected by the rendered dataset.

Abstract

from arXiv · show

We introduce MVDream, a diffusion model that is able to generate consistent multi-view images from a given text prompt. Learning from both 2D and 3D data, a multi-view diffusion model can achieve the generalizability of 2D diffusion models and the consistency of 3D renderings. We demonstrate that such a multi-view diffusion model is implicitly a generalizable 3D prior agnostic to 3D representations. It can be applied to 3D generation via Score Distillation Sampling, significantly enhancing the consistency and stability of existing 2D-lifting methods. It can also learn new concepts from a few 2D examples, akin to DreamBooth, but for 3D generation.

1 INTRODUCTION

3D creation is labor-intensive, while existing 2D-lifting methods can produce inconsistent views. MVDream combines 2D and multi-view learning to improve consistency while retaining broad generation capabilities.

  • 3D asset creation requires trained designers to work for hours or days on a single asset.
  • 2D diffusion models enable text-specified artistic assets, but 2D-lifting methods lack comprehensive multiview knowledge and 3D awareness.These limitations lead to regenerated content and drift across views.
  • MVDream jointly trains on multi-view renderings and 2D image-text pairs to combine view consistency with the generalizability of 2D diffusion models.The model generates a consistent set of multi-view images and acts as a 3D prior agnostic to the chosen representation.
  • MVDream produces 3D NeRF models without the multi-view consistency issue while matching or surpassing the diversity of other state-of-the-art methods.The model also supports few-shot identity learning from 2D images through multi-view DreamBooth-style fine-tuning.

2 RELATED WORK AND BACKGROUND

Prior work explores direct novel-view synthesis and 2D diffusion priors for 3D generation, but generalization and multiview consistency remain challenges. MVDream addresses these limitations by modeling cross-view dependencies while retaining a 2D diffusion backbone.

  • Novel-view synthesis methods directly generate views without reconstruction, but their generalization is bounded by training data and their outputs can remain geometrically inconsistent.The cited studies lack established evidence of generalizing to diverse image inputs, and some produce blurry 3D models.
  • The model adapts text-to-image UNets with 3D self-attention connecting views, while preserving the remaining network as a 2D model operating within each image.This design was found to maintain consistency even when the view gap is large.
  • 3D generative models have limited generalizability, while 2D-lifting methods use diffusion priors to supervise 3D representations through SDS.These approaches can generate arbitrary photorealistic objects without 3D training data but often suffer from multiview inconsistency and require per-model tuning.
  • MVDream improves robustness by producing satisfactory results with one parameter set instead of individually tuning prompts and hyperparameters for each 3D model.

3 METHODOLOGY

MVDream adapts 2D diffusion into a multi-view model that generates camera-controlled, mutually consistent images while retaining 2D generalizability. It then uses this model as a 3D prior for reconstruction, SDS-based generation, and DreamBooth-style personalization.

  • 3.1 Multi-view diffusion model: MVDream trains on rendered multi-view images and 2D image-text pairs to combine multi-view consistency with 2D-model generalizability.The model generates multiple views of the same scene using text and camera parameters, then serves as a multi-view prior for 3D generation.
  • 3.1.1 Multi-view consistent generation: 3D attention connects tokens across views, avoiding the content drift observed with simple temporal attention, including when viewpoint gaps are large.The inflated attention reshapes B × F × H × W × C features into B × FHW × C so all views participate in self-attention while inheriting 2D attention weights.
  • 3.1.2 Camera pose control: Camera parameters are embedded with a 2-layer MLP to provide distinguishable view differences and camera-pose control.The method compares relative position encoding, rotary embeddings, and absolute camera parameters before selecting camera-parameter embeddings.
  • 3.1.3 Training loss function: Joint training includes a 30% image-mode schedule that disables 3D attention and camera embeddings, supporting generalization from a larger-scale text-to-image dataset.The model is fine-tuned from Stable Diffusion v2.1 at 256×256 resolution while retaining its optimizer and ε-prediction settings.
  • 3.2 Text-to-3D generation: For 3D generation, MVDream replaces Stable Diffusion in existing SDS pipelines, changes camera sampling, and feeds camera parameters as inputs.The experiments focus on SDS rather than few-shot reconstruction; original prompts are used instead of direction-annotated prompts.
  • 3.2 Text-to-3D generation: The SDS pipeline uses an x0-reconstruction loss, annealed timestep bounds, and fixed negative prompts to improve generation quality and avoid low-quality dataset styles.The reconstruction loss performs similarly to original SDS and mitigates color saturation after CFG rescaling; simple reconstruction loss is sufficient for generation from scratch.
  • 3.3 Multi-view DreamBooth for 3D generation: DreamBooth fine-tunes the pretrained multi-view model on identity images with image and parameter-preservation losses while maintaining multi-view ability after fine-tuning.The preservation term keeps parameters near the original model, with balancing parameter λ set to 1.

4 EXPERIMENTS

MVDream is evaluated across multi-view image synthesis, NeRF generation, ablations, and personalization-related experiments. The results support strong image quality and consistency, with 78% average user preference over competing models.

  • Evaluation setup: The evaluation covers multi-view image generation, NeRF generation with multi-view score distillation, and related stability and quality experiments.The experiments use fine-tuned Stable Diffusion models on Objaverse and LAION, with multi-view image quality and consistency as one evaluation target.
  • Attention ablation: Inflated 2D self-attention achieves the best consistency among the tested attention modules without losing generation quality.Training attention parameters from scratch requires more data and time, while the inflated 2D strategy retains the existing architecture more effectively.
  • Multi-view image generation: MVDream generates unseen, potentially counterfactual prompts in styles different from its training prompts.The qualitative examples use the appended text suffix “, 3d asset” for generation.
  • User study: 78% of users prefer MVDream over other models on average across 40 prompts.The study collected 914 feedback responses from 38 users, who selected among five rendered videos and their corresponding text inputs.
  • View-count ablation: Increasing training views reduces the Janus problem, with the 4-view model nearly eliminating it compared with 1-view and 2-view models.All compared models use camera embeddings; the 1-view model still suffers severely, while the 2-view model reduces but does not eliminate the issue.
  • Score-distillation ablation: Time-step annealing improves shape completeness, negative prompts improve visual style, and CFG rescaling makes texture colors more natural.These effects are reported in the ablation of the techniques used for multi-view score distillation.
  • Multi-view DreamBooth: MVDream produces higher-quality DreamBooth3D results with better object details, including curly hair and fur texture on dogs.The authors attribute this to higher geometry consistency from the multi-view DreamBooth diffusion models during NeRF optimization.

5 DISCUSSION

The discussion reports extensions to non-orthogonal and larger view sets, comparisons with image-to-3D pipelines, and the broader role of MVDream as a multi-view 3D prior. It also identifies resolution, base-model, and rendering-style limitations.

  • Generalizability to more views: A 3D self-attention network can generate random multi-view images when trained on such data, although quality is worse than for orthogonal views.The main model is trained on four orthogonal views with the same elevation angle.
  • Discussion and conclusion: MVDream generates text-conditioned multi-view images, serves as a 3D prior for SDS, and supports few-shot personalized 3D generation.The model is fine-tuned on a mixture of 3D-rendered data and large-scale text-to-image data to retain generalizability while improving multi-view consistency.
  • Limitations: The current model is limited to 256×256 images, depends on the base model's generalizability, and inherits styles from its rendered training data.The authors suggest larger datasets, larger base models, and more diverse realistic rendering as possible remedies; the latter may be costly.

6 ETHICS STATEMENT

The ethics statement describes misuse, inherited biases, synthetic-content labeling, potential displacement of creative workers, and possible accessibility benefits.

  • Risks: Third-party fine-tuning could apply the model to violent or sexual content, while its Stable Diffusion foundation may inherit biases and unwanted-generation limitations.The authors recommend careful examination and synthetic labeling of images or models produced with the approach.
  • Workforce and access: The authors note a potential for generative automation to displace creative workers while also improving accessibility and growth in the creative industry.The statement presents both risks and possible benefits without resolving the trade-off.

7 REPRODUCIBILITY STATEMENT

The reproducibility statement says MVDream uses public models and datasets, provides implementation details, and plans to release code and checkpoints.

  • Resources: MVDream is built from publicly available Stable Diffusion, LAION, and Objaverse resources.The authors state that the implementation details should make reproducing the algorithm straightforward.
  • Release: The authors plan to release the code and model checkpoints publicly after paper submission.

A.1 2D DATA

Adding 2D data to multi-view diffusion improves generalizability, image quality, and text-image correspondence while training uses a diffusion loss with camera conditioning and noisy-image reconstruction.

  • 2D data: Adding 2D data clearly improves model generalizability, image quality, and text-image correspondence.The comparison in Fig. (13) evaluates training with and without 2D data.
  • Diffusion training: The diffusion model predicts noise for a noisy image conditioned on text, camera parameters, and the timestep.The training formulation uses xt as the noisy image and ϵθ as the multi-view diffusion model.
  • Training procedure: Training alternates diffusion updates on multi-view 3D data and 2D text-to-image data by sampling noisy inputs and backpropagating the loss.The pseudocode samples either four orthogonal views from a 3D example or four random 2D images before adding noise and updating the model.

A.3.1 DATA PREPARATION AND DIFFUSION MODEL TRAINING

MVDream is trained from filtered Objaverse renders together with 2D images, using four-view supervision and a fine-tuned Stable Diffusion model; the appendix reports additional qualitative comparisons and DreamBooth results.

  • Data preparation: The 3D training data comes from filtered Objaverse, yielding about 350K objects whose rendered images match their names.Objects are normalized and rendered from randomized camera settings before training.
  • Diffusion model training: Training samples 3D data 70% of the time and LAION data 30% of the time, using four orthogonal views for 3D examples.The four views are selected from 32 rendered views, with normalized camera parameters supplied for the 3D samples.
  • Diffusion model training: The model is fine-tuned from Stable Diffusion v2.1 at 256×256 resolution for 50,000 steps using a total batch size of 1,024.Training takes about three days on 32 Nvidia Tesla A100 GPUs.
  • Additional results: The appendix presents additional image-generation, baseline-comparison, and DreamBooth results for MVDream.Figures 14–17 cover training-prompt generations, 3D-generation comparisons, and comparisons with DreamBooth3D.
Loading 2308.16512v4…