Source-linked AI summary
DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Model
Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, Kai Zhang
TL;DR
3D generation methods can be slow, artifact-prone, or dependent on ground-truth 3D supervision. DMV3D uses a transformer-based 3D reconstruction model to denoise multi-view diffusion and generate triplane NeRFs from text or images. It achieves fast single-stage generation, outperforms previous 3D diffusion models for text-to-3D, and reaches state-of-the-art quality for single-view reconstruction.
Problem
Existing 3D generation methods require lengthy per-asset optimization, may produce geometry artifacts, or typically need ground-truth 3D data for training.
Method
DMV3D integrates an LRM-based transformer that reconstructs clean triplane NeRFs from noisy multi-view images and renders denoised views within a single-stage diffusion framework.
Results
DMV3D achieves fast direct-inference 3D generation in ∼30s on a single A100 GPU, outperforms previous 3D diffusion models for text-to-3D, and achieves state-of-the-art single-view reconstruction quality.
Takeaways & Limitations
The framework supports text- and image-conditioned 3D generation while unifying 3D reconstruction and generation without per-asset optimization.
Takeaways & Limitations
Generated unseen-region textures can lack high-frequency details and have slightly washed-out colors.
Abstract
from arXiv · showhide
We propose \textbf{DMV3D}, a novel 3D generation approach that uses a transformer-based 3D large reconstruction model to denoise multi-view diffusion. Our reconstruction model incorporates a triplane NeRF representation and can denoise noisy multi-view images via NeRF reconstruction and rendering, achieving single-stage 3D generation in $\sim$30s on single A100 GPU. We train \textbf{DMV3D} on large-scale multi-view image datasets of highly diverse objects using only image reconstruction losses, without accessing 3D assets. We demonstrate state-of-the-art results for the single-image reconstruction problem where probabilistic modeling of unseen object parts is required for generating diverse reconstructions with sharp textures. We also show high-quality text-to-3D generation results outperforming previous 3D diffusion models. Our project website is at: https://justimyhxu.github.io/projects/dmv3d/ .
1 INTRODUCTION
DMV3D addresses slow, artifact-prone, or 3D-supervision-dependent approaches with a single-stage, category-agnostic framework that denoises multi-view images through 3D reconstruction. It supports text and single-image conditioning, probabilistic reconstructions, and direct inference in about 30 seconds.
- SDS-based 3D generation often requires hours of per-asset optimization and can produce geometry artifacts such as the multi-face Janus problem.
- Existing 3D diffusion models typically require ground-truth 3D models or point clouds for training, which are difficult to obtain for real images.
- DMV3D is a single-stage, category-agnostic diffusion model that generates triplane NeRFs from text or single-image inputs via direct model inference.
- The method integrates 3D NeRF reconstruction and rendering into a multi-view image denoiser trained end-to-end without direct 3D supervision.
- ∼30s on a single A100 GPU enables diverse, high-fidelity 3D objects through fast direct model inference.
- DMV3D bridges 2D and 3D generative models while unifying 3D reconstruction and generation.
2 RELATED WORK
Prior 3D generation methods include neural representations, 3D GANs, and 3D-aware diffusion models, but related diffusion approaches had not demonstrated scalable operation on highly diverse multi-view datasets. DMV3D uses an LRM-based 3D denoiser to target that setting.
- Neural representations support scene representation and neural rendering, including novel-view synthesis in single-scene overfitting scenarios with many multi-view images.
- 3D GANs extend 2D image synthesis capabilities to generating 3D-aware assets from unstructured collections of single-view images.
- Several 3D-aware diffusion models use direct 3D supervision to extend diffusion modeling into the 3D domain.
- RenderDiffusion and Viewset Diffusion use 2D diffusion models with 3D-aware denoisers, but neither was demonstrated on highly diverse datasets with multi-view data for more than 1M objects.
- DMV3D introduces an LRM-based 3D denoiser architecture for scalable, diverse, and high-quality 3D generation.
3 METHOD
DMV3D uses a reconstruction-based denoiser to progressively denoise noisy multi-view images by reconstructing and rendering a 3D representation. Its transformer architecture conditions on diffusion time, camera rays, and optional image or text inputs, while training supervises rendered views with image reconstruction losses.
- 3.1 MULTI-VIEW DIFFUSION AND DENOISING: DMV3D progressively denoises multi-view images by reconstructing a 3D representation and rendering denoised views.The reconstruction module maps noisy views to a 3D representation, and the renderer produces denoised images.
- 3.1 MULTI-VIEW DIFFUSION AND DENOISING: Novel-view supervision is added because supervising only input-view predictions can produce degenerate 3D solutions with view-aligned image planes.The training objective compares ground-truth images with renderings from randomly selected input and novel views.
- 3.2 RECONSTRUCTOR-BASED MULTI-VIEW DENOISER: The LRM-based denoiser reconstructs a clean triplane NeRF from noisy sparse-view posed images, whose renderings become denoising outputs.The triplane NeRF represents the asset’s shape and appearance and supports differentiable volume rendering.
- 3.2 RECONSTRUCTOR-BASED MULTI-VIEW DENOISER: Camera conditioning uses pixel-aligned Plücker rays instead of jointly applying the same conditioning strategy to camera and diffusion time.Each ray is parameterized by its origin and direction, then concatenated with image pixels before ViT tokenization.
- 3.3 CONDITIONAL GENERATION: The model supports image conditioning by keeping one view noise-free and text conditioning through CLIP embeddings injected with cross-attention.Other views are diffused and denoised using cues from the fixed image view.
- 3.4 TRAINING AND INFERENCE: Training samples diffusion time steps and camera poses, adds cosine-schedule noise, and uses L2 plus LPIPS image reconstruction losses.Additional novel viewpoints are sampled to supervise renderings for better quality; the loss weights are 1 for L2 and 2 for LPIPS.
4 EXPERIMENTS
DMV3D is evaluated for single-image reconstruction and text-to-3D generation, with experiments covering quantitative comparisons, qualitative results, robustness, diversity, and ablations. The study uses large-scale multi-view datasets and reports strong reconstruction quality, flexible inputs, and benefits from four-view conditioning and diverse supervision.
- 4.1 SETTINGS: The experiments train on approximately 730k Objaverse objects and add approximately 220k real MVImgNet captures for image-conditioned reconstruction.The text-to-3D model uses captions covering approximately 660k Objaverse objects.
- 4.1 SETTINGS: Single-image reconstruction is evaluated on GSO and ABO using PSNR, LPIPS, CLIP similarity, FID, and Chamfer distance.Novel-view rendering metrics compare rendered and ground-truth images, while Chamfer distance evaluates extracted mesh geometry.
- 4.2 RESULTS AND COMPARISONS: DMV3D achieves the best scores across all reported metrics on both GSO and ABO, with more realistic geometry and sharper appearance details than baselines.The qualitative comparison reports higher-quality geometry and sharper appearance details than competing methods.
- 4.2 RESULTS AND COMPARISONS: Two-stage 3D diffusion baselines often produce incomplete shapes and blurry textures, while Zero-1-to-3 lacks a 3D output and suffers viewpoint inconsistency.Magic123 provides a 3D model but requires hours of per-asset optimization and can exhibit Janus artifacts.
- 4.2 RESULTS AND COMPARISONS: DMV3D achieves the best text-to-3D precision among the compared category-agnostic models supporting fast direct inference.The evaluation uses CLIP precision and averaged precision with two ViT models.
- 4.3 ANALYSIS, ABLATION, AND APPLICATION: Using more input views generally improves quality, but the gain from six over four views is marginal, so four views are used by default.Some metrics, including PSNR and FID, are better for the four-view model than the six-view model.
- 4.3 ANALYSIS, ABLATION, AND APPLICATION: Different random seeds reproduce the frontal input while varying unseen back-side shape and appearance, demonstrating diverse reconstructions.The model also robustly reconstructs geometry and appearance from synthetic, real, and generated out-of-domain images.
- 4.3 ANALYSIS, ABLATION, AND APPLICATION: Removing MVImgNet can produce unrealistic flat shapes, while removing novel-view supervision causes view-aligned image pasting instead of plausible 3D reconstruction.The Plucker coordinate-based camera conditioning also outperforms its ablated alternative quantitatively.
5 CONCLUSION
DMV3D is a single-stage diffusion framework for fast, text- and image-conditioned 3D generation, using multi-view denoising through triplane NeRF reconstruction and rendering. The paper reports strong generation and reconstruction results while identifying limitations in texture detail, resolution, backgrounds, and 2D prior usage.
- Conclusion: DMV3D generates 3D assets by denoising multi-view image diffusion with a large-transformer-based triplane NeRF denoiser.The model reconstructs a clean triplane NeRF from noisy multi-view images and renders denoised views.
- Conclusion: The framework supports text and image conditioning, direct diffusion inference, and fast generation without per-asset optimization.The abstract reports approximately 30 seconds per asset on a single A100 GPU.
- Conclusion: The method outperforms previous 3D diffusion models for text-to-3D generation and achieves state-of-the-art single-view reconstruction quality.These outcomes are reported across various testing datasets for single-view reconstruction.
- Limitations: Unseen-object textures can lack high-frequency details and have slightly washed-out colors.The authors identify improving texture fidelity as future work.
- Limitations: The current system uses low-resolution input images and triplanes, supports objects without backgrounds, and is trained from scratch without strong 2D foundation-model priors.The authors propose high-resolution generation, background modeling, and use of 2D image priors as future directions.
- Ethics and scope: The training dataset is about 1M samples, smaller than datasets used for 2D diffusion models, which may limit coverage of testing diversity and introduce distribution bias.The authors also note possible unsatisfying outputs for out-of-knowledge inputs and potential training-data leakage.
A.1 ROBUSTNESS EVALUATION.
The robustness evaluation tests the image-conditioned model on GSO renderings with camera fields of view and lighting conditions that differ from some training renderings. Table 4 reports this robustness evaluation on GSO.
- Robustness evaluation: The evaluation uses GSO renderings with different camera Field-Of-Views and lighting conditions to assess robustness.The training data combines diverse MVImgNet conditions with Objaverse renderings using a constant 50° FOV and uniform lighting.
- Robustness evaluation: Testing includes images with other FOV angles and complex environmental lighting beyond the constant-FOV, uniform-lighting Objaverse renderings.MVImgNet supplies additional diversity in camera FOVs and lighting conditions during training.
- Robustness evaluation: Table 4 is titled “Robustness on GSO dataset.”The table summarizes the robustness evaluation described in the accompanying text.
A.2 QUANTATIVE EVALUATION ON MVIMGNET.
The MVImgNet ablation examines whether adding diverse real data improves generalization. The reported qualitative and quantitative comparisons indicate improved generalization without compromising reconstructed appearance or geometry on GSO.
- MVImgNet ablation: MVImgNet’s diverse real data improves generalization to real or out-of-domain data.The paper cites Figure 7 as qualitative evidence for this effect.
- MVImgNet ablation: Reconstructions with and without MVImgNet have similar appearance and geometry on the GSO dataset.The authors interpret this similarity as evidence that MVImgNet improves generalization without compromising reconstruction quality.
- MVImgNet ablation: Table 5 reports the quantitative ablation comparing models trained with and without MVImgNet.The comparison is conducted on the GSO dataset.
- MVImgNet ablation: Figure 7 provides a qualitative comparison of the model trained with and without MVImgNet.The figure focuses on the effect of including MVImgNet in training.
A.3 IMPLEMENTATION DETAILS.
The implementation uses PyTorch and guided diffusion with AdamW optimization, while image- and text-conditioned models share a closely related architecture. Table 6 lists the detailed model configurations and attention notation.
- Implementation details: The experiments use PyTorch, a codebase built on guided diffusion, and AdamW with weight decay 0.05 and beta values (0.9, 0.95).These are the reported optimizer and implementation settings.
- Implementation details: Table 6 presents detailed configurations for the image-conditioned models.The table includes attention-related configuration details.
- Implementation details: The text-conditioned architecture closely mirrors the image-conditioned models but differs in how the condition signal is injected.This distinction separates the conditioning mechanisms while retaining the broader architecture.
- Implementation details: In Table 6, “Att.” denotes attention, while a and c represent self-attention and cross attention.The caption defines the notation used in the implementation table.
A.4 VIEW NUMBERS
The section qualitatively compares reconstruction quality across different numbers of input views. One view is blurry, four views are clearer, and six views provide no significant improvement over four.
- One input view produces a very blurry predicted novel view.
- Four input views make the reconstruction results much clearer than using one view.
- Six views offer no significant improvement over four, indicating performance saturation.
A.5 MORE COMPARISON.
The section adds qualitative comparisons for single-image reconstruction and presents them in Figure 9. Figure 8 is identified as a qualitative comparison across different view numbers.
- Additional qualitative comparisons address single-image reconstruction.
- Figure 8 presents qualitative comparisons across different view numbers.
- Figure 9 presents qualitative comparisons on single-image reconstruction.