Source-linked AI summary
Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformer
Shuang Wu, Youtian Lin, Feihu Zhang, Yifei Zeng, Jingxi Xu, Philip Torr, Xun Cao, Yao Yao
TL;DR
Scalable 3D representations remain difficult for capturing intricate geometry distributions. Direct3D addresses this with a compact triplane latent and image-conditioned 3D diffusion, and experiments report superior quality and generalization over existing approaches.
Problem
Existing implicit 1D latent representations struggle to capture structured 3D information and provide an unstructured, under-constrained space for modeling and sampling.
Method
Direct3D combines D3D-VAE, which directly supervises decoded geometry in a compact triplane latent, with D3D-DiT, which fuses cross-plane positional information and pixel- and semantic-level image conditions.
Results
Direct3D surpasses previous image-to-3D approaches in generation quality and generalization ability, establishing a new state-of-the-art for 3D content creation.
Takeaways & Limitations
Direct3D enables high-fidelity image-to-3D generation without multi-view diffusion models or SDS optimization.
Takeaways & Limitations
Direct3D is limited to generating individual or multiple objects and cannot generate large-scale scenes.
Abstract
from arXiv · showhide
Generating high-quality 3D assets from text and images has long been challenging, primarily due to the absence of scalable 3D representations capable of capturing intricate geometry distributions. In this work, we introduce Direct3D, a native 3D generative model scalable to in-the-wild input images, without requiring a multiview diffusion model or SDS optimization. Our approach comprises two primary components: a Direct 3D Variational Auto-Encoder (D3D-VAE) and a Direct 3D Diffusion Transformer (D3D-DiT). D3D-VAE efficiently encodes high-resolution 3D shapes into a compact and continuous latent triplane space. Notably, our method directly supervises the decoded geometry using a semi-continuous surface sampling strategy, diverging from previous methods relying on rendered images as supervision signals. D3D-DiT models the distribution of encoded 3D latents and is specifically designed to fuse positional information from the three feature maps of the triplane latent, enabling a native 3D generative model scalable to large-scale 3D datasets. Additionally, we introduce an innovative image-to-3D generation pipeline incorporating semantic and pixel-level image conditions, allowing the model to produce 3D shapes consistent with the provided conditional image input. Extensive experiments demonstrate the superiority of our large-scale pre-trained Direct3D over previous image-to-3D approaches, achieving significantly better generation quality and generalization ability, thus establishing a new state-of-the-art for 3D content creation. Project page: https://nju-3dv.github.io/projects/Direct3D/.
1 Introduction
Direct3D addresses inefficient indirect image-to-3D pipelines with a native 3D latent diffusion model. Its D3D-VAE and D3D-DiT support scalable, image-conditioned generation with detailed geometry and strong image consistency.
- Motivation: Existing image-to-3D methods often generate multi-view images before reconstruction or SDS optimization, raising efficiency concerns.Direct3D instead advocates direct 3D-shape generation from single-view images.
- Scope: Direct3D is presented as a native 3D generative model scalable to in-the-wild input images without multi-view diffusion or SDS optimization.The method can use images generated by existing text-to-image diffusion models as inputs while training only on 3D data.
- D3D-VAE: D3D-VAE encodes 3D point clouds into a compact triplane latent and directly supervises decoded geometry with semi-continuous surface sampling.This preserves detailed 3D information without using rendered images as supervision signals.
- D3D-DiT: D3D-DiT is a scalable image-conditioned 3D diffusion transformer that fuses positional information across latent triplane planes.It incorporates pixel-level and semantic-level image information to generate shapes consistent with the input image.
- Results: Direct3D surpasses previous image-to-3D approaches in generation quality and generalization ability, establishing a new state-of-the-art.The authors report extensive experiments supporting this conclusion.
2 Related Work
Related work explores multiple 3D representations, directly trained generators, multi-view reconstruction pipelines, and latent diffusion models. Direct3D responds to limitations involving reconstruction supervision, latent structure, and explicit 3D representation.
- 3D Representations: Prior 3D generation methods use point clouds, voxels, SDFs, and neural representations to capture appearance and geometric detail.NeRF-based methods and occupancy-grid outputs represent different choices for generated 3D structure.
- Multi-view Reconstruction: Multi-view pipelines generate several object views from one image and reconstruct a 3D shape using sparse reconstruction or SDS-based techniques.Subsequent work improves either multi-view generation or reconstruction models.
- Direct 3D Diffusion: Direct 3D diffusion methods fit neural representations or use joint triplane training, but separate NeRF fitting can hinder generalization to diverse shapes.These approaches address the challenge of training diffusion models directly on 3D data.
- Latent 3D Generation: VAE-based methods encode 3D shapes into implicit or explicit latent spaces before diffusion generates the latent representation.Shap-E uses an implicit latent, while 3DGen uses an explicit triplane latent for efficiency.
- Remaining Limitations: Prior methods often rely on rendering loss or implicit latents, limiting reconstruction quality, diffusion efficiency, or the availability of compact explicit 3D representations.Direct3D combines explicit 3D latent representation with direct 3D supervision to improve VAE reconstruction.
3 Methods
Direct3D uses a compact explicit triplane latent space for native 3D generation, combining a D3D-VAE with an image-conditioned diffusion transformer. The method encodes and reconstructs geometry with semi-continuous surface supervision, then injects pixel- and semantic-level image information into every DiT block.
- Direct3D replaces indirect multi-view generation and SDS optimization with native direct generation of 3D shapes from single-view images.
- D3D-VAE: The D3D-VAE encodes point clouds with positions and normals into an explicit triplane latent representation using cross-attention and self-attention.Learnable tokens query point-cloud features before self-attention produces the latent representation z.
- D3D-VAE: A latent-to-triplane decoder upsamples the compact latent representation into high-resolution triplane feature maps, whose interpolated features support occupancy prediction.
- D3D-VAE: Semi-continuous surface sampling supervises occupancy near object surfaces with continuous values while retaining discrete occupancy elsewhere.The geometric mapping network predicts occupancy from triplane-interpolated features, and the SDF determines the supervision regime.
- D3D-VAE: The D3D-VAE is optimized with Binary Cross-Entropy and KL losses, with KL regularization preventing excessive variance in the latent space.
- D3D-DiT: D3D-DiT models the explicit triplane latent with a transformer so information can communicate across planes and preserve 3D spatial structure.A 2D U-Net would lack communication between the three planes.
- D3D-DiT: The image-conditioned diffusion transformer injects DINO-v2 pixel-level features and CLIP semantic features into each DiT block.Pixel-level conditioning targets high-frequency alignment, while cross-attention incorporates semantic image tokens for semantic consistency.
4 Experiments
Direct3D is evaluated on image-to-3D, text-to-3D, and textured-mesh generation, using a large-scale implementation and qualitative or user-study comparisons. The experiments report high-quality meshes, strong image consistency, and applicability to texture synthesis.
- Implementation Details: The D3D-VAE uses 81,920 point clouds with normals and a compact latent representation, while D3D-DiT is trained with a 28-layer DiT-XL/2 configuration.The VAE uses latent resolution r = 32, encoder channel dimension de = 768, and latent channel dimension dz = 16; inference uses 50 DDIM steps with guidance scale 7.5.
- Image-to-3D Generation: Direct3D consistently generates high-quality meshes aligned with conditional images, outperforming baselines that exhibit artifacts, coarse geometry, semantic mismatches, or detail inconsistencies.Compared baselines include Shap-E, Michelangelo, One-2-3-45, and InstantMesh on the GSO image-to-3D task.
- Text-to-3D Generation: Direct3D produces high-quality meshes from text-conditioned images, whereas baseline methods fail in almost all reported cases.All methods use the same generated image as input for the qualitative comparison.
- Text-to-3D Generation: A user study with 46 volunteers found that D3D-DiT achieved superior mesh quality and consistency compared with other methods.Participants rated rotating mesh videos on quality and consistency with the input images using scores from 1 to 5.
- Textured Mesh Generation: Direct3D meshes can be textured with existing synthesis methods, with SyncMVD used to obtain detailed textured meshes.The textured-mesh visualizations use meshes generated by Direct3D.
5 Conclusion
The paper concludes that Direct3D directly generates 3D shapes from single images using compact latent representations and image-conditioned diffusion. Experiments report superior quality and generalizability, while the method remains limited to object-level generation rather than large-scale scenes.
- Conclusion: Direct3D combines D3D-VAE and image-conditioned D3D-DiT to generate 3D shapes directly from single images without multi-view reconstruction.The image-conditioned transformer integrates pixel-level and semantic image information to maintain consistency with conditional images.
- Conclusion: Extensive image-to-3D and text-to-3D experiments report superior generation quality and generalizability over existing methods.The conclusion characterizes Direct3D as surpassing existing methods in both evaluation tasks.
- Limitations: Direct3D currently generates individual or multiple objects but cannot generate large-scale scenes.The authors identify large-scale scene generation as future work.
A.1 Ablation Studies
The ablations examine explicit triplane latents, semi-continuous surface sampling, the D3D-DiT architecture, and pixel-level alignment. These studies link the design choices to reconstruction detail, mesh quality, and image consistency.
- Explicit triplane latent: The explicit triplane latent recovers high-frequency geometric details more effectively than Michelangelo’s implicit 1D latent representation.Figure 7 compares reconstruction results between the two latent representations.
- Semi-continuous surface sampling strategy: Semi-continuous surface sampling alleviates optimization difficulty from abrupt occupancy changes near object surfaces.The strategy samples and supervises surface points in both continuous and discrete manners.
- 2D U-Net vs D3D-DiT: Neither the SD 1.5 nor SD 2.1 2D U-Net architectures produces satisfactory meshes, while D3D-DiT performs better in the comparison.The diffusion models are trained on roll-out triplane latent representations using conditional images generated by Hunyuan-DiT.
- Effectiveness of the pixel-level alignment module: Removing pixel-level alignment preserves relatively high mesh quality but reduces consistency with conditional-image details.The ablation reports mismatches in structures such as a coffee machine’s exterior and a statue’s wings.
A.2 More visualizations
The appendix provides additional visualizations and ablation figures for surface sampling and diffusion-network architecture. The figures are intended to support qualitative comparisons across these design choices.
- Semi-continuous surface sampling strategy: Figure 8 presents an ablation study of the semi-continuous surface sampling strategy.It compares reconstruction behavior with and without the strategy.
- Diffusion-network architecture: Figure 9 presents qualitative comparisons between diffusion models using different network architectures.The comparison concerns D3D-DiT and alternative 2D U-Net architectures.
- More visualizations: Figure 10 contains additional visualizations of Direct3D results.The paper refers to these as more visualizations without specifying a distinct comparison.