PaperlayerView exact revision on arXiv

Source-linked AI summary

DiT-3D: Exploring Plain Diffusion Transformers for 3D Shape Generation

Shentong Mo, Enze Xie, Ruihang Chu, Lewei Yao, Lanqing Hong, Matthias Nießner, Zhenguo Li

arXiv:2307.01831v1cs.CVcs.AIcs.LG

TL;DR

The paper asks whether plain diffusion Transformers can match their 2D success in 3D shape generation, where U-Net-based diffusion methods have dominated. It introduces DiT-3D for denoising voxelized point clouds with 3D adaptations and efficient 2D-to-3D transfer, achieving state-of-the-art ShapeNet generation quality and diversity.

  • Problem

    Plain diffusion Transformers had not been established for 3D shape generation, while most existing 3D diffusion approaches used U-Net backbones.

  • Method

    DiT-3D applies a plain diffusion Transformer to voxelized point clouds using 3D positional and patch embeddings, window attention, and devoxelized prediction.

  • Results

    DiT-3D achieves state-of-the-art ShapeNet generation and, against DPF-Net for chair generation, decreases 1-NNA@CD by 12.89 and increases COV@CD by 7.74.

  • Takeaways & Limitations

    Plain diffusion Transformers provide a scalable alternative to U-Net approaches for high-fidelity and diverse 3D point-cloud generation.

  • Takeaways & Limitations

    The work does not explore other 3D modalities such as signed distance fields and meshes or large-scale training on more 3D shapes.

Abstract

from arXiv · show

Recent Diffusion Transformers (e.g., DiT) have demonstrated their powerful effectiveness in generating high-quality 2D images. However, it is still being determined whether the Transformer architecture performs equally well in 3D shape generation, as previous 3D diffusion methods mostly adopted the U-Net architecture. To bridge this gap, we propose a novel Diffusion Transformer for 3D shape generation, namely DiT-3D, which can directly operate the denoising process on voxelized point clouds using plain Transformers. Compared to existing U-Net approaches, our DiT-3D is more scalable in model size and produces much higher quality generations. Specifically, the DiT-3D adopts the design philosophy of DiT but modifies it by incorporating 3D positional and patch embeddings to adaptively aggregate input from voxelized point clouds. To reduce the computational cost of self-attention in 3D shape generation, we incorporate 3D window attention into Transformer blocks, as the increased 3D token length resulting from the additional dimension of voxels can lead to high computation. Finally, linear and devoxelization layers are used to predict the denoised point clouds. In addition, our transformer architecture supports efficient fine-tuning from 2D to 3D, where the pre-trained DiT-2D checkpoint on ImageNet can significantly improve DiT-3D on ShapeNet. Experimental results on the ShapeNet dataset demonstrate that the proposed DiT-3D achieves state-of-the-art performance in high-fidelity and diverse 3D point cloud generation. In particular, our DiT-3D decreases the 1-Nearest Neighbor Accuracy of the state-of-the-art method by 4.59 and increases the Coverage metric by 3.51 when evaluated on Chamfer Distance.

1 Introduction

DiT-3D addresses the open question of whether plain diffusion Transformers can generate 3D shapes effectively, replacing the predominantly used U-Net design with a voxel-based Transformer. Its adaptations and ImageNet initialization support scalable, high-quality generation on ShapeNet.

  • 1 Introduction: Plain diffusion Transformers remain underexplored for 3D shape generation because most existing 3D diffusion methods use U-Net backbones.The paper investigates whether the Transformer advantages observed in 2D image generation extend to 3D shapes.
  • 1 Introduction: By training only 0.09MB of parameters, DiT-3D achieves comparable quality and diversity across all metrics through parameter-efficient transfer from DiT-2D.The reported training parameters decrease from 32.8MB to 0.09MB.
  • 1 Introduction: DiT-3D is the first plain diffusion Transformer for point-cloud shape generation that denoises voxelized point clouds.The model extends DiT-2D with 3D-specific components for 3D generation.
  • 1 Introduction: DiT-3D combines 3D positional and patch embeddings, 3D window attention, and 2D ImageNet pre-training to improve performance while maintaining efficiency.These modifications adapt the Transformer to voxelized point clouds and reduce the cost of 3D self-attention.
  • 1 Introduction: DiT-3D achieves state-of-the-art high-fidelity 3D shape generation on ShapeNet against previous non-DDPM and DDPM baselines.The evaluation covers diverse object classes and includes qualitative visualizations and ablations.

2 Related Work

Prior 3D shape-generation research spans probabilistic, diffusion, and Transformer-based generative models. DiT-3D builds on these lines by applying a plain diffusion Transformer to voxelized 3D point clouds.

  • 2 Related Work: 3D shape-generation methods include variational autoencoders, generative adversarial networks, normalizing flows, energy-based models, and diffusion models.These approaches target synthesis of high-fidelity point clouds or meshes.
  • 2 Related Work: Diffusion models generate shapes by learning a reverse denoising process from noisy data, following the forward Gaussian noising process of DDPMs.Prior 3D diffusion pipelines explore different representations and generation procedures.
  • 2 Related Work: Diffusion Transformers have demonstrated high-fidelity image generation by replacing or adapting U-Net-style architectures with Transformer-based denoising models.Representative designs include DiT and U-ViT.
  • 2 Related Work: DiT-3D extends this Transformer-based diffusion direction to voxelized 3D point clouds using 3D positional embeddings, patch embeddings, and window attention.The pipeline predicts point-cloud noise after processing voxel tokens.

3 Method

DiT-3D adapts plain diffusion Transformers to voxelized 3D point clouds through 3D embeddings, windowed attention, and devoxelized prediction. It also supports parameter-efficient transfer from 2D DiT models to 3D generation.

  • 3.2 Diffusion Transformer for 3D Point Cloud Generation: DiT-3D operates DDPM denoising directly on voxelized point clouds rather than latent codes, using a plain Transformer architecture.Voxelization converts unordered, sparse point coordinates into dense representations for Transformer processing.
  • 3.2 Diffusion Transformer for 3D Point Cloud Generation: 3D positional and patch embeddings convert voxel inputs into localized token embeddings and incorporate time and class conditions for multi-class generation.The model uses frequency-based sine-cosine 3D positional embeddings and 3D patchification.
  • 3.2 Diffusion Transformer for 3D Point Cloud Generation: 3D window attention reduces the cost of self-attention caused by the cubic growth of token length at higher voxel resolutions.Windowed attention replaces global attention by grouping tokens with window size R.
  • 3.2 Diffusion Transformer for 3D Point Cloud Generation: A devoxelized prediction layer transforms decoded voxel tokens into an N × 3 noise prediction matching the point-cloud space.The output is linearly decoded, unpatchified into a voxel tensor, and then devoxelized.
  • 3.3 Efficient Modality/Domain Transfer with Parameter-efficient Fine-tuning: Parameter-efficient fine-tuning transfers DiT knowledge from ImageNet-pretrained 2D models to 3D point-cloud generation and across object classes.The approach leverages the structural and parameter similarity between DiT-2D and DiT-3D.

4 Experiments

DiT-3D is evaluated on ShapeNet using standard point-cloud metrics and comparisons against prior generation methods. Results show strong benchmark performance, benefits from its 3D components and transfer learning, and scalability across design choices.

  • 4.1 Experimental Setup: The evaluation uses ShapeNet Chair, Airplane, and Car point clouds, with Chamfer Distance and Earth Mover’s Distance underlying 1-NNA and Coverage.Each shape uses 2,048 sampled points, following prior dataset splits and preprocessing.
  • 4.2 Comparison to State-of-the-art Works: 12.89 1-NNA@CD and 7.80 1-NNA@EMD decreases, with 7.74 COV@CD and 3.8 COV@EMD increases, outperform DPF-Net on chair generation.DiT-3D also gains 4.59 1-NNA@CD and 1.61 1-NNA@EMD, plus 3.51 COV@CD and 2.21 COV@EMD, over LION.
  • 4.3 Experimental Analysis: Adding voxel diffusion, 3D positional embeddings, and 3D window attention improves generation, while window attention reduces training cost by 44.86 hours.The ablation reports lower 1-NNA and higher COV after introducing these components.
  • 4.3 Experimental Analysis: 2D ImageNet initialization decreases 1-NNA by 2.92@CD and 0.18@EMD, while domain transfer trains only 0.09MB parameters with comparable quality and diversity.These results support modality and domain transfer through parameter-efficient fine-tuning.
  • 4.3 Experimental Analysis: Increasing voxel size from 16 to 64 improves all metrics, and model scaling from S/4 to XL/4 lowers 1-NNA while generally increasing COV.The reported model sizes range from 32.8MB to 674.7MB.

5 Conclusion

The paper presents DiT-3D as a plain diffusion transformer for voxelized point-cloud denoising. It combines 3D adaptations and efficient transfer mechanisms, achieving state-of-the-art high-fidelity and diverse 3D generation.

  • 5 Conclusion: DiT-3D directly denoises voxelized point clouds with 3D positional and patch embeddings, 3D window attention, and devoxelization layers.The architecture is designed to reduce 3D attention cost while predicting denoised point clouds.
  • 5 Conclusion: DiT-3D supports parameter-efficient fine-tuning with modality and domain transferability through the scalability of its Transformer architecture.The conclusion connects Transformer scalability with flexible transfer settings.
  • 5 Conclusion: Empirical results demonstrate state-of-the-art performance in high-fidelity and diverse 3D point-cloud generation.The appendix notes that limitations and broader impact are discussed separately.

A.1 Results on Multi-class Training

DiT-3D supports multi-class training with learnable class embeddings, allowing one global model to achieve competitive results against category-specific models.

  • A.1 Results on Multi-class Training: Competitive results across all metrics show that multi-class DiT-3D matches category-specific models on chair generation.Training classes expand from Chair to Chair,Car and Chair,Car,Airplane while testing remains on chairs.
  • A.1 Results on Multi-class Training: One global model can serve all three classes instead of requiring repeated training of separate class-specific models.The approach uses learnable class embeddings as the conditioning mechanism.

A.2 Effect of Sampling Steps

Sampling-step experiments show that DiT-3D performs best when inference uses 1000 DDPM sampling steps.

  • A.2 Effect of Sampling Steps: 1000 sampling steps produce the best results across CD and EMD, with the lowest 1-NNA and highest COV.The observed trend agrees with conclusions from prior DDPM work.

B.1 Comparisons with State-of-the-art Works

Qualitative comparisons show that DiT-3D generates higher-quality 3D point clouds than several prior methods, including diffusion and voxel-based baselines.

  • B.1 Comparisons with State-of-the-art Works: Visual comparisons position DiT-3D against SetVAE, DPM, and PVD as qualitative state-of-the-art baselines.PVD is identified as a strong diffusion model based on point voxels.
  • B.1 Comparisons with State-of-the-art Works: DiT-3D produces higher-fidelity and more diverse point clouds than SetVAE, DPM, and PVD.The comparison covers generated point clouds from all three object classes.
  • B.1 Comparisons with State-of-the-art Works: With sampling progressing toward 1000 steps, generated chair shapes become more realistic rather than resembling initial random noise.The diffusion visualization follows six samples from noise to final shapes.

B.3 More Visualizations of Generated Shapes

Additional visualizations across three object classes demonstrate that DiT-3D produces high-fidelity and diverse shapes through direct point-cloud denoising.

  • B.3 More Visualizations of Generated Shapes: Visualizations across all three classes demonstrate high-fidelity and diverse shapes from the proposed 3D design components.The model performs the denoising process directly from point clouds across three categories.

C Discussion

DiT-3D generates high-fidelity and diverse shapes across several ShapeNet categories, while the authors identify broader modality and scaling limitations. The model may also inherit biases from the ShapeNet training data, requiring care in real-world deployment.

  • Limitations and future work: The authors have not explored DiT-3D on other 3D modalities, such as signed distance fields or meshes, or at large scale on more 3D shapes.These directions are left for future work.
  • Broader impact: Training on the ShapeNet benchmark might cause DiT-3D to learn internal biases in the data, which should be addressed before real-world deployment.The authors flag biased problems as a broader-impact concern.
  • Qualitative results: The diffusion visualizations show progression from random noise to final 3D shapes for Chair, Airplane, and Car generation.Each sequence is arranged from left to right.
  • Qualitative results: Qualitative visualizations present high-fidelity and diverse generated shapes for Chair, Airplane, and Car categories.The figures show category-specific results for these three object classes.
Loading 2307.01831v1…