PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion-SDF: Text-to-Shape via Voxelized Diffusion

Muheng Li, Yueqi Duan, Jie Zhou, Jiwen Lu

arXiv:2212.03293v2cs.CVcs.AIcs.GRcs.LG

TL;DR

Text-to-shape generation needs representations and generators that can produce diverse 3D shapes matching natural-language conditions. Diffusion-SDF uses a patch-based SDF autoencoder followed by Voxelized Diffusion with UinU-Net, and reports high-quality, diverse, text-conforming outputs across text-conditioned tasks. Its evaluation is limited by the Text2Shape dataset’s two ShapeNet categories and the lack of additional shape-text benchmarks.

  • Problem

    Existing text-to-shape approaches struggle to generate highly diversified, high-quality 3D shapes that conform to input text semantics.

  • Method

    Diffusion-SDF uses a patch-based SDF autoencoder and Voxelized Diffusion with UinU-Net to generate patch-independent voxelized SDF representations.

  • Results

    Diffusion-SDF generates high-quality and highly diverse 3D shapes across text-to-shape synthesis, completion, and manipulation tasks.

  • Takeaways & Limitations

    The framework supports text-conditioned 3D shape generation and related completion and manipulation tasks using voxelized SDFs.

  • Takeaways & Limitations

    Evaluation uses Text2Shape’s two ShapeNet categories, and a lack of shape-text datasets prevents validation on additional benchmarks.

Abstract

from arXiv · show

With the rising industrial attention to 3D virtual modeling technology, generating novel 3D content based on specified conditions (e.g. text) has become a hot issue. In this paper, we propose a new generative 3D modeling framework called Diffusion-SDF for the challenging task of text-to-shape synthesis. Previous approaches lack flexibility in both 3D data representation and shape generation, thereby failing to generate highly diversified 3D shapes conforming to the given text descriptions. To address this, we propose a SDF autoencoder together with the Voxelized Diffusion model to learn and generate representations for voxelized signed distance fields (SDFs) of 3D shapes. Specifically, we design a novel UinU-Net architecture that implants a local-focused inner network inside the standard U-Net architecture, which enables better reconstruction of patch-independent SDF representations. We extend our approach to further text-to-shape tasks including text-conditioned shape completion and manipulation. Experimental results show that Diffusion-SDF generates both higher quality and more diversified 3D shapes that conform well to given text descriptions when compared to previous approaches. Code is available at: https://github.com/ttlmh/Diffusion-SDF

1. Introduction

Diffusion-SDF addresses text-to-shape generation with voxelized signed distance fields and a two-stage diffusion pipeline designed for diverse, text-conforming 3D shapes.

  • 1. Introduction: Implicit representations encode shapes as continuous fields evaluable at arbitrary resolution, unlike explicit representations with watertightness or topology limitations.The passage contrasts point clouds, meshes, and occupancy grids with implicit functions.
  • 1. Introduction: Existing implicit generative models support reconstruction and completion but generally cannot generate shapes matching a specified text condition.The limitation concerns conditioning novel 3D shape generation on natural-language descriptions.
  • 1. Introduction: Text guidance offers flexible 3D generation and supports iterative synthesis and fine-grained editing for non-expert users.The paper motivates natural-language control as a flexible interface for creating and manipulating 3D content.
  • 1. Introduction: Prior text-to-shape methods still struggle to produce highly diverse, high-quality shapes that conform to input semantics.The paper identifies quality, diversity, and text conformance as simultaneous challenges.
  • 1. Introduction: Diffusion-SDF maps voxelized TSDFs to patch-independent local Gaussian latents, then uses Voxelized Diffusion with UinU-Net to generate shape representations.UinU-Net adds a local-focused inner network inside an outer U-Net to reconstruct local patch features from noise.
  • 1. Introduction: The framework uses a combined explicit-implicit representation consisting of voxelized signed distance fields.This representation combines voxelization with an implicit signed-distance formulation.

2. Related Work

Related work spans unconditional, image-conditioned, and text-conditioned 3D generation, while diffusion models have become prominent for diverse generative modeling. Diffusion-SDF focuses on text-conditioned synthesis using implicit voxelized SDF representations and a two-stage pipeline.

  • 2. Related Work: Prior generative 3D methods synthesize shapes unconditionally, from images, or from text, using both explicit and implicit representations.The paper positions its work among several conditioning modalities and representation choices.
  • 2. Related Work: Diffusion models are reported to offer training stability and generative diversity across image, speech, and 3D synthesis tasks.Examples include point-voxel generation, point-cloud diffusion, and hierarchical latent diffusion.
  • 2. Related Work: Diffusion-SDF is organized as Stage I SDF Autoencoding followed by Stage II Voxelized Diffusion Models.The two stages form the paper’s overall generation pipeline.

3. Method

Diffusion-SDF uses a two-stage pipeline that encodes voxelized TSDF shapes into patch-independent latent representations and generates them with text-conditioned diffusion. A UinU-Net denoiser combines patch-focused processing with spatial context, while masking supports text-conditioned shape completion.

  • SDF autoencoding: A patch-joint decoder rearranges latent patch embeddings into a voxel-grid embedding to recover patch-to-global and patch-to-patch information during reconstruction.The decoder reconstructs the SDF field from the latent patches.
  • SDF autoencoding: The SDF autoencoder uses a VAE-like Gaussian latent distribution trained with L1 reconstruction and KL-regularization losses.The KL term mildly encourages each learned patch latent toward a standard normal distribution.
  • Voxelized diffusion: Diffusion models gradually transform data into Gaussian noise and learn a reverse process that denoises the samples through a fixed Markov chain.The model trains a noise estimator with an expected squared error objective.
  • Voxelized diffusion: The diffusion model operates on latent voxel-grid embeddings using a 3D U-Net denoiser, with UinU-Net adding an inner patch-focused network and spatial transformer.The inner network mainly uses 1 × 1 × 1 convolutions for patch-wise features, while the spatial transformer supplies patch-to-patch information.
  • Text conditioning: Text conditioning encodes captions with a pretrained text encoder and uses cross-attention plus classifier-free guidance to steer latent SDF generation.The guided score extrapolates toward the text-conditioned estimate and away from the empty-caption estimate.
  • Text-guided shape completion: For text-conditioned shape completion, mask diffusion preserves known patches during reverse sampling while generating missing patches with a pretrained voxelized diffusion model.At each denoising step, estimated results are combined with forward-sampled values for unmasked patches.

4. Experiments

Experiments evaluate Diffusion-SDF on text-to-shape generation, UinU-Net ablations, text-guided completion, and manipulation. The results indicate strong quality, semantic conformance, diversity, patch recovery, and text-guided editing capabilities, with patch-grid boundaries remaining a completion limitation.

  • Text-Conditioned Shape Generation: Diffusion-SDF achieves high generation quality and text-shape conformance while producing substantially more diverse samples than previous implicit-representation methods.The evaluation uses IoU, Acc, CLIP-S, and TMD; lower TMD indicates greater diversity.
  • Text-Conditioned Shape Generation: The method generates varied shapes from different text descriptions and multiple shape outputs from the same text query.Qualitative comparisons report satisfactory quality alongside high diversity for identical text inputs.
  • Ablation Studies: UinU-Net generally improves generated-shape quality by recovering independently distributed patch representations through patch-mutual and patch-independent noise estimation.The inner network is designed to preserve patch independence while retaining global shape information.
  • Ablation Studies: Removing inner-outer concatenation decreases accuracy and CLIP-S, while removing spatial attention reduces accuracy by weakening patch-to-patch information.These ablations link the information-transmission mechanisms to semantic authenticity, text conformance, and full-shape semantics.
  • Text-Guided Shape Completion: For text-guided completion, the method generates missing parts that blend with partial inputs, but patch-grid cut regions remain a limitation.The reported remedy is higher resolution with smaller patch size.
  • Text-Guided Shape Manipulation: For text-guided manipulation, the method modifies local structures or overall shape characteristics, with t_mid controlling preservation versus destruction of original features.The authors use relatively large t_mid values because shape characteristics usually form early in the reverse process.

5. Limitations and Conclusion

Diffusion-SDF uses a two-stage patch-wise autoencoder and Voxelized Diffusion pipeline with UinU-Net denoisers to generate diverse, high-quality 3D shapes. Its evaluation is limited by the Text2Shape dataset’s two ShapeNet categories and the lack of additional shape-text benchmarks.

  • Limitations: Evaluation on Text2Shape is restricted to two ShapeNet categories, limiting generalization to other categories.The authors identify this dataset scope as a limitation of the demonstrated generalization.
  • Limitations: The lack of current shape-text datasets prevents validation on additional benchmarks.The authors propose introducing more datasets and exploring zero-shot text-to-shape generation using knowledge from 2D vision-language models.
  • Conclusion: The framework generates voxelized SDFs conditioned on text through a patch-wise autoencoder followed by Voxelized Diffusion with UinU-Net denoisers.The pipeline produces patch-independent Gaussian SDF representations before diffusion-based generation.
  • Conclusion: Diffusion-SDF supports various text-to-shape synthesis tasks and generates highly diverse, high-quality 3D shapes.The conclusion reports results across multiple text-to-shape tasks.

A. Qualitative Illustrations

The section provides additional qualitative examples covering text-conditioned generation, completion, and manipulation from varied inputs.

  • Text-conditioned shape generation: Additional samples illustrate diversified text-conditioned shape generation.Figure 9 presents extra generated samples.
  • Text-conditioned shape completion: Further results show text-conditioned shape completion for different input shapes.Figure 10 displays completion results based on different initial shapes.
  • Text-conditioned shape manipulation: Several results illustrate text-conditioned shape manipulation from diverse initial shapes.Figure 11 presents manipulation examples using diverse starting shapes.
Loading 2212.03293v2…