Source-linked AI summary
SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation
Yen-Chi Cheng, Hsin-Ying Lee, Sergey Tulyakov, Alexander Schwing, Liangyan Gui
TL;DR
Manual 3D asset creation is difficult for inexperienced users, while textures are often difficult to collect and limited in quality. SDFusion compresses 3D shapes into a latent space for diffusion-based generation with multiple controllable modalities, and it outperforms prior work across shape completion, image-based reconstruction, and text-to-shape while supporting text-based texturing.
Problem
Manual 3D asset creation requires creativity, design skills, and sophisticated software, while 3D textures are difficult to collect and often limited in quality.
Method
SDFusion uses an SDF-based 3D representation, compresses shapes into a low-dimensional latent space for diffusion, and uses task-specific encoders, cross-attention, and classifier-free guidance for multi-modal control.
Results
SDFusion quantitatively and qualitatively outperforms prior work on shape completion, image-based 3D reconstruction, and text-to-shape, while supporting jointly weighted modalities and text-based 3D texturing.
Takeaways & Limitations
The framework combines multiple conditioning modalities into one flexible 3D generation tool and extends generated shapes with textures using a pretrained 2D text-to-image model.
Takeaways & Limitations
SDFusion is trained on high-quality SDF representations and object-centric data, leaving simultaneous operation across representations and application to entire 3D scenes as future directions.
Abstract
from arXiv · showhide
In this work, we present a novel framework built to simplify 3D asset generation for amateur users. To enable interactive generation, our method supports a variety of input modalities that can be easily provided by a human, including images, text, partially observed shapes and combinations of these, further allowing to adjust the strength of each input. At the core of our approach is an encoder-decoder, compressing 3D shapes into a compact latent representation, upon which a diffusion model is learned. To enable a variety of multi-modal inputs, we employ task-specific encoders with dropout followed by a cross-attention mechanism. Due to its flexibility, our model naturally supports a variety of tasks, outperforming prior works on shape completion, image-based 3D reconstruction, and text-to-3D. Most interestingly, our model can combine all these tasks into one swiss-army-knife tool, enabling the user to perform shape generation using incomplete shapes, images, and textual descriptions at the same time, providing the relative weights for each input and facilitating interactivity. Despite our approach being shape-only, we further show an efficient method to texture the generated shape using large-scale text-to-image models.
1. Introduction
SDFusion targets accessible 3D asset creation by combining diverse human-provided conditions in a latent diffusion framework. It supports multiple generation tasks and extends shape generation with text-guided texturing.
- Motivation: Manual 3D asset creation requires design skills and specialized software, limiting accessibility for inexperienced users.Images or rough sketches may be accompanied by text describing details that are difficult to express graphically.
- Framework: SDFusion uses signed distance functions, latent-space diffusion, task-specific encoders, cross-attention, and classifier-free guidance for flexible 3D generation.The latent representation allows scaling to 128^3 resolution while conditioning supports multiple inputs.
- Results: SDFusion achieves superior sample quality, greater conditioning flexibility, and reduced memory usage compared with a latent-space autoregressive model.The comparison is made against a recently proposed autoregressive model using an encoded latent space.
- Texturing: Generated shapes can be textured by combining an off-the-shelf 2D diffusion model, neural rendering, and score distillation sampling.The texturing pipeline uses textual descriptions as conditional variables.
- Capabilities: The model handles shape completion, image-based 3D reconstruction, and text-to-shape generation with multiple conditioning modalities.Users can jointly control modalities and adjust their relative weights.
2. Related Work
Prior 3D generative work explores diverse representations and increasingly uses 2D supervision, while diffusion remains less explored for 3D data. SDFusion applies latent diffusion to SDF representations and combines 2D and 3D modeling directions.
- 3D Representations: 3D generative models have explored point clouds, voxel grids, meshes, and signed distance functions, each with different tradeoffs.SDFs offer a balance of expressivity, memory efficiency, and downstream applicability.
- Diffusion Models: Diffusion models have achieved strong quality, diversity, and expressiveness in many 2D tasks but remain less fully explored for 3D data.The contrast motivates adapting diffusion methods to 3D representations.
- SDFusion: SDFusion applies diffusion models to SDFs using latent diffusion and classifier-free conditional generation.The latent technique addresses the computational demands of modeling reasonable SDF resolutions.
3. Approach
SDFusion compresses high-dimensional 3D shapes into a compact latent space for diffusion-based generation, then conditions denoising on multiple modalities. It also uses 2D diffusion guidance to texture generated shapes.
- 3.1. 3D Shape Compression of SDF: A compact latent representation makes diffusion modeling of high-resolution 3D shapes computationally feasible.The framework first compresses the volumetric T-SDF and applies diffusion in the lower-dimensional space.
- 3.1. 3D Shape Compression of SDF: A 3D VQ-VAE encodes a T-SDF X into latent vector z and decodes its quantized representation back into a reconstructed shape.The encoder, decoder, and codebook are jointly optimized with reconstruction, commitment, and vector-quantization objectives.
- 3.2. Latent Diffusion: The latent diffusion model adds Gaussian noise progressively and trains a time-conditional 3D U-Net to predict the noise.At inference, denoising begins from standard normal noise and the decoder maps the result back to a 3D T-SDF.
- 3.3. Learning the Conditional Distribution: Task-specific encoders and cross-attention incorporate multiple conditional modalities into the diffusion process.Shape completion uses partial observations, image reconstruction uses CLIP, and text generation uses BERT.
- 3.3. Learning the Conditional Distribution: Classifier-free guidance assigns each modality a weight, with larger weights giving that modality greater influence on conditional generation.Dropout and zero-filled conditions support the classifier-free guidance mechanism.
- 3.4. 3D Shape Texturing with a 2D Model: The texturing pipeline converts a generated T-SDF into a density tensor, fixes that density, and learns color through a neural radiance field.Score distillation sampling supplies update directions using gradients from a 2D text-to-image diffusion model.
4. Experiments
SDFusion is evaluated across shape completion, single-view reconstruction, text-guided generation, multi-conditional generation, and 3D texturing. It outperforms prior methods on the evaluated single-condition tasks while supporting controllable combinations of modalities and textured outputs.
- Shape Completion: SDFusion outperforms prior methods on shape completion across fidelity and diversity, especially for high-resolution BuildingNet data.Completion is evaluated on ShapeNet and BuildingNet using UHD for fidelity and TMD for diversity.
- Single-view 3D Reconstruction: SDFusion outperforms other methods on both Chamfer Distance and F-score for Pix3D single-view reconstruction.The method also produces higher-quality shapes that are more visually consistent with image objects across camera poses.
- Text-guided Generation: SDFusion outperforms AutoSDF on text-guided generation with lower confusion and produces diverse shapes faithful to specific descriptions.Examples include “L-shaped table” and “table with two surfaces.”
- Multi-conditional Generation: SDFusion jointly conditions generation on partial shapes with text or images, producing diverse results consistent with the provided conditions.For ambiguous images, such as rear-view chairs, it can produce diverse predictions.
- Multi-conditional Generation: Weights control the relative importance of partial-shape, image, and text inputs, enabling fine-grained user control during generation.Increasing image weight makes outputs more image-like, while increasing text weight makes them more “egg-shaped” in the shown example.
- 3D Shape Texturing: A pretrained 2D diffusion model provides meaningful, diverse text guidance for texturing generated 3D shapes.The pipeline interprets style and material concepts and can produce diverse textures from one description.
5. Conclusion
SDFusion combines latent-space SDF diffusion with multimodal conditioning and a 2D-to-3D texturing application. The authors identify broader 3D representations, non-object-centric data, and stronger 2D–3D integration as future directions.
- SDFusion encodes 3D shapes into a low-dimensional latent space for diffusion and uses class-specific encoders with cross-attention for multimodal conditions.Classifier-free guidance enables control over the relative weights of different modalities.
- A pretrained 2D text-to-image model is used to texture generated 3D shapes.
- The current model is trained on high-quality SDFs and object-centric data, motivating support for multiple 3D representations and entire 3D scenes.
- Further work should explore combining models trained on 2D and 3D data more extensively.