Source-linked AI summary
Diffusion-SDF: Conditional Generative Modeling of Signed Distance Functions
Gene Chou, Yuval Bahat, Felix Heide
TL;DR
High-quality 3D generation remains challenging for diffusion models, especially when reconstructing shapes from partial or noisy observations. Diffusion-SDF represents geometry with neural SDFs and diffuses their modulated latent representations with geometric conditioning, achieving realistic unconditional generation and conditional reconstruction across several input types. The paper reports favorable generation and completion performance while noting that existing conditional SDF methods were limited to full-view, low-noise point clouds.
Problem
Diffusion models remain in the early stages of generating complex 3D shapes, while real reconstruction often requires plausible outputs from partial and noisy observations.
Method
Diffusion-SDF uses neural SDFs as a unified 3D representation and modulates them into latent vectors for diffusion with geometric conditioning.
Results
Diffusion-SDF achieves favorable performance in shape generation and completion and supports realistic unconditional generation plus conditional reconstruction from partial inputs.
Takeaways & Limitations
The method applies diffusion to 3D implicit representations across shape completion, single-view reconstruction, and real-scanned point-cloud reconstruction.
Takeaways & Limitations
Prior conditional SDF methods addressed full-view point clouds with low noise, leaving partial and noisy real-world observations insufficiently covered.
Abstract
from arXiv · showhide
Probabilistic diffusion models have achieved state-of-the-art results for image synthesis, inpainting, and text-to-image tasks. However, they are still in the early stages of generating complex 3D shapes. This work proposes Diffusion-SDF, a generative model for shape completion, single-view reconstruction, and reconstruction of real-scanned point clouds. We use neural signed distance functions (SDFs) as our 3D representation to parameterize the geometry of various signals (e.g., point clouds, 2D images) through neural networks. Neural SDFs are implicit functions and diffusing them amounts to learning the reversal of their neural network weights, which we solve using a custom modulation module. Extensive experiments show that our method is capable of both realistic unconditional generation and conditional generation from partial inputs. This work expands the domain of diffusion models from learning 2D, explicit representations, to 3D, implicit representations.
1. Introduction
Diffusion-SDF extends probabilistic diffusion modeling to 3D implicit geometry, targeting generation and reconstruction from diverse imperfect observations. It represents shapes with neural SDFs and uses modulation to make SDF diffusion practical while preserving geometric conditioning.
- Motivation: 3D reconstruction from imperfect observations is probabilistic because partial visual evidence can correspond to multiple plausible shapes.The setting includes occlusion and camera measurement errors in applications such as self-driving and robotic grasping.
- Representation: Diffusion-SDF uses neural signed distance functions to represent surfaces described by point clouds and 2D images.SDFs support arbitrary test-time resolution, small memory footprints, and strong generalization.
- Applications: Diffusion-SDF unifies shape completion, single-view reconstruction, and real-scanned point-cloud reconstruction as conditional generation tasks.The method conditions training on multiple input types and samples diverse completions from the learned distribution.
- Method: The method diffuses implicit neural functions by learning to reverse their network weights while adding geometric constraints through a modulation module.The modulation module compresses SDFs into latent vectors that map to individual SDFs with a base network.
- Results: The authors report favorable performance in shape generation and completion compared with existing methods, alongside clean and diverse 3D mesh generation.The introduction presents this as the outcome of extensive experiments across the proposed generation and reconstruction settings.
2. Related Work
Prior 3D generative methods include point-cloud diffusion, probabilistic completion models, and implicit-function approaches, but existing systems often lack complex geometry or robust conditional reconstruction. Diffusion-SDF addresses these gaps with SDF-based conditioning and modulation.
- Diffusion models: Diffusion models for 3D tasks remain relatively early in producing complex shapes, despite high-quality image-generation results.Existing 3D approaches include methods that diffuse point clouds and methods that diffuse implicit neural representations.
- Shape generation: Many partial-shape reconstruction methods are deterministic even though partial inputs can correspond to multiple completed shapes.Prior probabilistic approaches use autoregressive, GAN, vector-quantized implicit, and diffusion models for multimodal reconstruction.
- SDF generative modeling: Concurrent SDF-generative approaches either use differentiable rendering and 2D GANs or convert SDFs into wavelet representations for diffusion.The cited wavelet-based method cannot perform conditional generation and reverses wavelets as a surrogate for implicit functions.
- Conditional reconstruction: Earlier conditional SDF methods generally use full-view, low-noise point clouds and therefore do not address partial and noisy real-world observations.Diffusion-SDF targets plausible reconstructions from partial and noisy point clouds.
3. Diffusion Models and Neural SDFs
The paper describes canonical diffusion as progressive noise addition followed by learned reverse denoising, then defines neural SDFs as coordinate-based functions whose zero-level sets form surfaces. These foundations support diffusion over learned SDF representations.
- Diffusion process: The forward diffusion process repeatedly adds Gaussian noise to data until the result approximates an isotropic Gaussian.The process is Markovian and uses a variance schedule β_t.
- Diffusion process: A closed-form parameterization samples a noisy state from the clean data using ᾱ_t and Gaussian noise ε.This avoids explicitly simulating every intermediate forward step during training.
- Reverse diffusion: The learned reverse process predicts denoising transitions that invert the forward diffusion from noisy states toward data samples.The model can predict the reverse mean or equivalently the injected noise; this work predicts the reverse mean for SDF generation.
- Sampling: Generation starts from Gaussian noise and iteratively denoises it using ancestral sampling.Each reverse step combines the learned mean with a fixed-noise term.
- Neural SDFs: A neural SDF maps spatial coordinates and a conditioning point cloud to signed distance values, with the surface represented by its zero-level set.The formulation supports coordinate-based decoding conditioned on point-cloud features.
4. Diffusing Neural Signed Distance Functions
Diffusion-SDF represents SDFs with regularized latent vectors, diffuses those vectors, and decodes generated outputs into complete SDFs. Conditioning, cross-attention, and end-to-end SDF losses support reconstructions consistent with varied inputs.
- Modulating SDFs: The pipeline jointly trains an SDF representation and VAE to compress each SDF into a latent vector z.The latent vectors are later used as diffusion-model data points and can be decoded back into SDF representations.
- Modulating SDFs: The modulation module maps SDFs to regularized one-dimensional latent vectors because directly diffusing thousands of SDF parameters is impractical.The VAE regularizes its posterior toward a zero-mean Gaussian prior with standard deviation 0.25, matching the distribution toward which diffusion processes converge.
- Modulating SDFs: Empirically, modulation represents substantially more complex and diverse geometries than existing implicit-function methods.
- Diffusing Modulation Vectors: The diffusion model adds Gaussian noise to latent vectors and learns to reconstruct the original denoised vector z0 from noisy inputs and timestep embeddings.At test time, it iteratively denoises a Gaussian sample and passes the resulting latent vector into the joint SDF-VAE model for marching-cubes reconstruction.
- Conditioning Mechanisms: Conditional generation encodes point clouds or images into shape features and injects them through cross-attention into the diffusion model.The model can guide reconstructions from partial point clouds, real-scanned point clouds, and 2D images; zero-masking the condition with probability 80% increases diversity and reduces overfitting.
- End-to-End Training for Geometry Constraints: End-to-end training feeds VAE outputs into the diffusion model and adds an SDF loss on denoised latent vectors to impose geometric constraints.The authors state that connecting the modules introduces explicit SDF information into diffusion training and allows more complex geometries.
5. Experiments
Experiments evaluate Diffusion-SDF on unconditional generation, sparse partial-point-cloud completion, additional conditioning modalities, scalability, latent interpolation, and ablations. Results show clean and diverse generation, strong completion quality and diversity, conditioning across scanned point clouds and images, graceful scaling, and a continuous latent space.
- Experimental setup: The experiments cover unconditional generation, sparse partial-point-cloud completion, scanned point-cloud and image conditioning, scalability, interpolation, and ablations.Unconditional generation uses Acronym, partial completion uses cropped point clouds, single-view reconstruction uses rendered images, and scanned reconstruction uses YCB point clouds.
- Unconditional generation: Diffusion-SDF produces clean meshes with thin structures and diverse geometries in unconditional generation.The evaluation uses MMD, COV, and 1-NNA; the authors also compute average Chamfer distance to training objects to assess uniqueness.
- Conditional generation: Reconstructions preserve conditioned geometry across sparse point clouds, real scans, and single images, including chair-leg configurations, drill curves, plane engines, and couch pillows.The unified conditional formulation samples 3D shapes given a condition and is trained with noisy or incomplete scanned point clouds and image encodings.
- Shape completion: The method outperforms all baselines in MMD and TMD for sparse partial-point-cloud completion, but not UHD.MMD measures quality, TMD diversity, and UHD fidelity to the partial input; the setup uses 128 points with 50% cropped.
- Latent interpolation: Linear interpolation between latent vectors produces gradual semantic shape changes, supporting a continuous latent space whose variables control geometry.Interpolated vectors are decoded and meshed, with headboard changes cited as an example.
- Ablation experiments: The ablation reports that skipping fine-tuning yields clean, realistic generations but reduces diversity and complexity, while guidance strength trades diversity against fidelity.The authors also report similar outputs from concatenation and cross-attention, but higher memory consumption for concatenation.
6. Conclusion
Diffusion-SDF is a probabilistic model for generating diverse shapes from learned SDF distributions and for completing shapes across multiple input modalities. The authors identify faster inference, better latent relationships, broader conditioning, appearance, and scene synthesis as future directions.
- Diffusion-SDF generates diverse shapes from a distribution of learned SDFs.
- The method is assessed for shape generation and completion from various input modalities.
- Future work includes faster diffusion inference, improved latent relationships for interpretability and fidelity, alternative conditioning such as text-to-shape, appearance modeling, and full scene synthesis.