Source-linked AI summary
MeshDiffusion: Score-based Generative 3D Mesh Modeling
Zhen Liu, Yao Feng, Michael J. Black, Derek Nowrouzezahrai, Liam Paull, Weiyang Liu
TL;DR
Generating detailed 3D meshes is difficult because prior representations and post-processing can lose surface detail or introduce artifacts. MeshDiffusion directly models meshes with deformable tetrahedral grids using diffusion, and reports high-quality generation across multiple tasks, while relying on a two-stage setup and limited-resolution architecture.
Problem
Existing scalable mesh-generation approaches rely on sub-optimal post-processing and can produce overly smooth or noisy surfaces lacking fine-grained geometric details.
Method
MeshDiffusion represents meshes with deformable tetrahedral grids and trains a diffusion model using a 3D CNN-based U-Net.
Results
MeshDiffusion produces high-quality meshes, preserves sharp geometric details, supports conditional generation with differentiable rendering, and performs across unconditional generation, conditional generation, and interpolation.
Takeaways & Limitations
Direct diffusion modeling of tetrahedral-grid meshes can generate fine details for shapes with arbitrary topology and can use 2.5D information for mesh completion.
Takeaways & Limitations
The method uses a reconstruction-then-generation pipeline, does not fully incorporate differentiable rendering into diffusion training and inference, and its naïve architecture limits input-grid resolution.
Abstract
from arXiv · showhide
We consider the task of generating realistic 3D shapes, which is useful for a variety of applications such as automatic scene generation and physical simulation. Compared to other 3D representations like voxels and point clouds, meshes are more desirable in practice, because (1) they enable easy and arbitrary manipulation of shapes for relighting and simulation, and (2) they can fully leverage the power of modern graphics pipelines which are mostly optimized for meshes. Previous scalable methods for generating meshes typically rely on sub-optimal post-processing, and they tend to produce overly-smooth or noisy surfaces without fine-grained geometric details. To overcome these shortcomings, we take advantage of the graph structure of meshes and use a simple yet very effective generative modeling method to generate 3D meshes. Specifically, we represent meshes with deformable tetrahedral grids, and then train a diffusion model on this direct parametrization. We demonstrate the effectiveness of our model on multiple generative tasks.
1 INTRODUCTION
MeshDiffusion targets high-quality 3D mesh generation by directly modeling meshes with deformable tetrahedral grids and diffusion models. The approach addresses limitations of voxel, point-cloud, and implicit representations while supporting multiple generative tasks.
- Motivation: Voxel and point-cloud methods lack fine-level surface geometry and require reconstruction for practical mesh-based applications.They are also relatively hard for artists to edit, while reconstruction can introduce noise and topological artifacts.
- Motivation: SDF-based methods can extract meshes but often require careful training and may produce overly smooth shapes that lose sharp edges and semantic details.Their MLP parameterization embeds a smoothness prior and can make them costly to render.
- Motivation: Meshes preserve local geometric information through vertices and connectivity, making them suitable for graphics pipelines and geometry-aware modeling.Surface normals can be derived locally from mesh vertices, retaining rich geometric details.
- Approach: MeshDiffusion trains a diffusion model on meshes parameterized by a discretized, uniform tetrahedral grid.The grid constrains topology and spatial configuration to a small but representative family of meshes.
- Contributions: The model generates high-quality meshes, supports conditional generation with a differentiable renderer, and is validated qualitatively, quantitatively, and through ablations.The reported tasks include unconditional generation, conditional generation, and interpolation.
2 RELATED WORK
Prior 3D shape-generation methods commonly use voxels, point clouds, implicit fields, or known-topology mesh templates. These approaches trade processing convenience against explicit surface quality, rendering efficiency, or topology flexibility.
- 3D Shape Generation: Voxel and point-cloud generators are accessible but do not explicitly encode surfaces, requiring reconstruction before relighting or simulation.The resulting representations are often noisy and require additional processing.
- 3D Shape Generation: Implicit neural-field generators can learn shapes from 2D images with differentiable rendering but usually require post-processing to extract explicit meshes.They can also be more time-consuming for image rendering.
- Mesh Reconstruction: Direct mesh generation is straightforward when topology is known, but generic mesh reconstruction remains challenging because object structures can be highly complex.Template deformation is especially suited to settings such as human faces with known topology.
- Score-based Generative Models: Score-based models learn the gradient of the log probability rather than modeling probability distributions directly.This avoids the often intractable normalization constant and enables score-matching training.
3 PRELIMINARIES
MeshDiffusion combines deformable tetrahedral-grid mesh parameterization with score-based diffusion modeling. DMTet defines mesh geometry through interpolated SDF values, while diffusion learns scores through a noisy forward process and reverse-time sampling.
- DMTet: DMTet represents arbitrary-topology meshes with a deformable tetrahedral grid whose vertices carry SDF values.Marching tetrahedra extracts triangular zero-surfaces from the grid.
- DMTet: Within each tetrahedron, SDF values are interpolated using barycentric coordinates, and zero crossings define mesh vertices on tetrahedron edges.The resulting mesh vertex position is computed by linear interpolation of edge endpoints and their SDF values.
- Diffusion Models: Diffusion models define a time-dependent forward process that moves data toward a target distribution and learn the corresponding score function.The score is approximated by a neural network conditioned on the noisy sample and time.
- Diffusion Models: Training minimizes a weighted squared error between the predicted and ground-truth scores across diffusion times and data samples.The weighting function λ(t) is part of the model design.
- Sampling: Generated samples are obtained by solving the reverse SDE from an initial target-distribution sample using numerical solvers.DDPM is a discrete diffusion formulation with Gaussian transitions and a noise schedule β_t.
4 MeshDiffusion: DIFFUSION MODEL ON MESHES
MeshDiffusion uses a uniform deformable tetrahedral-grid representation, a 3D CNN score model, and a two-stage reconstruction-then-generation pipeline. SDF normalization reduces mesh artifacts caused by denoising noise, while the same framework supports conditional completion.
- Representation: BCC-initialized tetrahedral grids provide a uniform, nearly regular representation with preserved 3D translational symmetry.Vertex deformations are treated as attributes, enabling diffusion modeling on the grid.
- 3D Convolutional Networks: MeshDiffusion converts tetrahedral grids into voxel-like tensors and uses a 3D U-Net instead of a graph neural network.The choice is motivated by CNN capacity, contextual information, and embedded spatial priors.
- Training Objective: The training pipeline first fits tetrahedral grids to rendered RGBD observations, then trains a diffusion model on the fitted grids.The fitting objective combines image, depth, Chamfer, and penalty losses.
- Reducing Noise Effects: Naïve denoising can create non-smooth surfaces and topology changes because equal input-space errors can produce unequal mesh-vertex and topological errors.Small SDF perturbations can produce undesirable holes under marching tetrahedra.
- Reducing Noise Effects: The method normalizes SDF values to ±1 through a two-pass optimization scheme and treats them as floats during standard Gaussian diffusion training.Inference rounds generated SDFs to ±1 and can optionally refine deformation vectors conditionally.
- Conditional Generation: Conditional generation fits a tetrahedral grid to a single RGBD view, masks occluded regions, and uses diffusion to complete the missing geometry.The process can optionally refine originally unmasked vertices near completion end.
5 EXPERIMENTS AND RESULTS
Experiments evaluate MeshDiffusion across unconditional and conditional mesh generation, using qualitative comparisons, shape metrics, rendered-view FIDs, ablations, and texture transfer. The results indicate sharp geometric detail, competitive proxy scores, plausible single-view completion, and useful compatibility with texture generation.
- 5.2 UNCONDITIONAL GENERATION: MeshDiffusion produces the sharpest qualitative samples and preserves finer geometric details than the compared models.The paper attributes this partly to piecewise-linear interpolation, unlike the smoother interpolation assumed by SDF-based methods.
- 5.2.1 QUANTITATIVE EVALUATION: MeshDiffusion achieves better point-cloud metrics in most cases, indicating stronger capture of geometric details.Evaluation samples 2048-point clouds from generated meshes and ground-truth shapes, alongside light field distance.
- 5.2.1 QUANTITATIVE EVALUATION: MeshDiffusion obtains competitive rendered-view FID scores compared with SDF-StyleGAN and recent state-of-the-art methods.FID averages scores across 24 uniformly distributed camera poses, but the paper cautions that rendered-view FID is not a genuine 3D-shape score.
- 5.2.1 QUANTITATIVE EVALUATION: Ablations show that SDF normalization benefits diffusion modeling and that the customized diffusion model suits mesh generation better.These findings are reported for the Chair category in ShapeNet.
- 5.3 CONDITIONAL GENERATION: Conditioning on a single-view RGBD image yields plausible and reasonable mesh completion results.Near the end of diffusion, originally fixed tetrahedral vertices are allowed to update slightly to accommodate imperfect single-view geometry estimates.
- 5.5 TEXT-CONDITIONED TEXTURE GENERATION: Generated meshes can be textured with TEXTure to produce fairly realistic and reasonable textured 3D models.The paper suggests that more advanced texture-generation methods could better expose the benefits of the generated geometry.
6 DISCUSSIONS
The discussion identifies optimization and modeling limitations while outlining simpler diffusion designs and broader training constraints. It highlights unresolved issues in topology, rendering integration, resolution, and dataset construction.
- Optimization issues with DMTet: DMTet can fit geometries but struggles with complex topology, may create invisible holes, and can produce floating or isolated meshes.The authors call for better optimization, regularization, and mesh parametrizations.
- Diffusion model design: The current 3D-CNN U-Net with DDPM is effective with few architectural and hyperparameter changes.The authors note that more advanced diffusion models and memory-efficient architectures could be substituted.
- Limitations: The method uses a two-stage reconstruction-then-generation approach rather than amortizing the cost of differentiable rendering during diffusion training.The renderer is used during tetrahedral-grid creation, not throughout diffusion training and inference.
- Limitations: The naïve diffusion architecture limits tetrahedral-grid resolution, and resolution 64 cannot capture some fine details.The authors suggest better architectures or adaptive resolution as possible remedies.
7 CONCLUDING REMARKS
MeshDiffusion generates fine-detail 3D shapes with arbitrary topology using diffusion over tetrahedral-grid parameterizations. The authors position it as a basis for future work on high-fidelity shape, texture, motion, and simulation generation.
- 7 CONCLUDING REMARKS: MeshDiffusion generates fine details in 3D shapes with arbitrary topology through diffusion on tetrahedral-grid parameterizations.The conclusion describes the model as minimally designed while outperforming baselines.
- 7 CONCLUDING REMARKS: The model can be used when only 2.5D information is available, despite training and inference in 3D space.The conclusion presents this as a demonstrated capability.
- 7 CONCLUDING REMARKS: Future directions include text-conditioned mesh generation, joint texture-and-geometry synthesis, motion synthesis, and physical simulation.These directions are stated as future work rather than completed capabilities.
A.1 ARCHITECTURE AND LOSSES
The appendix describes reconstruction losses, tetrahedral-grid preprocessing, deformation and SDF handling, artifact removal, and the DDPM U-Net implementation. These choices combine image-, depth-, geometry-, and regularization-based supervision.
- Losses: RGB reconstruction uses averaged pixel-wise log-L1 loss over sampled camera poses and tetrahedral grids.The camera poses look toward the object center, and x is sampled from the tetrahedral-grid dataset.
- Losses: Depth supervision uses L2 loss beyond depth 1.0 and L1 loss otherwise, with a second rasterized layer weighted by 0.1.The background depth is set to 20 while object depth lies in [−1, 1].
- Losses: Chamfer loss samples 50,000 points from each ground-truth and predicted mesh at every iteration.SDF regularization penalizes differences between neighboring tetrahedral vertices to simplify occluded geometry and aid optimization.
- Grid and deformation: The tetrahedral grid starts from dense BCC tiling in [−1, 1]^3, while boundary vertices that break translational symmetry are removed.Deformation vectors are clipped after gradient updates, and larger deformation ranges help capture details at low resolution.
- Grid and artifact handling: Training includes SDF scaling, visual-hull culling, topology-aware depth handling, and removal of non-mesh-generating tetrahedral complexity.These procedures target convergence, floating artifacts, topological holes, and dataset complexity.
- Diffusion architecture: The diffusion model uses a DDPM adapted from prior work with a 3D U-Net, cubic-grid augmentation, masked artificial sites, and small random translations for augmentation.The U-Net uses base width 64; higher-resolution grids reduce layers and double the base width.
C QUANTITATIVE METRICS FOR 3D MESHES
The paper evaluates generated meshes with distributional, matching, coverage, classification, and silhouette-based metrics, while visualizing denoising trajectories, nearest neighbors, and comparisons with GET3D.
- Quantitative metrics: MMD measures average nearest-neighbor point-cloud distance, while Coverage measures how much of one set is represented by another.Coverage is interpreted through nearest-neighbor matching between point-cloud sets.
- Quantitative metrics: 1-NNA uses a leave-one-out 1-NN classifier, with lower scores indicating better coverage of one set by another.The metric assesses whether elements in one set are important for representing the other.
- Quantitative metrics: JSD measures distance between ground-truth and generated point-cloud distributions, while LFD compares shape silhouettes using light-field descriptors.LFD combines local-region and contour-based descriptors.
- Qualitative evaluation: Generation trajectories visualize predicted x0 during DDPM inference from T = 250 to T = 50 because earlier predictions are only noise.The predicted grids are clipped to [0, 1] for visualization.
- Qualitative evaluation: Figure 9 shows nearest neighbors of generated validation samples, and Figure 10 provides a qualitative comparison between MeshDiffusion and GET3D.The cited passages identify the visualizations but do not specify additional figure encodings.
- Qualitative evaluation: On ShapeNet cars, GET3D tends to have fewer noticeable holes, whereas MeshDiffusion shows fewer failure cases and more plausible novel examples.The authors attribute these observations possibly to differences between adversarial image training and supervised denoising.
G INNER STRUCTURE GENERATION
MeshDiffusion generates car meshes with inner structures invisible from outside, and compares them against GET3D using rendered inner-structure examples and surface point-cloud metrics.
- G INNER STRUCTURE GENERATION: MeshDiffusion generates inner structures invisible from outside because it is explicitly trained with 3D information.The comparison uses MeshDiffusion at resolution 128 and GET3D samples.
- G INNER STRUCTURE GENERATION: Figure 11 compares the inner structures of generated car meshes from GET3D and MeshDiffusion.
- G INNER STRUCTURE GENERATION: Table 5 reports shape metrics on surface point clouds from generated car meshes for MeshDiffusion and GET3D.The evaluation follows GET3D’s procedure and uses a generated sample set five times larger than the validation set.
H MORE UNCONDITIONAL GENERATION SAMPLES
The paper presents additional unconditional MeshDiffusion samples across chairs, cars, airplanes, rifles, and tables.
- H MORE UNCONDITIONAL GENERATION SAMPLES: Additional unconditional samples include generated 3D chair meshes.
- H MORE UNCONDITIONAL GENERATION SAMPLES: Additional unconditional samples include generated 3D car meshes.
- H MORE UNCONDITIONAL GENERATION SAMPLES: Additional unconditional samples include generated 3D airplane meshes.
- H MORE UNCONDITIONAL GENERATION SAMPLES: Additional unconditional samples include generated 3D rifle meshes.
- H MORE UNCONDITIONAL GENERATION SAMPLES: Additional unconditional samples include generated 3D table meshes.