Source-linked AI summary
Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis
Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu, Sanja Fidler
TL;DR
High-resolution 3D content creation is difficult, especially when representations must preserve detail, topology, and efficiency. DMTET combines a deformable tetrahedral-grid SDF with differentiable surface extraction and surface-based training, achieving higher-quality synthesis and reconstruction than existing methods.
Problem
Existing 3D representations struggle to combine detailed geometry, arbitrary topology, and efficient high-resolution synthesis from simple user guides.
Method
DMTET combines a deformable tetrahedral-grid SDF, differentiable Marching Tetrahedra, adaptive subdivision, and losses defined on the extracted surface.
Results
DMTET significantly outperforms existing methods on coarse-voxel synthesis and point-cloud reconstruction, while running 10 times faster than alternative implicit methods at inference.
Takeaways & Limitations
The hybrid representation supports high-quality shapes with finer geometric details and arbitrary topology from coarse voxel guides.
Abstract
from arXiv · showhide
We introduce DMTet, a deep 3D conditional generative model that can synthesize high-resolution 3D shapes using simple user guides such as coarse voxels. It marries the merits of implicit and explicit 3D representations by leveraging a novel hybrid 3D representation. Compared to the current implicit approaches, which are trained to regress the signed distance values, DMTet directly optimizes for the reconstructed surface, which enables us to synthesize finer geometric details with fewer artifacts. Unlike deep 3D generative models that directly generate explicit representations such as meshes, our model can synthesize shapes with arbitrary topology. The core of DMTet includes a deformable tetrahedral grid that encodes a discretized signed distance function and a differentiable marching tetrahedra layer that converts the implicit signed distance representation to the explicit surface mesh representation. This combination allows joint optimization of the surface geometry and topology as well as generation of the hierarchy of subdivisions using reconstruction and adversarial losses defined explicitly on the surface mesh. Our approach significantly outperforms existing work on conditional shape synthesis from coarse voxel inputs, trained on a dataset of complex 3D animal shapes. Project page: https://nv-tlabs.github.io/DMTet/.
1 Introduction
DMTET targets high-resolution 3D shape creation from simple voxel guides by combining implicit shape representation with explicit surface supervision. It addresses the need for detailed geometry, arbitrary topology, and efficient computation, outperforming prior methods on challenging tasks.
- High-quality 3D content requires rich geometric details and complex topology, but creating individual assets demands substantial expert effort and development time.
- Effective 3D representations should capture local details, support arbitrary topology, and remain memory- and computation-efficient for interactive inference.
- Neural implicit methods represent complex geometry and topology at arbitrary resolution but commonly train on SDF or occupancy values rather than explicit surface supervision.
- DMTET combines a deformable tetrahedral-grid implicit function with differentiable Marching Tetrahedra and surface supervision for high-resolution synthesis from coarse voxels.
- DMTET demonstrates better reconstruction quality than state-of-the-art methods on two 3D tasks while requiring lower computation cost and achieving 10 times faster inference than alternative implicit methods.
2 Related Work
Prior 3D synthesis methods use voxels, deep implicit fields, or directly generated surfaces, each imposing trade-offs in resolution, efficiency, artifacts, mesh extraction, or topology.
- Voxel-based Methods: Voxel methods support convolutional processing and strong reconstruction results, but their computational and memory costs grow cubically with resolution.
- Deep Implicit Fields: Deep implicit fields represent shapes as continuous-function zero level sets with arbitrary topology and infinite resolution.
- DMTET is positioned between these families by combining implicit representation with explicit surface processing.
- Deep Implicit Fields: Deep implicit methods may render fine-detail artifacts because they supervise sampled function values, and extracting meshes requires an expensive iso-surfacing step.
- Surface-based Methods: Surface-based methods directly predict meshes but typically predefine topology, limiting their ability to model distributions with complex topology variations.
3 Deep Marching Tetrahedra
DMTET is a hybrid 3D representation and network designed for high-quality object synthesis from inputs such as coarse voxels.
- DMTET uses a hybrid 3D representation for high-resolution reconstruction and synthesis, with network architecture and training objectives described separately.
3.1 3D Representation
DMTET encodes shapes with a deformable tetrahedral-grid SDF, selectively refines regions near the predicted surface, converts the result into a mesh with differentiable Marching Tetrahedra, and further subdivides that surface.
- 3.1.1 Deformable Tetrahedral Mesh: The representation encodes signed distance values on a deformable tetrahedral grid, whose vertices can deform to represent geometry efficiently.
- 3.1.1 Deformable Tetrahedral Mesh: SDF values inside each tetrahedron are obtained by barycentric interpolation of the values at its four vertices.
- 3.1.2 Volume Subdivision: Surface tetrahedra and their immediate neighbors are selectively subdivided by adding edge midpoints, with new SDF values averaged from the edge endpoints.
- 3.1.3 Marching Tetrahedra: Marching Tetrahedra identifies surface topology from vertex SDF signs and computes iso-surface vertices at zero crossings along sign-changing edges.
- 3.1.3 Marching Tetrahedra: The extracted-surface loss can back-propagate to both grid vertex positions and SDF values because the singular case is not evaluated when edge signs differ.
- 3.1.4 Surface Subdivision: A differentiable surface subdivision module increases representation power and visual quality using learnable parameters based on the Loop Subdivision scheme.
3.2 DMTET: 3D Deep Conditional Generative Model
DMTET generates high-resolution meshes from point clouds or coarse voxelized shapes using a deformable tetrahedral SDF representation and staged surface refinement. Its generator combines MLP-based initial predictions, GCN refinement, adaptive subdivision, learnable mesh subdivision, and a surface-based discriminator.
- DMTET maps an input point cloud or coarse voxelized shape x to a high-resolution 3D mesh M.
- An MLP predicts initial SDF values and per-vertex features, while a GCN refines vertices near the current surface.
- Surface refinement predicts vertex-position offsets and SDF residuals, allowing local topology changes through sign flips and improving local geometry by moving vertices.
- DMTET iteratively subdivides surface tetrahedra and their neighbors, retaining surface-proportional computation that scales quadratically rather than cubically with resolution.
- After marching tetrahedra extraction, a GCN performs learnable surface subdivision to reduce quantization and Loop Subdivision approximation errors.
- A 3D discriminator evaluates signed-distance fields around randomly selected high-curvature target vertices, with gradients back-propagated to predicted mesh vertices.
3.3 Loss Function
DMTET is trained end-to-end with losses defined on the final predicted mesh, combining surface alignment, adversarial realism, and regularization. The regularizers constrain SDF values and vertex deformations to address disconnected components and artifacts.
- DMTET supervises all modules through errors defined on the final predicted mesh in an end-to-end training procedure.
- Surface alignment combines L2 Chamfer distance and normal consistency between sampled ground-truth and predicted mesh points.
- The adversarial loss uses LSGAN to improve the realism of generated shapes.
- Because surface losses mainly affect near-iso-surface vertices and do not specify inside-versus-outside signs, training can produce disconnected components.
- An SDF regularization loss addresses sign-related ambiguity, while L2 regularization on vertex deformations helps avoid artifacts.
- The final objective is a weighted sum of five terms: Chamfer, normal-consistency, adversarial, SDF, and deformation losses.
4 Experiments
DMTET is evaluated on coarse-voxel shape synthesis and noisy point-cloud reconstruction, achieving strong qualitative and quantitative results across both tasks. Ablations and component analyses examine subdivision modules, adversarial training, deformable tetrahedral grids, and marching tetrahedra.
- 3D Shape Synthesis from Coarse Voxels: DMTET significantly outperforms all baselines across every reported metric for animal-shape super-resolution from coarse voxels.The evaluation uses L2 and L1 Chamfer Distance, normal consistency, Light Field Distance, and Cls score.
- 3D Shape Synthesis from Coarse Voxels: Adversarial loss adds geometric details such as nails, ears, eyes, and mouths, improving the realism and visual quality of generated shapes.Human judges also preferred DMTET's details over all baselines in most comparisons.
- 3D Shape Synthesis from Coarse Voxels: DMTET faithfully generates high-quality details from human-created coarse voxels with body-part proportions that differ from its training shapes.These inputs were collected from Turbosquid and represent shapes unseen during training.
- Ablation Studies: Volume subdivision beats the base model in 78% of better-looking and 61% of realistic-detail cases, while surface subdivision adds 62% wins on both criteria.The ablation introduces volume subdivision first and surface subdivision second.
- Point Cloud 3D Reconstruction: For noisy point-cloud reconstruction, DMTET improves local geometry, supports different topologies, produces fewer artifacts, and runs significantly faster than neural implicit methods.Its advantages are attributed to efficient marching tetrahedra training, joint topology and geometry optimization, vertex deformations, and surface loss.
- Component Analysis: Marching tetrahedra consistently outperforms marching cubes when querying the same number of points, while tetrahedral staggered grids better capture thin structures at limited resolution.Deformable-grid DMTET also exceeds oracle marching-tetrahedra performance, indicating that surface optimization mitigates discretization errors.
5 Conclusion
DMTet synthesizes high-resolution 3D shapes from simple guides by combining implicit and explicit representations. Experiments show higher-quality geometry than existing methods, supporting broader access to 3D content creation.
- DMTet synthesizes high-resolution 3D shapes using simple user guides such as coarse voxels.
- Its representation combines the advantages of implicit and explicit 3D representations.
- DMTet produces significantly higher-quality shapes with better geometric details than existing methods.This is supported by quantitative metrics and an extensive user study.
- Upscaling coarse voxel shapes such as Minecraft creations may help democratize 3D content creation.
6 Broad Impact
The work targets the gap between expert-created high-quality 3D content and the coarse shapes that novices can readily build. It aims to help novice users upscale low-resolution shapes into high-resolution, beautiful assets.
- AR/VR, robotics, architecture, gaming, and film rely on high-quality 3D content.
- Creating such content requires experienced artists and significant development time.
- Platforms like Minecraft let millions of users create coarse shapes with simple blocks.
- The work aims to enable novice users to upscale simple, low-resolution shapes into high-resolution, beautiful shapes.
7 Disclosure of Funding
The paper reports funding from NVIDIA and additional scholarship revenue for two authors from the University of Toronto and the Vector Institute.
- This work was funded by NVIDIA.
- Tianchang Shen and Jun Gao received additional student scholarships from the University of Toronto and the Vector Institute.The disclosure states these scholarships were not in direct support of this work.