Source-linked AI summary
Learning Deformable Tetrahedral Meshes for 3D Reconstruction
Jun Gao, Wenzheng Chen, Tommy Xiang, Clement Fuji Tsang, Alec Jacobson, Morgan McGuire, Sanja Fidler
TL;DR
Learning-based 3D reconstruction lacks a representation that balances geometric fidelity, volumetric modeling, and learnability across varied inputs. DefTet deforms tetrahedral vertices and predicts occupancy, and the paper reports comparable or higher reconstruction quality while running significantly faster than similar-quality existing work.
Problem
Learning-based 3D reconstruction uses point, voxel, surface-mesh, and implicit-function representations with different limitations in resolution, topology, and producing meshes.
Source: §1 · p. 1Method
DefTet represents shapes in a tetrahedral volume and learns vertex positions and tetrahedron occupancy with differentiable reconstruction losses.
Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 4Results
DefTet achieves comparable or higher reconstruction quality while running significantly faster than existing work with similar quality.
Source: §6 · p. 9Takeaways & Limitations
DefTet directly produces tetrahedral meshes and supports arbitrary topology without inference-time post-processing.
Source: §1 · p. 2Neural prediction does not guarantee nonflipping tetrahedra; regularization losses only partially address this limitation.
Source: §5.4 · p. 9
Abstract
from arXiv · showhide
3D shape representations that accommodate learning-based 3D reconstruction are an open problem in machine learning and computer graphics. Previous work on neural 3D reconstruction demonstrated benefits, but also limitations, of point cloud, voxel, surface mesh, and implicit function representations. We introduce Deformable Tetrahedral Meshes (DefTet) as a particular parameterization that utilizes volumetric tetrahedral meshes for the reconstruction problem. Unlike existing volumetric approaches, DefTet optimizes for both vertex placement and occupancy, and is differentiable with respect to standard 3D reconstruction loss functions. It is thus simultaneously high-precision, volumetric, and amenable to learning-based neural architectures. We show that it can represent arbitrary, complex topology, is both memory and computationally efficient, and can produce high-fidelity reconstructions with a significantly smaller grid size than alternative volumetric approaches. The predicted surfaces are also inherently defined as tetrahedral meshes, thus do not require post-processing. We demonstrate that DefTet matches or exceeds both the quality of the previous best approaches and the performance of the fastest ones. Our approach obtains high-quality tetrahedral meshes computed directly from noisy point clouds, and is the first to showcase high-quality 3D tet-mesh results using only a single image as input. Our project webpage: https://nv-tlabs.github.io/DefTet/
1 Introduction
Learning-based 3D reconstruction must handle imperfect inputs and ambiguous or occluded regions, but existing representations make different trade-offs in resolution, topology, and mesh output. DefTet uses a deformable tetrahedral volume with learned vertex positions and occupancy to produce detailed shapes across varied inputs and topologies.
1 Introduction
- Voxel grids support 3D CNNs and varying topology, but high-resolution reconstruction is memory-intensive, while discontinuous octree structures are cumbersome for learning-based methods.High-quality voxel reconstruction depends on grid resolution.Source: §1 · p. 1
- The neural network takes a point cloud or image as input and deforms an initial tetrahedral mesh while predicting tetrahedron occupancy.Source: Figure 1 · p. 2
- Implicit-function and point-cloud methods need post-processing to produce meshes, while surface-mesh methods typically require a predefined genus that limits the objects they can handle.Source: §1 · p. 2
- DefTet deforms tetrahedral vertices and predicts each tetrahedron’s occupancy, allowing the volume to adapt to geometry while representing arbitrary topology.The approach is differentiable with respect to standard reconstruction losses and optimizes both vertex placement and occupancy.Source: §1 · p. 2
- Its volumetric representation captures local surface detail at lower memory cost than existing volumetric approaches, and directly produces tetrahedral meshes without inference-time post-processing.The paper also demonstrates reconstruction from noisy point clouds and images, including single-image input.Source: §1 · p. 2
2 Related Works
Prior learning-based reconstruction methods trade off resolution, topology, and mesh structure, while established tetrahedral-meshing methods rely on discrete operations that impede end-to-end learning.
2.1 3D Reconstruction
- Voxel reconstruction trades memory for fidelity at higher grid resolutions; deforming a tetrahedral grid allows explicit vertex placement and more flexible geometry.The authors hypothesize that hierarchical structure could also benefit their representation, leaving it for future work.Source: §2.1 · p. 2
- Mesh methods can preserve well-formed surfaces but struggle with varying topology; alternatives may produce holes or degenerate meshes, while DEFTET jointly optimizes occupancy and deformation.Differentiable Marching Cubes constrains vertices to regular-grid edges, whereas DEFTET permits 3D vertex deformation subject to avoiding flipped tetrahedra.Source: §2.1 · p. 3
2.2 Tetrahedral Meshing
- Existing tetrahedral-meshing methods use discrete, non-differentiable operations, making them unsuitable for end-to-end deep learning.The passage describes lattice-based methods and Delaunay-based approaches including QuarTet, TetGen, and TetWild.Source: §2.2 · p. 3
3 Formulation
DefTet represents shapes in a tetrahedralized bounded space, predicting vertex offsets and tetrahedron occupancy to form surfaces. Its training objectives support both 3D reconstruction supervision and differentiable rendering from 2D images.
3.1 Tetrahedron Parameterization
- DefTet predicts vertex offsets and tetrahedron occupancy; faces between tetrahedra with different occupancy define the object surface.The network takes an image or point cloud as input and predicts the offsets and occupancies.Source: §3.1 · p. 3, §3.2 · p. 4
- A fixed tetrahedralization of a bounded unit cube supports arbitrary topology, while vertex deformation accommodates finer geometry with less aliasing.Each tetrahedron represents a subvolume, and its triangular faces can represent surface segments.Source: §3.1 · pp. 3–4
3.2 DEFTET
- At inference, occupancy is thresholded, and faces shared by two occupied tetrahedra are discarded to extract the surface mesh.Source: §3.2 · p. 4
- Training combines reconstruction and geometric-regularization losses, including terms for deformation, volume, Laplacian smoothness, and tetrahedron distortion.Volume and AMIPS losses help avoid flipped tetrahedra.Source: §3.2 · pp. 4–5
3.2.1 Reconstruction Loss with 3D Ground-Truth
- With 3D supervision, occupancy uses binary cross-entropy against winding-number labels assigned from each tetrahedron’s centroid.Occupancy changes with the tetrahedron’s vertex positions as it deforms.Source: §3.2.1 · p. 5
- The 3D deformation loss fits predicted surface faces to target samples and target samples to their nearest predicted faces, and is differentiable with respect to occupancy and vertex positions.Source: §3.2.1 · p. 5
3.2.2 Reconstruction Loss via Differentiable Rendering
- For 2D supervision, a differentiable renderer projects tetrahedral-mesh geometry and per-vertex colors into an image using ray-hit faces and visibility.Face attributes are interpolated, and visibility depends on each face and the faces in front of it.Source: §3.2.2 · p. 5
- Rendered color and visibility are combined per pixel for image-loss supervision, with gradients back-propagated to vertex positions and colors.Tetrahedron occupancy is the maximum of its four vertex visibility values; the mask is optional when available.Source: §3.2.2 · p. 6
4 Applications
DefTet is applied to point-cloud and single-image reconstruction, tetrahedral meshing, and multi-view scene reconstruction. The applications use task-specific neural architectures, and the point-cloud comparison reports similar accuracy to OccNet at substantially higher speed.
4 Applications
- For some applications, a differentiable Laplace-smoothing layer improves visual results while retaining the same quantitative accuracy.The layer is added after the network output, with gradients propagated through it during training.Source: §4 · p. 6
4.1 Point Cloud 3D Reconstruction
- Point-cloud reconstruction uses PointVoxel CNN features and predicts vertex offsets and tetrahedron occupancy.Vertex features are interpolated at vertex positions, and occupancy features at tetrahedron centroids.Source: §4.1 · p. 6
4.2 Single Image 3D Reconstruction
- Single-image reconstruction adopts the DVR architecture and can be trained with either 3D or 2D supervision.Source: §4.2 · p. 6
4.3 Tetrahedral Meshing
- Tetrahedral meshing uses ground-truth manifold surfaces and 3D supervision; learning-based inference predicts an initial deformation that optimization then refines.The optimization-based approach alternates occupancy estimation with deformation to minimize the reconstruction loss.Source: §4.3 · p. 6
4.4 Novel View Synthesis and Multi-view 3D Reconstruction
- Multi-view reconstruction and novel-view synthesis optimize a consistency loss through a differentiable renderer, with adaptive subdivision for scene complexity.This application omits the mask and produces explicit tetrahedral meshes usable in simulation or artist editing.Source: §4.4 · p. 6
4 Applications
- 14x faster than OccNet with similar accuracy on point-cloud reconstruction, as reported for Table 1.Source: Table 1 · p. 7
5 Experiments
Experiments evaluate DEFTET on ShapeNet point-cloud reconstruction and single-image reconstruction with 2D supervision, then assess tetrahedral meshing and novel-view synthesis. Across these settings, the reported results combine reconstruction quality with speed, mesh-quality, and application-specific comparisons.
5.1 Point Cloud 3D Reconstruction
- DEFTET matches implicit methods at 14x speed, outperforms mesh-based representations and FIXEDTET, and has comparable performance to voxels at 303 versus 603 grid size with lower memory footprint.The resolution comparison is for models trained on the Chair category.Source: §5.1 · p. 7, Figure 4 · p. 7
5.2 Single Image 3D Reconstruction
- DEFTET has accuracy comparable to DVR with faster inference, while DIB-R runs fastest but cannot handle complex shapes with holes; FIXEDTET produces bumpy shapes and lower scores.DVR takes more than 10 seconds to post-process its prediction; applying Marching Tet to DEFTET predictions extracts a more accurate iso surface.Source: §5.2 · p. 8, Table 2 · p. 8
5.3 Tetrahedral Meshing
- DEFTET matches QuarTet on distortion, outperforms TetGen and TetWild there, and matches TetGen and TetWild but outperforms QuarTet on surface-distance metrics.TetGen failed on 13 shapes and QuarTet on 2, while DEFTET and TetWild succeeded on all shapes; neural inference significantly speeds up meshing while achieving similar performance.Source: §5.3 · p. 8, Table 3 · p. 9
5.4 Novel View Synthesis and Multi-view 3D Reconstruction
- 421 seconds average versus Nerf’s 36,000 seconds, and 3 or 4 minutes versus at least half an hour to reach 30 PSNR; DEFTET has lower PSNR at convergence.DEFTET directly optimizes vertex colors, occupancies, and positions; its exported geometries include materials for other applications such as physics-based simulation.Source: §5.4 · p. 9, Figure 7 · p. 9, Figure 8 · p. 9
6 Conclusion
DEFTET parameterizes solid tetrahedral meshes for 3D reconstruction, optimizing vertex placement and occupancy. The conclusion reports direct tetrahedral-mesh prediction from noisy point clouds and single images, with reconstruction quality comparable to or better than existing work and significantly faster runtime.
6 Conclusion
- DEFTET parameterizes solid tetrahedral meshes for reconstruction and optimizes vertex placement and occupancy.The approach is differentiable with respect to standard 3D reconstruction loss functions.Source: §6 · p. 9
Broader Impact
The work presents a learning-based method for tetrahedral meshing of arbitrary shapes from point clouds and images, targeting 3D reconstruction applications including VR/AR, simulation, and robotics.
Broader Impact
- The method meshes arbitrary objects from point clouds and images.The authors frame learning-based tetrahedral meshing as an advancement on a longstanding graphics problem.Source: Broader Impact · p. 10
- The approach targets 3D reconstruction applications in VR/AR, simulation, and robotics.These are among the application areas the authors identify for 3D reconstruction.Source: Broader Impact · p. 10
- The authors do not anticipate ethics issues with the work.Source: Broader Impact · p. 10