Source-linked AI summary
MeshSDF: Differentiable Iso-Surface Extraction
Edoardo Remelli, Artem Lukoianov, Stephan R. Richter, Benoît Guillard, Timur Bagautdinov, Pierre Baque, Pascal Fua
TL;DR
Explicit meshes are needed for important applications, but standard isosurface extraction prevents continuous implicit fields from serving as differentiable mesh parameterizations. MeshSDF differentiates surface samples through signed distance fields, enabling end-to-end differentiable meshes with variable topology; the paper reports advantages in single-view reconstruction and CFD shape optimization. Its scope includes author-identified constraints involving topology changes in Marching Cubes and local minima in aerodynamic optimization.
Problem
Applications such as CFD and physically based rendering require explicit surface meshes, but Marching Cubes is not differentiable with respect to the underlying implicit field.
Method
MeshSDF derives how implicit-field perturbations change local surface samples, then uses a custom backward pass after mesh extraction to preserve end-to-end differentiability.
Results
MeshSDF produces explicit surface representations with arbitrary topology and no resolution limit, and gives an edge over state-of-the-art algorithms in single-view reconstruction and CFD shape optimization.
Takeaways & Limitations
MeshSDF combines continuous implicit modeling with objective functions that require explicit surface meshes, including topology-variant parameterizations.
Takeaways & Limitations
Marching Cubes cannot swap equal-valued edge signs through backpropagation, preventing topology changes during differentiation; aerodynamic results also vary with initialization, possibly because of latent-space local minima.
Abstract
from arXiv · showhide
Geometric Deep Learning has recently made striking progress with the advent of continuous Deep Implicit Fields. They allow for detailed modeling of watertight surfaces of arbitrary topology while not relying on a 3D Euclidean grid, resulting in a learnable parameterization that is not limited in resolution. Unfortunately, these methods are often not suitable for applications that require an explicit mesh-based surface representation because converting an implicit field to such a representation relies on the Marching Cubes algorithm, which cannot be differentiated with respect to the underlying implicit field. In this work, we remove this limitation and introduce a differentiable way to produce explicit surface mesh representations from Deep Signed Distance Functions. Our key insight is that by reasoning on how implicit field perturbations impact local surface geometry, one can ultimately differentiate the 3D location of surface samples with respect to the underlying deep implicit field. We exploit this to define MeshSDF, an end-to-end differentiable mesh representation which can vary its topology. We use two different applications to validate our theoretical insight: Single-View Reconstruction via Differentiable Rendering and Physically-Driven Shape Optimization. In both cases our differentiable parameterization gives us an edge over state-of-the-art algorithms.
1 Introduction
Continuous deep implicit fields model detailed, arbitrary-topology surfaces without fixed-resolution grids, but explicit mesh applications remain hindered by non-differentiable isosurface extraction. MeshSDF addresses this gap by differentiating surface samples with respect to signed distance fields and applying the resulting representation to reconstruction and aerodynamic optimization.
- Continuous deep implicit fields represent detailed watertight surfaces of arbitrary topology without relying on fixed-resolution 3D grids or meshes.
- Explicit meshes remain necessary for applications including CFD simulations and physically based rendering.
- Marching Cubes converts implicit surfaces into meshes but is not fully differentiable, preventing direct use of continuous implicit fields as explicit-mesh parameterizations.
- Existing work either approximates extraction with low-resolution or fixed-topology methods or reformulates objectives in the implicit domain, which can be difficult for complex tasks such as CFD optimization.
- MeshSDF derives a closed-form derivative of surface samples with respect to the implicit field, enabling end-to-end differentiable meshes with arbitrary topology and no resolution limit.
- The method is evaluated through differentiable single-view reconstruction and physically driven CFD shape optimization, where it reportedly outperforms state-of-the-art algorithms.
2 Related Work
Related work has pursued continuous implicit representations, learned approximations to mesh extraction, or implicit-domain objectives, but each leaves important constraints for explicit, differentiable meshes.
- From Discrete to Continuous Implicit Surface Models: Dense-grid implicit surface methods provide watertight, variable-topology representations but are limited in resolution by memory requirements.
- From Discrete to Continuous Implicit Surface Models: Continuous deep implicit fields remove the fixed-resolution limitation and have been used for single-view reconstruction and shape completion.
- Converting Implicit Functions to Surface Meshes: Non-differentiable isosurface extraction has largely prevented implicit representations from being used in applications requiring explicit surface parameterizations.
- Converting Implicit Functions to Surface Meshes: Marching Cubes samples fields on a grid, detects edge zero-crossings, and builds meshes through a lookup table, but its vertex interpolation does not permit topology changes through backpropagation.
- Related Approaches: Prior approaches either emulate extraction with neural networks or avoid meshes by expressing objectives directly in implicit fields.
- Emulating Iso-Surface Extraction: Neural extraction methods can support topology changes but may require tracking all possible topologies, limiting practical output resolution.
- Reformulating Objective Functions in terms of Implicit Fields: Implicit-domain reformulations can require complex derivations for each loss, fixed-resolution grids, or specialized procedures such as sphere tracing.
3 Method
MeshSDF converts deep signed distance fields into explicit, end-to-end differentiable meshes while allowing topology changes. It derives surface-sample gradients from local signed-distance perturbations and uses them to optimize mesh-based objectives.
- Motivation: Mesh optimization commonly maps low-dimensional parameters to mesh vertices, which implies a fixed topology.The paper motivates an implicit alternative for topology-changing optimization, but this requires differentiable conversion to explicit meshes.
- Deep Implicit Surface Representation: MeshSDF uses a learned latent-code parameterization of signed distance fields whose zero level sets define the target surfaces.The network maps latent vectors to signed distance fields, while training uses surface samples and regularization.
- Differentiable Iso-Surface Extraction: Marching Cubes extracts meshes by sampling an implicit field on a grid, but its linear-interpolation vertex placement is not fully differentiable through topology changes.The forward pass still uses Marching Cubes for iso-surface extraction, while the backward pass uses the derived differentiable relation.
- Differentiable Iso-Surface Extraction: The method derives surface-sample motion from local signed-distance perturbations and uses surface normals to differentiate vertex positions with respect to the implicit field.A negative perturbation locally inflates the surface and a positive perturbation deflates it; the resulting relation supports the backward pass.
- Topology-Variant Parameterization: A topology-variant experiment transforms a sphere into a torus while optimizing either a surface-to-surface or image-to-image objective.The transformation changes genus from 0 to 1 and backpropagates surface gradients without reformulating the losses in terms of implicit surfaces.
- Differentiable Iso-Surface Extraction: Gradients from objectives defined on surface samples can be propagated to the latent code, producing an end-to-end differentiable mesh representation that captures changing topologies.MeshSDF evaluates the field and extracts a mesh in the forward pass, then assembles the backward pass using the chain rule and an additional surface-sample evaluation.
4 Experiments
The experiments test MeshSDF on differentiable topology changes, single-view reconstruction, and CFD-driven shape optimization. Across these applications, differentiable surface refinement improves reconstruction results, while MeshSDF supports topology changes and lower-drag, semantically correct shapes.
- Differentiable Topology Changes: MeshSDF demonstrates differentiable topology changes by smoothly transforming a sphere into a torus while minimizing surface or image losses.The losses operate directly on explicit surfaces, without reformulating them as implicit-surface objectives.
- Single-View Reconstruction: MeshSDF refines an encoder-predicted latent code through differentiable rasterization during single-view reconstruction.The method performs 400 Adam gradient-descent iterations and retains the code with the smallest task loss.
- Single-View Reconstruction: At least 12% improvement over MeshSDF (raw) on all ShapeNet metrics enables MeshSDF to outperform other state-of-the-art reconstruction approaches.The comparison isolates the benefit of end-to-end refinement over the same encoder-decoder baseline.
- Single-View Reconstruction: On Pix3D, the full MeshSDF model outperforms all prior approaches in every metric, while MeshSDF (raw) performs on par with more sophisticated camera-aware methods.The model is trained only on synthetic chair renders and is not fine-tuned on Pix3D images.
- Shape Optimization: For CFD-driven optimization, MeshSDF produces lower drag values than competing parameterizations while allowing topology changes and semantically correct surfaces.The quantitative evaluation uses eight randomly chosen cars, and qualitative results compare FreeForm, PolyCube, and MeshSDF.
5 Conclusion
MeshSDF extracts explicit surface meshes from Deep Signed Distance Functions while preserving end-to-end differentiability, enabling objective functions that require explicit surface meshes.
- MeshSDF extracts 3D surface meshes from Deep Signed Distance Functions while preserving end-to-end differentiability.
7 Broader Impact
The proposed techniques target Computer Assisted Design by applying deep learning to engineering shape-design problems involving aerodynamic, hydrodynamic, and heat-transfer performance.
- The techniques target Computer Assisted Design through shape design for aerodynamic, hydrodynamic, and heat-transfer performance.
8 Supplementary Material
The supplement explains why Marching Cubes is not differentiable and formally proves the paper’s iso-surface differentiation result. It also presents the geometric perturbation argument underlying the derivative.
- 8.1 Non-differentiability of Marching Cubes: The supplement explains Marching Cubes’ non-differentiability and provides a formal proof of the main differentiability theorem.
- 8.1 Non-differentiability of Marching Cubes: Marching Cubes samples an implicit field on a discrete grid, detects edge zero-crossings, assembles topology, and estimates vertex locations.
- 8.1 Non-differentiability of Marching Cubes: Its linear-interpolation vertex function is discontinuous when s_i = s_j, preventing sign swaps through backpropagation and therefore topology changes during differentiation.
- 8.2 Iso-surface Differentiation: For a signed distance perturbation, the local surface displacement is Δv = −n(v′)Δs(v), linking field changes to surface geometry through the surface normal.
- 8.2 Iso-surface Differentiation: The supplement uses Fig. 7 to illustrate the geometric proof of the local surface-change relation.
8.3 Accelerating Iso-Surface Extraction
Iso-surface extraction can use any extraction technique because the differentiation method is extraction-independent. In iterative optimization, evaluating the implicit field only near prior threshold crossings reduces field-sampling cost, with comparisons including DMC.
- The iso-surface differentiation method is independent of the technique used to extract surface samples.
- During iterative optimization, the method re-evaluates the implicit field only at grid corners whose previous values were below a threshold in absolute value.
- The supplement compares the approach with Deep Marching Cubes on genus-0 cow and genus-1 rubber-duck shapes using Chamfer l2 distance.
- Figure 8 depicts accelerated iso-surface extraction by evaluating the field where new zero-crossings are expected between iterations.
8.5 End-to-End Training
MeshSDF’s differentiable iso-surface extraction allows gradients from surface-distance metrics to reach the implicit field, enabling direct end-to-end training.
- MeshSDF backpropagates gradients from a surface-distance metric such as Chamfer l2 distance to the implicit representation.The metric compares sampled points P and Q from two surfaces.
- The resulting gradient computation enables MeshSDF to minimize the target metric directly during training.
- Fine-tuning pretrained DeepSDF models with Chamfer distance improves performance, motivating MeshSDF as the reported variant.
8.6 Single View Reconstruction
MeshSDF reconstructs shapes from single images and refines them through differentiable rasterization. On ShapeNet and Pix3D, image-based refinement improves reconstruction against standard metrics and outperforms prior methods in F-score.
- Pipeline: The SVR pipeline conditions a deep implicit field on an input image, then predicts an initial latent code and mesh for reconstruction.The encoder maps images to latent vectors used to evaluate the implicit function.
- Differentiable refinement: At inference, MeshSDF projects mesh vertices and facets through differentiable rasterization and minimizes silhouette loss with respect to the latent code.Optimization uses Adam for 400 gradient-descent iterations and retains the code with the smallest task loss.
- Results: Image-based refinement allows MeshSDF to outperform prior methods on ShapeNet in F-score.
- Evaluation: On Pix3D, the evaluation uses 2894 untruncated, unoccluded chair images and ground-truth bounding boxes for fair comparison.
- Qualitative results: The supplementary results include qualitative comparisons, failure cases, and animations of iterative reconstruction refinement.Reported failure cases are samples where refinement does not improve reconstruction.
8.7 Aerodynamic Shape Optimization
MeshSDF provides a learned, differentiable parameterization for aerodynamic shape optimization, combining mesh-based pressure prediction with latent-space regularization. The experiments compare it with hand-crafted and vertex-based alternatives while addressing unrealistic or unstable designs through constraints and conservative regularization.
- Optimization pipeline: The aerodynamic pipeline encodes a car shape with DeepSDF, extracts a mesh, predicts its pressure field with a Mesh CNN, and computes drag as the objective.
- Data and simulation: The experiment trains on cleaned, re-triangulated ShapeNet car meshes and uses OpenFoam simulations to obtain ground-truth pressure values.Each simulation uses 15 meters per second inflow velocity and airflow density 1.18.
- Baselines: Direct vertex-wise optimization produces unrealistic designs that minimize predicted drag but fail to converge in CFD simulations, supporting low-dimensional regularization.
- Baselines: MeshSDF is compared with scaling, FreeForm, and PolyCube parameterizations, which impose structured deformation restrictions or regularization.
- Constraints: Soft constraints reserve space for the driver and engine, but may be violated during optimization.
- Regularization: A standard latent-code penalty prevented most unrealistic shapes, but some converged designs remained coarse and noisy.
- Optimization behavior: Different initial shapes yield different final designs, which the authors associate with possible local minima in the DeepSDF latent space.
8.8 Comparison to implicit field differentiable rendering
MeshSDF uses differentiable meshes with an off-the-shelf rasterizer instead of differentiable sphere tracing for implicit fields. In the reported rendering comparison, it achieves lower loss with fewer function evaluations and shorter runtime.
- Method comparison: MeshSDF projects surface triangles and rasterizes them in parallel, avoiding the sequential ray queries required by sphere tracing.
- Comparison: MeshSDF records 0.004625 loss, 266k function evaluations, and 0.29 runtime, compared with Liu20’s 0.005973, 898k, and 1.24.
- Additional results: The supplementary figures provide comparative SVR results on ShapeNet and Pix3D, plus Pix3D failure cases where silhouette-based refinement misses fine topological details.
- Additional results: The aerodynamic optimization results are additionally illustrated in Figure 16.