PaperlayerView exact revision on arXiv

Source-linked AI summary

Flexible Isosurface Extraction for Gradient-Based Mesh Optimization

Tianchang Shen, Jacob Munkberg, Jon Hasselgren, Kangxue Yin, Zian Wang, Wenzheng Chen, Zan Gojcic, Sanja Fidler, Nicholas Sharp, Jun Gao

arXiv:2308.05371v1cs.GRcs.CVcs.LG

TL;DR

Gradient-based mesh optimization needs an isosurface representation that is both differentiable in practice and flexible enough to preserve features without unstable or degenerate geometry. FlexiCubes adds adjustable parameters to a Dual Marching Cubes-based extraction scheme, and extensive evaluations report improved mesh quality and geometric fidelity across applications, while retaining documented continuity and intersection limitations.

  • Problem

    Existing isosurface methods can lack flexibility for high-quality feature-preserving meshes or suffer numerical instability when used for gradient-based optimization.

  • Method

    FlexiCubes adapts Dual Marching Cubes with additional degrees of freedom for locally positioning extracted vertices and optimizing representation parameters with the scalar field.

  • Results

    FlexiCubes provides significant benefits across inverse rendering, physical and geometric energy optimization, and generative 3D modeling, with meshes that capture desired geometry at low element counts.

  • Takeaways & Limitations

    The representation supports application-specific objectives and regularizers, while optionally enabling hierarchical refinement and tetrahedral mesh extraction.

  • Takeaways & Limitations

    The method does not guarantee nonself-intersecting output and is not globally continuous because the extracted mesh can jump when the isosurface crosses a grid vertex.

Abstract

from arXiv · show

This work considers gradient-based mesh optimization, where we iteratively optimize for a 3D surface mesh by representing it as the isosurface of a scalar field, an increasingly common paradigm in applications including photogrammetry, generative modeling, and inverse physics. Existing implementations adapt classic isosurface extraction algorithms like Marching Cubes or Dual Contouring; these techniques were designed to extract meshes from fixed, known fields, and in the optimization setting they lack the degrees of freedom to represent high-quality feature-preserving meshes, or suffer from numerical instabilities. We introduce FlexiCubes, an isosurface representation specifically designed for optimizing an unknown mesh with respect to geometric, visual, or even physical objectives. Our main insight is to introduce additional carefully-chosen parameters into the representation, which allow local flexible adjustments to the extracted mesh geometry and connectivity. These parameters are updated along with the underlying scalar field via automatic differentiation when optimizing for a downstream task. We base our extraction scheme on Dual Marching Cubes for improved topological properties, and present extensions to optionally generate tetrahedral and hierarchically-adaptive meshes. Extensive experiments validate FlexiCubes on both synthetic benchmarks and real-world applications, showing that it offers significant improvements in mesh quality and geometric fidelity.

1 INTRODUCTION

Gradient-based mesh optimization needs extraction procedures that both differentiate effectively and flexibly fit surface features with few elements. FlexiCubes addresses this tension by adding locally adjustable degrees of freedom while preserving favorable mesh properties and supporting practical optimization.

  • Motivation: High-quality surface meshes are important for representing and processing 3D geometry, but excessive elements, self-intersections, slivers, or poor geometric fit can make them unsuitable.Automatic mesh generation increasingly supports inverse rendering, structural optimization, and generative 3D modeling.
  • Motivation: Gradient-based optimization commonly represents a mesh as the level set of an optimized scalar field or signed distance function.Direct vertex optimization can suffer from degeneracy and local minima without careful initialization, remeshing, and regularization.
  • Design goals: The desired procedure must provide effective gradient-based differentiation and allow individual vertices to adjust locally to surface features with few elements.These requirements are labeled Grad. and Flexible in the paper’s taxonomy.
  • Design goals: Existing methods often trade flexibility against optimization robustness; Marching Cubes fixes vertices to a lattice and cannot align generated meshes with non-axis-aligned sharp features.The paper identifies increased flexibility as potentially introducing degenerate geometry and self-intersections that hinder convergence.
  • FlexiCubes: FlexiCubes adapts Dual Marching Cubes and adds degrees of freedom that flexibly position extracted vertices while constraining the formulation to produce manifold, watertight, mostly intersection-free meshes.The formulation is designed to enable well-behaved differentiation with respect to the underlying mesh.
  • FlexiCubes: FlexiCubes is evaluated across inverse rendering, physical and geometric energy optimization, and generative 3D modeling, with extensions for hierarchical refinement and tetrahedral meshes.The paper reports that resulting meshes capture desired geometry at low element counts and are readily optimized by gradient descent.

2 RELATED WORK

Prior isosurface extraction methods use spatial decomposition, surface tracking, shrink wrapping, or learned mesh-generation schemes, but several are difficult to differentiate or limited in topology and optimization suitability. Differentiable implicit-surface methods address parts of this problem, motivating a specialized extraction representation for gradient-based optimization.

  • Spatial Decomposition: Spatial-decomposition methods divide space into cells and create polygons where the surface passes, with Marching Cubes as the representative approach.Marching Cubes has topological ambiguities and struggles with sharp features; Dual Contouring instead extracts per-cell vertices to fit local details.
  • Surface Tracking: Surface-tracking methods exploit neighboring surface samples, but differentiating through their discrete iterative updates is non-trivial.Marching Triangles is an early representative method, while later work adds adaptivity or sharp-feature alignment.
  • Shrink Wrapping: Shrink-wrapping methods deform or grow meshes toward an isosurface, but generally support limited topologies, may require manually selected critical points, and are difficult to differentiate through.These restrictions make them poorly suited to broad gradient-based mesh optimization.
  • Neural Mesh Generation: Neural mesh-generation approaches can use predefined topologies, surface patches, or coarse-to-fine prediction, but predefined structures may not generalize to complex topologies and patch collections may lack coherent surfaces.Two-stage approaches can generate different topologies by refining a coarse structure into a surface mesh.
  • Differentiable Mesh Reconstruction: Recent differentiable reconstruction methods extract meshes from implicit functions, including schemes based on topology expectations or specialized gradient sampling.Deep Marching Cubes scales poorly with increasing grid resolution, while other methods focus on differentiating through mesh extraction.

3 BACKGROUND AND MOTIVATION

Differentiable mesh optimization requires extraction methods that balance flexible feature representation with stable, high-quality geometry. Existing marching and dual methods each trade off feature fidelity, topology, or optimization stability, motivating FlexiCubes’ added vertex-position degrees of freedom on a Dual Marching Cubes foundation.

  • Problem setting: The differentiable extraction pipeline defines a scalar field, extracts its 0-isosurface, evaluates objectives on the mesh, and back-propagates gradients to the scalar function.This setting differs from extracting a mesh once from a fixed, known scalar field.
  • Marching methods: Marching extraction restricts vertices to grid edges, preventing alignment with sharp features and producing sliver triangles when the isosurface passes near grid vertices.Improved gradients do not remove this restricted output space.
  • Dual Contouring: Dual Contouring positions vertices within cells and captures sharp features, but QEF degeneracies can cause distant vertices, self-intersections, and numerically unstable differentiation.Constraining vertices can zero gradients, while sufficient regularization removes sharp-feature flexibility.
  • FlexiCubes: FlexiCubes introduces optimized degrees of freedom for local vertex adjustment while building on Dual Marching Cubes to address manifoldness and difficult configurations.The approach generalizes centroid positioning rather than relying solely on the scalar field.
  • Dual Marching Cubes: Dual Marching Cubes improves connectivity and can emit manifold meshes, but centroid positioning lacks the freedom to fit individual sharp features.QEF-based positioning retains drawbacks similar to Dual Contouring.

4 METHOD

FlexiCubes extends Dual Marching Cubes with learnable parameters for vertex positioning, quad splitting, and grid deformation while preserving robust differentiable extraction. It also supports tetrahedral and hierarchically adaptive meshes, with constraints addressing manifoldness and intersections.

  • Core representation: FlexiCubes extracts a triangle mesh from a scalar grid using Dual Marching Cubes and introduces three parameter sets for flexible optimization.The parameters control dual vertex positioning, quadrilateral splitting, and spatial grid deformation, optimized with the scalar function via automatic differentiation.
  • Dual Marching Cubes: Dual Marching Cubes determines connectivity from scalar signs and extracts dual vertices on primal faces, linking neighboring cell vertices into the dual mesh.Unlike ordinary Marching Cubes, it can extract multiple vertices from a primal face in configurations such as C13.
  • Flexible vertex positioning: Positive α weights shift edge crossings, while positive β weights move dual vertices within primal faces instead of fixing them at face centroids.The α and β ranges are restricted with tanh(·) + 1 activations; convex parameterizations keep extracted vertices within the cell and prevent nearly all self-intersections.
  • Flexible quad splitting: γ weights make non-planar quadrilateral triangulation differentiable during optimization and select the larger-product diagonal for final extraction.Optimization inserts an interpolated midpoint and four triangles, smoothly interpolating between the two diagonal choices; inference removes the midpoint.
  • Mesh extensions: FlexiCubes can output tetrahedral meshes conforming exactly to the extracted surface and hierarchically adaptive meshes with application-specific refinement policies.Adaptive extraction enforces sign consistency across octree levels, yielding nearly watertight meshes, while rare difficult tetrahedral configurations can produce small defects.
  • Optimization objectives: The representation supports application-specific geometric, rendering, physical, and mesh-quality objectives through differentiable optimization and dedicated regularizers.These objectives include geometric depth and SDF losses, image-space rendering losses, and mesh-quality regularizers.

5 EXPERIMENTS

Experiments evaluate FlexiCubes for mesh reconstruction and mesh-based regularization. Across synthetic comparisons, it improves geometric alignment and mesh quality while supporting objectives defined directly on the extracted mesh.

  • 5.1 Mesh Reconstruction: FlexiCubes is compared with differentiable Marching Cubes and DMTet, plus non-differentiable extractors using ground-truth SDFs.DMTet is additionally evaluated at resolutions matching output triangle counts.
  • 5.1 Mesh Reconstruction: The evaluation measures reconstruction accuracy with Chamfer distance, F-score, edge metrics, and inaccurate normals, alongside intrinsic triangle quality.Intrinsic quality includes aspect ratios, radius ratios, and minimum and maximum angles.
  • 5.1 Mesh Reconstruction: Post-processing extractors underperform, whereas FlexiCubes aligns better with ground-truth geometry than MC and DMTet while maintaining mesh quality comparable to NDC.The result highlights the role of end-to-end optimization in reducing discretization errors.
  • 5.2 Mesh Optimization with Regularizations: FlexiCubes supports automatic differentiation of mesh-dependent objectives, allowing regularizers unavailable as direct scalar-field functions to enter gradient-based optimization.This flexibility is contrasted with the rigid degrees of freedom of Marching Cubes.
  • 5.2 Mesh Optimization with Regularizations: Equilateral edge regularization produces more uniform triangles with only slight reconstruction degradation, while preserving geometric quality better than MC and DMTet.The comparison uses percentages of triangles with poor aspect ratio, radius ratio, or minimum angle.
  • 5.2 Mesh Optimization with Regularizations: FlexiCubes incorporates developability energy to penalize the smallest covariance-matrix eigenvalue of face normals, while Marching Cubes fails to preserve shape features under this objective.Developability measures stretching relative to a flat sheet without penalizing bending in one direction.

6 APPLICATIONS

Applications replace DMTet or other extraction components with FlexiCubes in photogrammetry, animation, generative modeling, and differentiable physics. These integrations report improved geometry, mesh quality, texture layouts, and physical-parameter recovery.

  • 6.1 Photogrammetry Through Differentiable Rendering: Replacing DMTet with FlexiCubes in nvdiffrec improves geometry reconstruction at equal triangle count on real-world photographic and LDraw scenes.The reported gains include more uniform tessellation, better small-detail capture, and higher detail in the Roller scene.
  • 6.1 Photogrammetry Through Differentiable Rendering: FlexiCubes produces fewer sliver triangles and nicer triangulation, which simplifies UV unwrapping and yields larger texture regions that improve filtering.The texture-layout effect is illustrated on the Porsche scene.
  • 6.2 Animated Meshing: End-to-end optimization over animated frames lets FlexiCubes jointly optimize a skinned mesh instead of fitting only a reference-pose mesh and re-skinning it afterward.The method uses differentiable mesh skinning and randomized cameras and animation frames.
  • 6.3 3D Mesh Generation: As a plug-and-play GET3D replacement, FlexiCubes achieves better FID scores across categories and generates models with more detail, fewer sliver triangles, and higher qualitative mesh quality.Only the generator’s final layer is modified to predict FlexiCubes parameters.
  • 6.4 Differentiable Physics: FlexiCubes extracts tetrahedral meshes for differentiable physics and rendering, enabling joint recovery of rest-pose shape and material parameters from multi-view deformation videos.The application models elastic objects with FEM simulation and neo-Hookean elasticity.

7 DISCUSSION

The discussion weighs FlexiCubes’ flexibility against computational and geometric limitations. It reports higher extraction cost, possible self-intersections, and discontinuities inherited from Dual Marching Cubes, while identifying future extensions.

  • 7.1 Performance: FlexiCubes is slower and more memory-intensive than DMTet and substantially more costly than ordinary Marching Cubes.These costs are generally small relative to downstream applications, whose resolution is often limited by rendering or neural-network evaluation.
  • 7.1 Performance: More concise extracted meshes may offset some pipeline memory costs despite the representation’s additional parameters.The application-level comparison reports that fewer triangles can lower overall memory requirements.
  • 7.2 Limitations: The core algorithm guarantees manifoldness but does not guarantee non-self-intersecting output because flexible vertex motion can create intersecting configurations.Additional care may be required when a watertight mesh is imperative.
  • 7.2 Limitations: FlexiCubes is not globally continuous because the mesh jumps when the isosurface crosses a grid vertex.The authors report that small local discontinuities do not obstruct stochastic optimization in practice.
  • 7.3 Future Work: Future directions include volumetric-rendering integration, 4D spatiotemporal meshing, and adaptive hierarchical extraction for generative modeling.These directions are presented as opportunities for improving gradient approximation and extending applications.

SUPPLEMENTAL MATERIAL

The supplement provides additional method, isosurfacing, experimental, baseline, qualitative, and application details beyond the main paper.

  • Supplemental Material: Supplementary sections expand the method, related isosurfacing techniques, experimental settings, qualitative results, and application details.The supplement is organized into Sections A through D covering these topics.

A.1 Tetrahedral Mesh Extraction

The tetrahedral extension resolves connectivity ambiguities in Dual Marching Cubes using case-specific construction rules, while also addressing adaptive-grid constraints and non-manifold configurations. A rare C18 configuration remains incompletely filled.

  • Ambiguity handling: Connectivity ambiguities arise when a cell contains multiple extracted mesh vertices, because incorrect connections can create intersections or holes.The ambiguity occurs when extending tetrahedralization to Dual Marching Cubes.
  • Ambiguity handling: Different-sign grid edges form a four-sided pyramid connected to four mesh vertices, which is subdivided into two tetrahedra.This construction is uniquely determined across DMC configurations.
  • Ambiguity handling: Same-sign grid edges use the two grid vertices and two vertices from consecutive adjacent cells, selecting a mesh vertex through the shared face’s different-sign edge.The rule identifies the shared face, then selects the mesh vertex corresponding to an edge with different signs.
  • Implementation and behavior: The C18 case does not completely fill the interior volume, although it rarely occurs during optimization and does not obstruct downstream applications.The authors identify this as a limitation of the method.
  • Adaptive and manifold extraction: Adaptive-grid SDF constraints interpolate finer-face vertex values from coarser-face values using precomputed bilinear weights to avoid cracks or non-manifold surfaces.SDF values at finer-face vertices that are not shared with the coarse face are not independently optimized.
  • Adaptive and manifold extraction: A modification for DMC cases C16 and C19 makes FlexiCubes’ extracted surface always 2-manifold on uniform grids.This addresses rare non-manifold outputs from the original Dual Marching Cubes algorithm.

B ANALYSIS

The analysis compares differentiable isosurfacing methods by optimizing an initially spherical SDF against rendered depth, silhouette, and SDF objectives. Baseline implementations include DMTet, Marching Cubes, Dual Contouring variants, and NDC.

  • Experimental setup: The experiment initializes every method with an SDF representing a sphere, extracts a mesh each iteration, and renders it from shared random camera views.Rendered depth and silhouette differences are compared with ground truth, alongside an SDF loss.
  • Baseline implementations: Marching Cubes and Dual Contouring were implemented in PyTorch to enable automatic differentiation, while DMTet uses its official implementation.The Dual Contouring implementation solves the quadratic error function with PyTorch’s linear solver.
  • Baseline implementations: The Dual Contouring baseline adds regularization when gradients are nearly coplanar, biasing the solution toward the centroid of associated zero-crossings.This regularizer prevents the solution from exploding to a distant location.
  • Baseline implementations: The study ablates Dual Contouring regularizer weights λ=1 and λ=0.01 and compares them with a centroid-only variant.The variants are denoted DCreg1, DCreg001, and DCcentroid.

C EXPERIMENTAL DETAILS

The experimental pipeline standardizes baseline inputs, regularization, losses, optimization, and object scaling across isosurfacing methods. Adaptive regularization is selected independently for each cube in the Dual Contouring setup.

  • Inputs and baselines: MC_SDF and NDC_SDF receive ground-truth SDF values evaluated at grid vertices, while DC_hermite additionally receives finite-difference gradients at zero-crossings.The NDC input scale is handled using an author-provided SDF computation function because the pretrained network is scale-sensitive.
  • Regularization: The Dual Contouring regularizer weight λ is chosen independently per cube by doubling it until the updated QEF solution lies inside the cube.The procedure begins with a small λ and follows an adaptive strategy.
  • Optimization setup: The reconstruction pipeline uses mask, depth, SDF, sign, and deviation losses, with respective primary loss scales of 1, 10, 2000, and 1 for mask, depth, SDF, and deviation.The sign-loss scale decays linearly from 0.2 to 0.01 during training.
  • Optimization setup: Each shape is optimized for 1000 iterations at learning rate 0.01 using uniform grids in [−1, 1]^3.Objects are centered at the origin and scaled so their bounding-box longest side equals 1.8.

C.2 Evaluation Metrics

The evaluation measures mesh geometry, sharp-feature reconstruction, topology, normals, triangle quality, and complexity using point-cloud, edge, angle, and manifold statistics. Mesh-reconstruction tables report these metrics alongside vertex and triangle counts.

  • Geometric fidelity: Chamfer Distance measures nearest-neighbor distance between sampled point clouds, using 100,000 points per mesh in the main mesh comparison.The nvdiffrec NeRF synthetic evaluation uses a different visible-triangle version and scale, so its values are not directly comparable.
  • Geometric fidelity: F1-score combines point-cloud precision and recall, counting matches when nearest-neighbor distance is below the 0.003 threshold.Predicted and ground-truth meshes are sampled into equal-sized point clouds.
  • Sharp features and normals: Edge Chamfer Distance and Edge F-score evaluate reconstruction of sharp features by comparing point-cloud samples classified as edge points.A point is classified as an edge point when the mean normal dot product with neighbors is below 0.2.
  • Sharp features and normals: Inaccurate normals report the percentage of nearest point pairs whose normal-angle difference exceeds 5 degrees.Normals are stored from the generating mesh faces before nearest-pair comparison.
  • Triangle quality: Aspect Ratio and Radius Ratio measure triangle regularity, with smaller values indicating better triangle quality.The evaluation follows the definitions implemented in PyVista.
  • Reported results: Mesh-reconstruction results report CD, F1, ECD, EF1, NV, NE, SI, IN>5°, SA, vertex count, and triangle count.The table uses distinct scales for CD and ECD as indicated in its caption.
  • Triangle quality and topology: The evaluation also records minimum and maximum triangle angles, non-manifold vertices, non-manifold edges, self-intersecting triangles, and triangles with angles below 10°.These metrics cover triangle geometry and topological defects.
  • Adaptive reconstruction: Adaptive-resolution reconstruction jointly optimizes mesh topology and octree structure, beginning from a low-resolution uniform voxel grid.A running average of each cell’s vertex losses guides later subdivision after shape convergence.

C.4 Additional Results

Additional results show FlexiCubes improving visual reconstruction quality across rotations and downstream mesh-generation settings. The method is also integrated into photogrammetry, animated-mesh simplification, and generative 3D modeling pipelines.

  • Additional comparisons: FlexiCubes avoids the stair-step artifacts seen with Marching Cubes on non-axis-aligned features and produces more uniform triangles than DMTet across poses.The comparison covers three poses of the same model.
  • Additional comparisons: The Ldev regularizer reduces visible seams in reconstructed shapes compared with omitting the regularizer.This comparison is shown in the influence study for Ldev.
  • Photogrammetry: In photogrammetry, FlexiCubes replaces DMTet in nvdiffrec while retaining the second optimization pass unchanged.The implementation uses λdev = 0.25 and scales the silhouette mask loss by λmask = 5.0.
  • Animated mesh simplification: FlexiCubes supports mesh simplification for animated objects using a coarse 32^3 grid and differentiable re-skinning during optimization.The end-to-end animation procedure requires gradients to propagate through topology-related operations.
  • 3D mesh generation: FlexiCubes serves as a plug-and-play differentiable extraction module in GET3D and is reported to produce significantly improved mesh quality.The generator is modified to predict the SDF, per-vertex deformation, and 21 FlexiCubes weights per cube.

D.4 Physics Simulation

The physics-simulation experiments use FlexiCubes to generate tetrahedral meshes for differentiable optimization from multi-view videos. The supplementary results emphasize both the integration with physical simulation and the numerical precautions needed for stable training.

  • Physical simulation: FlexiCubes generates tetrahedral meshes with well-defined topology that can be directly used in physical simulation and combined with differentiable rendering.The framework optimizes shape, material, and physical parameters from multi-view videos.
  • Training procedure: The training procedure separates shape-and-texture optimization from physical-parameter optimization because jointly optimizing geometry and physical parameters is unstable.The first stage assumes the object does not deform in the initial frame; the second optimizes mass density through differentiable simulation.
  • Numerical stability: Filtering tetrahedra with volume below 2e-7 significantly improves physical-simulation stability for shapes normalized in (-0.45, 0.45).Tiny-volume tetrahedra can cause numerical issues during simulation.
  • Visual comparisons: Figure 27 compares fixed-SDF and differentiable isosurfacing methods at a common 64^3 grid resolution, except DMTet uses an 80^3 tetrahedral grid to match output triangle counts.The comparison includes Marching Cubes, Dual Contouring, Neural Dual Contouring, Deep Marching Tetrahedra, and FlexiCubes.
  • Visual comparisons: Across the original NeRF scenes, FlexiCubes is described as providing more uniform tessellation and capturing small details better than the original nvdiffrec DMTet implementation.The Lego scene is cited as an example of improved small-detail capture.
Loading 2308.05371v1…