Voxel2Mesh: 3D Mesh Model Generation from Volumetric Data

1 Introduction

• Volumetric CNN outputs are commonly converted to surfaces with Marching Cubes and smoothing, a non-differentiable process that introduces artifacts and prevents end-to-end training.Mesh representations are often needed to model surface morphology and compute area-based statistics.Source: §1 · p. 1
• The architecture jointly decodes voxel features and meshes across resolutions, with the mesh decoder refining an initial surface using sampled voxel features.At each stage, features are sampled from the corresponding-resolution cube before the mesh is deformed and refined by adding vertices where needed.Source: §1 · p. 1, Figure 1 · p. 2
• Learned Neighborhood Sampling draws voxel-decoder features only where needed, in the neighborhood of output vertices.This sampling strategy is one of the architecture’s two identified key features.Source: §1 · p. 1
• Adaptive mesh unpooling concentrates vertices in high-curvature regions and avoids the exponentially large memory requirements of uniform unpooling.Accurately representing the surface requires dense vertices in high-curvature areas, but not elsewhere.Source: §1 · p. 2
• Voxel2Mesh outputs accurate 3D surfaces from volumes without post-processing and is reported to outperform state-of-the-art segmentation methods, especially with relatively small training sets.The evaluations use Electron Microscopy and MRI brain images, as well as CT liver scans.Source: §1 · p. 2

2 Related Work

• CNN-based volumetric methods dominate biomedical segmentation, while winners of two cited competitions used ensembles relying on volumetric CNNs.Researchers use meshes to analyze surfaces, while state-of-the-art volumetric outputs typically require conversion that introduces artifacts and prevents end-to-end training.Source: §2 · pp. 2–3

2.1 Importance of Surface Models

• Mesh representations support surface morphology analysis at required detail, but converting volumetric predictions with Marching Cubes and smoothing introduces artifacts and prevents end-to-end training.Volumes and distances can be estimated using voxels, whereas surface analysis is best accomplished using meshes.Source: §2.1 · p. 3

2.2 Deformable Models

• Deformable surface models can remove some conversion artifacts, but complicate processing, prevent end-to-end training, and are poorly suited to high inter-sample shape variation such as synaptic junctions.Source: §2.2 · p. 3

2.3 Deep Surfaces

• Pixel2Mesh and variants directly map 2D images to 3D meshes through progressively smaller feature maps, but the Pixel2Mesh-3D extension is suboptimal for 3D volumes.Its architecture is described as a straightforward extension of Pixel2Mesh.Source: §2.3 · pp. 3–4

3 Method

• Voxel2Mesh encodes an image volume into latent volumes, builds a pyramid of voxel features, and uses them to deform an initial sphere into a surface mesh.Source: §3 · p. 4

3.1 Mesh Decoder

• The mesh decoder uses graph convolutions to iteratively refine a sphere mesh, updating vertex coordinates from voxel and preceding mesh-block features.Its feature and vertex-update functions are implemented by four graph-convolution layers each.Source: §3.1 · p. 4

Learned Neighborhood Sampling (LNS)

• Learned Neighborhood Sampling selects voxel-feature sampling locations around mesh vertices instead of relying on exact vertex positions or predetermined neighborhoods.It samples only from the voxel feature map corresponding to the current mesh-deformation stage.Source: Learned Neighborhood Sampling (LNS) · pp. 4–5

Adaptive Mesh Unpooling

• Adaptive unpooling adds candidate vertices, retains those farther than a threshold from their parent edge, and restores connectivity using the continuous sphere-to-object mapping.Remeshing recomputes vertex neighbors without non-differentiable operations on vertex variables, preserving overall differentiability.Source: Adaptive Mesh Unpooling · p. 5

3.2 Loss function

• Cross entropy trains the voxel decoder and Chamfer distance to ground-truth boundary points trains the mesh decoder; normal, Laplacian, and edge-length losses regularize mesh convergence and smoothness.The complete loss spans L mesh-decoder stages.Source: §3.2 · pp. 5–6

4.1 Datasets

• Evaluation used mouse-cortex FIB-SEM synaptic volumes, 260 MRI cubes, and 20 CT cubes; Fig. 3 shows a 7,498-vertex Voxel2Mesh result versus 10,422 vertices for Pixel2Mesh-3D.The figure caption says adaptive unpooling obtains better results with fewer vertices.Source: §4.1 · p. 6, Figure 3 · p. 6

4.2 Baselines

• Baselines included Pixel2Mesh-3D, U-NET, V-NET, and 3D variants of TernausNet, LinkNet34, ResNet50, and ResNet50-SE.Fig. 4 shows input volumes, ground truth, CNN baseline, postprocessed CNN baseline, and Voxel2Mesh; orange boxes mark false-positive regions.Source: §4.2 · pp. 6–7, Figure 4 · p. 7

4.3 Comparative Results

• Voxel2Mesh performed best in all cases except synaptic cleft, where it came second; improvement was greatest on the liver and synaptic-junction datasets with fewer training samples.Table 1 reports comparative results on three datasets using IoU.Source: §4.3 · pp. 7–8, Table 1 · p. 8

4.4 Ablation Study

• LNS+AMU delivered a clear benefit over Pixel2Mesh-3D's corresponding sampling and unpooling modules, and over the fixed-neighborhood Hypothesis Sampling strategy.The ablation compares these strategies using Chamfer distance; Table 2 identifies the comparison scope.Source: §4.4 · p. 8, Table 2 · p. 8

• Voxel2Mesh maps image volumes directly to 3D surface meshes, making the usual post-processing steps unnecessary while preserving accuracy.The authors describe the architecture as bridging voxel-based and surface-based representations.Source: §5 · pp. 8–9
• Adaptive sampling and unpooling are key to the architecture's results.Source: §5 · p. 8
• Future work will extend the approach to structures with more complex topologies, such as synaptic clefts and their potential holes.Source: §5 · p. 9