Fully Convolutional Mesh Autoencoder using Efficient Spatially Varying Kernels

1 Introduction

• UV-space CNNs can suffer parameterization distortion and seams and do not readily extend to tetrahedral meshes, while prior mesh autoencoders have limits in mesh scope and fine-detail reconstruction.Earlier approaches also include graph convolutions that perform worse at reconstructing vertex positions.Source: §1 · pp. 1–2
• The paper proposes a template-free, fully convolutional autoencoder for arbitrary registered meshes, including tetrahedral and non-manifold meshes.The architecture uses novel mesh convolution and (un)pooling operators.Source: §1 · p. 2
• Spatially varying kernels account for irregular mesh sampling and connectivity, with local kernels sampled from a jointly learned global weight basis and low-dimensional per-vertex functions.This reduces parameters compared with assigning a separate unconstrained kernel to every vertex.Source: §1 · p. 2
• The proposed autoencoder and its convolution and (un)pooling operators exceed state-of-the-art performance on D-FAUST, which includes large pose and local-detail variations.The reported comparison is on a dynamic 3D human-body dataset.Source: §1 · p. 2
• The fully convolutional autoencoder produces semantically meaningful localized latent codes that support better semantic interpolation and artistic manipulation than global latent features.The paper presents localized interpolation as a capability of its mesh autoencoder.Source: §1 · p. 2

2.1 Deep Learning Efforts on 3D Mesh Autoencoders

• Mesh autoencoder approaches using parameterized 2D images can handle high-resolution meshes, but distortion and seam artifacts remain a limitation.Prior methods parameterize meshes to UV space, spheres, or a torus before applying 2D CNNs.Source: §2.1 · p. 2
• On surface meshes, Neural3DMM outperformed CoMA [26] and previous works, but neither applies to volumetric or non-manifold meshes; quadric sampling can yield discontinuous receptive fields that break bottleneck-feature locality.Source: §2.1 · p. 3

2.2 Graph Convolution Networks

• Graph convolution operators vary in scope: MeshCNN is not directly suited to reconstruction, while SplineCNN relies on 3D-position-derived pseudo-coordinates.MeshCNN’s edge-based method is also less suitable for high-resolution data because edges outnumber vertices.Source: §2.2 · p. 3
• Unlike methods that derive local variation from input features, this approach uses input-independent local coefficients for graph irregularity and shared weight bases for feature statistics.The targeted irregularities include varying sampling density and vertex degree.Source: §2.2 · p. 3
• The listed graph convolution methods do not support transpose convolution, whereas this method does, addressing an operator need for up-sampling and reconstruction.The related-work discussion identifies transpose convolution and unpooling as necessary for compression and reconstruction.Source: §2.2 · p. 3

3 Method

• The proposed stack combines vcConv/vcTransConv, density-weighted pooling and unpooling, and residual up/down blocks into a fully convolutional mesh autoencoder.Source: §3 · p. 4, §3.2 · p. 5, §3.3 · p. 6

3.1 vcConv and vcTransConv

• Learned local coefficients sample vertex-specific convolution weights from a globally shared basis, reducing parameters and memory relative to LCConv.The basis weights are shared across the layer, while coefficients vary across neighborhood vertices.Source: §3.1 · pp. 4–5

3.2 vdPool, vdUnpool, vdUpRes, and vdDownRes

• vdPool/vdUnpool learn vertex densities for aggregation, with coefficient normalization preserving input-feature means; residual blocks pair these operators with convolution or transpose convolution.Source: §3.2 · p. 5

3.3 Fully Convolutional Autoencoder

• Latent vertices placed on body regions yield localized receptive fields, allowing interpolation at the right-arm vertex to alter that arm alone, unlike the less-localized quadric-simplification fields.Source: §3.3 · p. 6

4.1 Generality

• The model was tested on tetrahedral and non-manifold meshes, with reconstruction errors of 0.2 mm and 4.1 cm, respectively.The tetrahedral test used deformed dragon meshes; the non-manifold test used animated tree meshes.Source: §4.1 · p. 6, Figure 3 · p. 7
• At 0.25% compression, the high-resolution human-mesh test error was 3.01 mm for meshes with 154k vertices.The network’s bottleneck had 18 vertices and 64 dimensions per vertex.Source: §4.1 · p. 6, Figure 3 · p. 7

4.2 Localized Latent Feature Manipulation

• The model reconstructed D-FAUST hand gestures by interpolating selected latent vertices, and swapping a foot code transferred pose while preserving the recipient’s leg identity.Interpolation used latent vertices at the five fingertips and wrist; the foot-code swap produced a new foot pose for Man A.Source: §4.2 · p. 7, Figure 5 · p. 8

4.3 Comparison of 3D Mesh Autoencoder Models

• Our model achieves over 30% lower testing error and over 40% lower training error than Neural3DMM, with fewer training epochs and learning parameters.Both models use a 0.3% compression ratio; ours trained for 200 epochs and Neural3DMM for 300.Source: §4.3 · pp. 7–8
• Against MeshCNN, the comparison required down-sampling D-FAUST meshes to 750 vertices because full-resolution use was infeasible under memory and speed constraints.The MeshCNN reconstruction network still did not perform well after 200 training epochs.Source: §4.3 · p. 8

4.4 Comparison of Network Layers

• Increasing the proposed layers’ capacity lowered errors, while removing residual layers, replacing convolution with pooling, or normalizing weight bases increased errors.The experiments used the same D-FAUST setting and compared block designs, errors, parameter counts, and training GPU memory.Source: §4.4 · p. 8, Table 1 · p. 9
• Among convolution operators, LCConv had the lowest error but used around 135 times more parameters and twice the training GPU memory; other alternatives had higher errors than vcConv.The authors report vcConv as best for accuracy and memory consumption among the compared operators.Source: §4.4 · p. 9
• Replacing vcConv with average, max, or quadric pooling increased reconstruction error.The pooling comparison kept the vcConv layers unchanged.Source: §4.4 · p. 9

• The mesh autoencoder achieves state-of-the-art results for regressing arbitrary registered mesh types.Source: §5 · p. 9
• A shared kernel basis supplies natural analogs of classic 2D CNN operations while avoiding naive locally connected networks’ high parameter cost.Source: §5 · p. 9
• The method is the first mesh autoencoder to demonstrate localized interpolation.Source: §5 · p. 9
• Training graphs must share topology, so mesh connectivity must already be solved in the training set.Source: §5 · p. 9

• The work could potentially affect future entertainment and communication industries and allow more efficient storage and transport of 3D data.Source: §6 · p. 9

• The disclosure lists Chenglei Wu, Chen Cao, Yuting Ye, Jason Saragih, Yaser Sheikh, Yi Zhou, Zimo Li, and Hao Li with their stated institutional affiliations.Source: §7 · p. 10

A.1 Graph Sampling Algorithm

• For stride s sampling, the algorithm selects vertices per connected component, removes their (s−1)-ring neighbors, and searches s-ring vertices via a queue until all are marked.Users may manually mark vertices for inclusion before the search begins.Source: §A.1 · p. 13
• In down-sampling, output vertices correspond to sampled input vertices, and output edges connect pairs at distance at most 2s −1 in the input graph.Each sampled vertex’s neighborhood contains its r-ring neighbors for kernel radius r.Source: §A.1 · p. 13
• Up-sampling reverses the down-sampling graphs and stride, forming each output vertex’s neighborhood from its r-ring neighbors in the sampled graph and their corresponding input vertices.Source: §A.1 · p. 13

A.2 Network Implementation

• A vertex-connection table stores neighborhood indices, while coefficient, basis, and mask tensors encode convolution parameters and vacant entries.The table has N × Max(Ei) entries; coefficients have size N × Max(Ei) × M, and the basis has size M × O × I.Source: §A.2 · p. 13
• Networks in Sections 4.1 and 4.2 use residual blocks for down- and up-sampling, illustrated in Figure 8.Figure 8 is captioned as a residual block for down/upsampling.Source: §A.2 · p. 13, Figure 8 · p. 14
• Training and testing are GPU-parallelized in PyTorch, and kernels are precomputed using equation 2 for inference.Networks use batch size 16, learning rate 0.0001, and a 0.9 learning-rate decay every epoch.Source: §A.2 · p. 13
• Figure 9 depicts artifacts along seam lines in the UV-mapping method.Source: Figure 9 · p. 14

B Experiment Details

• The 959-vertex dragon tetrahedral mesh uses five down- and five up-sampling blocks, compressing to two vertices with 16 latent codes each.The mesh lies in a roughly 20 × 20 × 20 cm bounding box; all blocks use s = 2, r = 2, and M = 37.Source: B Experiment Details · p. 14
• The tree mesh has 328 disconnected components, which were connected by adding edges between close component pairs.Four down-sampling blocks compress its 20k vertices to 149 vertices, with eight latent codes per vertex; four blocks then up-sample.Source: B Experiment Details · p. 14
• The high-resolution human autoencoder compresses to 18 latent vertices with 64 dimensions each, a 0.75% compression rate.It uses six down- and six up-sampling blocks and L1 Laplacian loss.Source: B Experiment Details · p. 14
• The 57k-vertex hand experiment reached 1.06 mm mean point-to-point error and 0.036 mean L1 RGB error after 205 epochs.Meshes include coordinates and RGB color; the latent code uses six manually selected fingertip and wrist vertices, with 64 dimensions per vertex and 0.11% compression.Source: B Experiment Details · p. 14

C Additional Figures

• Figure 9 shows artifacts along seam lines in UV-mapping-based mesh CNN methods.Source: C Additional Figures · p. 14
• The dragon tetrahedral mesh originated at Stanford Computer Graphics Laboratory and was simulated using Vega.Source: C Additional Figures · p. 14
• The 3D tree model was downloaded from TurboSquid as the cherry-blossom-tree-3d-1189864 model.Source: C Additional Figures · p. 14