Source-linked AI summary
Deep Geometric Functional Maps: Robust Feature Learning for Shape Correspondence
Nicolas Donati, Abhishek Sharma, Maks Ovsjanikov
TL;DR
Shape correspondence methods often require extensive training data or depend on handcrafted descriptors that generalize poorly across shape structures. This paper learns features directly from raw 3D geometry and combines them with a regularized functional-map extraction layer and loss. The resulting pipeline generalizes better, remains robust with limited training data, and achieves strong correspondence accuracy.
Problem
Learning-based shape correspondence methods require substantial training data or handcrafted descriptors, limiting robustness and generalization across diverse shapes.
Method
A hybrid network learns pointwise features from raw 3D geometry and uses a regularized functional-map layer with a spectral loss.
Results
The method generalizes well with few training shapes, including state-of-the-art SHREC re-meshed performance with average error 0.048 using 500 training shapes.
Takeaways & Limitations
Raw-geometry feature learning combined with functional-map regularization enables accurate dense correspondence without relying on a template or a priori parametric model.
Takeaways & Limitations
As a supervised method, the approach requires at least partial training correspondences and costly, potentially unstable data augmentation for non-aligned shapes.
Abstract
from arXiv · showhide
We present a novel learning-based approach for computing correspondences between non-rigid 3D shapes. Unlike previous methods that either require extensive training data or operate on handcrafted input descriptors and thus generalize poorly across diverse datasets, our approach is both accurate and robust to changes in shape structure. Key to our method is a feature-extraction network that learns directly from raw shape geometry, combined with a novel regularized map extraction layer and loss, based on the functional map representation. We demonstrate through extensive experiments in challenging shape matching scenarios that our method can learn from less training data than existing supervised approaches and generalizes significantly better than current descriptor-based learning methods. Our source code is available at: https://github.com/LIX-shape-analysis/GeomFmaps.
1. Introduction
Shape correspondence matters across several applications, but learning-based methods often need substantial training data and depend on handcrafted descriptors that weaken robustness and generalization. The proposed approach learns from raw 3D geometry and combines this with regularized functional-map extraction.
- Shape correspondence supports applications including texture or deformation transfer and statistical shape analysis.
- Learning-based correspondence methods typically require large training datasets to generalize across diverse shape classes.
- Handcrafted descriptors can make learned models less robust and less accurate, reducing generalization to new shape classes or instances.
- The method learns features directly from 3D shapes and combines them with strong regularization through a novel spectral correspondence extraction layer.
- A functional-map spectral loss in a reduced basis reduces overfitting while preserving accurate correspondences and supporting limited training data.
2. Related Work
Prior correspondence methods span handcrafted, learned, spectral, spatial, and raw-geometry approaches. A central limitation is that functional-map pipelines depend on input descriptors, which can omit information and generalize poorly across datasets.
- Functional maps represent correspondences with small matrices in reduced bases, simplifying the associated optimization problems.
- Functional-map estimation relies strongly on input descriptors, and regularizers or robust penalties do not resolve severe inconsistencies in difficult cases.
- Descriptor-learning methods transform handcrafted features such as SHOT, Heat, or Wave kernel signatures rather than learning directly from raw geometry.
- Input features can omit information and vary across datasets, causing poor cross-dataset generalization; SHOT is especially sensitive to mesh connectivity.
- Spatial alternatives include mesh convolutions, depth-map learning, feature-space methods, and point-cloud approaches such as 3D-CODED.
- The proposed hybrid approach combines spectral and spatial strategies, achieving accurate results with little training data on a challenging human-shape benchmark.
3. Shape Matching and Functional Maps
The functional-map pipeline computes a compact spectral representation from two meshes, aligns descriptor functions, optimizes a map, and converts it to pointwise correspondence. Deep variants learn descriptor transformations, while this work combines raw point-cloud features with functional-map structure.
- Given two triangle meshes, the pipeline computes a map between their vertices through four main functional-map steps.
- It computes the first few Laplace–Beltrami eigenfunctions on each shape to form reduced spectral bases.
- Descriptor functions are represented by coefficient matrices A and B in the respective spectral bases.
- The optimal functional map balances descriptor preservation with regularization of the map’s structural properties.
- The optimization has kM × kN unknowns, independent of the number of points on the shapes.
- The estimated spectral map is converted into a point-to-point map and can then be refined iteratively.
- Deep functional-map methods learn descriptor transformations, but they remain dependent on handcrafted input features and supervised or structural map losses.
- 3D-CODED learns spatial features with a PointNet-based variational autoencoder but relies on a deformable template and extensive supervised training data.
4. Method
The method jointly learns point-wise features directly from input shape geometry and extracts robust functional maps through a differentiable, regularized layer. Spectral losses and functional-map supervision reduce sensitivity to descriptor inconsistencies and support accurate correspondence learning with limited data.
- Feature learning: A Siamese point-cloud network learns point-wise features directly from the source and target shapes using shared parameters.The feature extractor uses point-cloud convolutions and is applied with the same weights to both shapes.
- Feature learning: The feature extractor takes only point clouds, making it robust to changes in mesh connectivity.This design separates feature learning from the mesh structure.
- Functional-map extraction: The learned features are projected into spectral bases and used to compute a functional map through the differentiable FMReg layer.The spectral descriptors are formed using basis projections and Moore pseudoinverses before map estimation.
- Functional-map extraction: FMReg solves regularized linear systems for each functional-map row, preserving differentiability while improving robustness to low-dimensional spectral descriptors.With k eigenvectors, the operation inverts k different k × k matrices.
- Spectral supervision: The supervised spectral loss penalizes the Frobenius distance between the predicted and ground-truth functional maps.Functional-map supervision avoids converting predictions to soft pointwise maps and avoids the associated geodesic distance matrix computation.
- Spectral supervision: Functional-map ground truth can be obtained from pointwise maps or sparse landmarks, including partial template-to-remeshed-shape correspondences.This representation is useful when complete bijective correspondences are unavailable.
- Inference and refinement: After training, the predicted functional map can be converted directly to point-to-point correspondence or refined with ZoomOut.ZoomOut progressively increases the number of spectral basis functions while navigating between spectral and spatial domains.
5. Results
The experiments evaluate generalization across datasets, training-set sizes, and remeshed geometry. The method performs strongly with limited training data and maintains robustness where competing methods degrade.
- Experimental settings: The evaluation spans remeshed FAUST, SCAPE, and SHREC19, including cross-dataset tests and synthetic SURREAL training with 100–5000 shapes.These settings probe dataset transfer, training-data dependence, and adaptation from synthetic triangulations to challenging remeshed shapes.
- Experiment 1: Our method performs best overall on Experiment 1 and gives good results across settings even without refinement.The comparison includes automatic spectral methods, FMNet variants, SURFMNet, and supervised 3D-CODED.
- Experiment 2: 0.048 average error on SHREC remeshed is achieved with only 500 training shapes, yielding state-of-the-art results.Results in Figure 3 compare the method with 3D-CODED under varying SURREAL training sizes; reported values are multiplied by 100.
- Experiment 2: The method’s results remain consistent as training shapes decrease, whereas 3D-CODED consistently suffers from reduced training data.This directly tests dependence on training-set size.
- Qualitative results: With 2000 training shapes, both the method and 3D-CODED produce good or excellent texture transfers, but with 100 shapes 3D-CODED often fails.Fully spectral methods fail in the 2000-shape setting because training and test connectivity differ.
6. Conclusion, Limitations & Future Work
The paper concludes that a hybrid raw-geometry and functional-map network improves robustness and reduces overfitting. It also identifies supervision and costly data augmentation as limitations, with unsupervised and invariant alternatives proposed for future work.
- Conclusion: The hybrid network combines a raw-data feature extractor with a robust functional map layer operating across spatial and spectral domains.The stated goal is improving robustness and reducing overfitting in learned shape correspondences.
- Limitations: The supervised approach requires at least partial correspondences between training shapes.This is identified as a primary limitation of the method.
- Limitations: Predicting non-aligned shapes requires data augmentation, which can be costly and unstable.The limitation is tied to the method’s handling of non-aligned inputs.
- Future work: Future work targets an unsupervised spectral loss that avoids symmetry ambiguity and invariant feature extractors that reduce reliance on data augmentation.These are proposed directions rather than demonstrated capabilities of the present method.
A . Additional details on KPConv [47]
The appendix describes the point-cloud convolutional feature extractor used by the method. KPConv processes local point-cloud signals with learnable kernels and hierarchical sampling, followed by a specified encoder–decoder architecture.
- Inputs: The network takes a 3D point cloud P ∈ R^N×3 with a D-dimensional feature signal F ∈ R^N×D.The signal may include the 3D coordinates of points.
- Operators: KPConv transfers convolution and pooling operations from image networks to point-cloud structures.Convolution produces signals over points, while pooling adjusts point-cloud density.
- Kernel: Each local kernel uses a radius-r 3D ball, K kernel points, and learnable matrices W_k mapping D-dimensional inputs to D′-dimensional outputs.The kernel is centered locally and depends on learnable parameters.
- Architecture: The feature extractor uses four strided convolutional blocks that halve point density while doubling feature-space size at each block.This builds progressively higher-level features over downsampled point clouds.
- Architecture: Four up-sampling layers reconstruct signals on the full point cloud through skip connections and 1D convolutions.The architecture therefore combines hierarchical downsampling with feature recovery.
- Ablation study: Table 2 reports ablations intended to demonstrate that all individual building blocks are needed for optimal performance.The paper describes the study as evaluating the importance of each component.
B . Ablation study
The ablation study evaluates the feature extractor, sampling strategy, regularized functional-map layer, and refinement step. All components contribute to optimal performance, while regularization notably accelerates convergence and improves robustness.
- The ablations use 100 random 3D-CODED shapes for training and 20 re-meshed FAUST shapes for testing, changing connectivity between train and test.
- The feature-extractor ablation replaces the raw point-cloud extractor with FMNet and SHOT descriptors, using the same 30 eigenvectors.
- The sampling ablation replaces KPConv grid sampling with PointNet and random sampling of 1500 points because PointNet cannot handle adaptable batch sizes.
- Regularization makes models converge much faster, reaching optimal results within 500 epochs, and avoids fatal Cholesky-decomposition errors seen with the previous layer.
- All individual components are needed to achieve optimal performance in the ablation study.
- The regularized functional-map layer produces maps that nearly commute with shape Laplacians, restricting them toward diagonal matrices in the Laplacian eigenbasis.
C . More quantitative results
Across two experiments and varied training settings, the method generally achieves the best accuracy against state-of-the-art baselines while learning from limited data. Its template-free spectral approach also supports generalization, although back-bent SCAPE shapes remain difficult.
- The method achieves the best results in all but one evaluated setting, including comparisons with state-of-the-art methods.The exception is training on 2000 SURREAL shapes and testing on re-meshed SCAPE.
- The results show that accurate correspondences can be learned from point-cloud characterizations using only a small amount of training data.
- The template-free method is more general than 3D-CODED, trains faster, and does not require a large training set.
- The method’s low-data advantage is partly attributed to being template-free and partly spectral, which facilitates adaptation to new 3D-shape categories.
- Back-bent SCAPE shapes cause relatively low performance because truncated spectral approximations omit refined facial and hand features, producing frequent left-to-right symmetric matches.These shapes are absent from the training sets, increasing the mean test error.
D . Visualization of some descriptors learned by our method
The learned descriptors are formed from raw point clouds, projected into shape-specific Laplace–Beltrami eigenbases, and visualized as localized spectral features. Grid sampling and intrinsic spectral projections help transfer descriptors across changed connectivity, while limited bases omit fine details.
- Descriptors are learned from raw source and target point clouds and projected onto the respective Laplace–Beltrami eigenbases.The projections are called spectral descriptors.
- The pipeline’s quantitative visualizations include results for different methods across all settings of Experiment 1 and two settings of Experiment 2.
- Grid sampling regularizes point-cloud inputs across hierarchies, while spectral projection transfers descriptions into an intrinsic surface space less dependent on mesh connectivity.
- The learned spectral descriptors are localized and highlight specific body regions, including shoulders, scalp, thigh, torso, and elbow.
- Intrinsic translation helps distinguish geometrically nearby components, such as the two arms of a shape.
- With 30 eigenvectors, descriptors capture coarse human segmentation such as head, arms, body, and legs, but not small hands, feet, or facial features.More than 100 eigenvectors may expose finer descriptors, but could increase overfitting.
E . Additional Texture transfer on SHREC’19 re-meshed
Additional texture-transfer results compare the method with 3D-CODED, FMNet, and Unsupervised FMNet under low- and high-data settings. The method handles connectivity changes and a challenging rotated pose better than baselines, while the pipeline combines sampling, spectral projection, map computation, and loss evaluation.
- The qualitative comparison evaluates the method with and without ZoomOut against 3D-CODED, FMNet, and Unsupervised FMNet using 2000 and 100 training shapes.
- FMNet fails under the connectivity change between training and test shapes, as also reflected in the quantitative results.
- On a challenging bent and slightly off-axis rotated pair, 3D-CODED reconstructs poorly even with 2000 training shapes, whereas the proposed method performs well with both data regimes.
- The pipeline downsamples source and target shapes, learns point-cloud characterizations, projects them into Laplace–Beltrami eigenbases, computes a regularized functional map, and compares it with ground truth.