Source-linked AI summary
Deep Functional Maps: Structured Prediction for Dense Shape Correspondence
Or Litany, Tal Remez, Emanuele Rodolà, Alex M. Bronstein, Michael M. Bronstein
TL;DR
Dense correspondence for deformable 3D shapes is difficult when learning methods rely on descriptors or reference-domain labels and defer correspondence construction. This paper learns correspondences directly with a functional-map structured prediction model, achieving strong benchmark performance across challenging settings while remaining limited under extreme partiality and sensitive to topological changes.
Problem
The central gap is that prior learning approaches use descriptors or reference-domain labels instead of directly learning the correspondence between two shapes.
Method
A deep residual network takes dense descriptors on two shapes and outputs a soft correspondence through a functional-map structured prediction pipeline.
Results
The method consistently outperforms existing descriptor and correspondence learning methods across recent benchmarks, with improved global behavior after refinement over MoNet on synthetic shapes.
Takeaways & Limitations
The model-free functional-map framework adapts across shape categories and challenging data including synthetic models, real scans, acquisition artifacts, and partiality.
Takeaways & Limitations
Extreme partiality remains a failure case, while Laplacian eigenfunctions make topological changes particularly challenging.
Abstract
from arXiv · showhide
We introduce a new framework for learning dense correspondence between deformable 3D shapes. Existing learning based approaches model shape correspondence as a labelling problem, where each point of a query shape receives a label identifying a point on some reference domain; the correspondence is then constructed a posteriori by composing the label predictions of two input shapes. We propose a paradigm shift and design a structured prediction model in the space of functional maps, linear operators that provide a compact representation of the correspondence. We model the learning process via a deep residual network which takes dense descriptor fields defined on two shapes as input, and outputs a soft map between the two given objects. The resulting correspondence is shown to be accurate on several challenging benchmarks comprising multiple categories, synthetic models, real scans with acquisition artifacts, topological noise, and partiality.
1. Introduction
The paper targets dense correspondence for deformable 3D shapes by integrating correspondence computation into learning rather than relying on descriptors or reference-domain labels. Its structured prediction model takes descriptors as input, returns matches end-to-end, and outperforms existing learning methods across benchmarks.
- Motivation: Real-world 3D scanning artifacts and non-rigid motion make fast, reliable shape correspondence difficult.The motivation includes applications ranging from Kinect gaming to LIDAR-based autonomous cars.
- Limitations of prior learning approaches: Descriptor-based methods can produce inaccurate solutions, while label-space methods require fixed point counts and an intermediate reference model.Descriptor-focused learning may also require costly post-processing to obtain accurate correspondences.
- Proposed approach: The proposed task-driven approach includes correspondence computation directly in the learning procedure.Neural networks provide the flexible computational structure used to integrate this operation.
- Contributions: The framework introduces a structured prediction model that supports end-to-end training from base descriptors to matches.The model is designed to return correspondences rather than only intermediate descriptors.
- Results: The approach consistently outperforms existing descriptor and correspondence learning methods on several recent benchmarks.This is the paper’s stated comparative outcome across its evaluations.
2. Related work
Prior work includes functional-map methods, descriptor learning, and correspondence learning through label spaces or neural architectures. The paper positions its contribution as using functional maps as a core component of correspondence learning rather than mainly as post-processing.
- Functional maps: Functional maps represent correspondences as linear operators between function spaces and compact matrices in truncated Laplacian eigenbases.With k much smaller than the number of points n, the representation reduces optimization variables.
- Functional maps: Functional maps have usually been used for post-processing initial correspondences, but rarely as building blocks in correspondence learning pipelines.This identifies the specific gap addressed by the paper.
- Descriptor learning: Descriptor-learning methods learn spectral descriptors or descriptor-combination weights, whereas this paper optimizes a map in the spatial domain.The distinction is framed relative to prior functional-map-based descriptor learning.
- Correspondence learning: Correspondence learning has used random forests and CNN-based architectures to learn invariance to pose changes, clothing, or surface structure.The related approaches include shallow random forests, extrinsic CNNs, and geodesic CNNs.
3. Background
Functional maps represent shape correspondences as compact linear operators between function spaces, typically expressed in truncated Laplacian eigenbases. This contrasts with label-based approaches that classify points relative to an intermediate reference shape and construct correspondences afterward.
- Manifolds: Shapes are modeled as two-dimensional Riemannian manifolds with square-integrable functions and Laplace-Beltrami eigenfunctions forming an orthonormal basis.The Laplacian eigenvalues form a discrete spectrum, and the eigenfunctions support Fourier-style function expansions.
- Functional correspondence: A functional map is a linear operator T: L2(X) → L2(Y) that maps functions on one shape to functions on another.Point-to-point matching is recovered as the special case in which delta functions map to delta functions.
- Functional correspondence: Using Laplacian eigenfunctions and truncating after k coefficients yields a compact k × k representation, typically with k ≪ n.The truncated representation is band-limited and reduces the dimensionality relative to the number of shape points.
- Functional correspondence: Functional correspondence solves for the matrix C from corresponding functions represented by their Fourier coefficients, with dense descriptor fields such as HKS or SHOT used in practice.The descriptor fields provide the corresponding functions whose spectral representations enter the functional-map problem.
- Label space: Label-based methods assign each query vertex a reference-shape index, then compose maps through that reference domain or solve a least-squares problem using learned descriptors.This formulation treats correspondence as pointwise classification and constructs the full map a posteriori.
4. Deep Functional Maps
Deep Functional Maps learns descriptors for the correspondence task itself by coupling a residual network with functional-map and soft-correspondence computations. Its geometrically structured loss weights errors by geodesic distance rather than treating all classification mistakes equally.
- Deep Functional Maps: The model learns pointwise descriptors whose use in a functional-map pipeline induces accurate correspondences, rather than optimizing descriptors independently of map recovery.The network takes manually designed descriptors as input and improves them under a geometrically meaningful criterion.
- Soft correspondence: The soft correspondence matrix represents a rank-k spatial correspondence approximation whose entries can be interpreted as probabilities of mapping points between shapes.Columns are normalized and interpreted as probability distributions over the target shape.
- FMNet architecture: FMNet applies shared-weight residual processing to descriptor fields, projects refined descriptors into Laplacian eigenbases, and then computes functional and soft correspondence maps.The functional-map and soft-correspondence layers are non-parametric and support the structured loss.
- Structured loss: The soft error uses probability-weighted geodesic distance, favoring nearby mismatches over distant ones.This gives the loss a geometric criterion for mapping nearby source points to nearby target points, unlike classification costs that treat deviations equally.
- FMNet architecture: The functional map is obtained by solving a least-squares problem in closed form as C = ˆGˆF† before computing the soft correspondence.The pseudo-inverse provides the closed-form solution used by the non-parametric layer.
5. Implementation details
The implementation combines SHOT inputs, a residual network, low-resolution functional-map prediction, and an upscaling procedure for dense full-resolution correspondence. The reported comparison emphasizes that descriptor proximity alone need not produce a good functional correspondence.
- Data: Input shapes are down-sampled to 15K vertices when necessary, and each vertex receives a 352-dimensional SHOT descriptor.Meshes below 15K vertices remain at full resolution; SHOT is selected for fast computation and local robustness to missing parts.
- Network: The network uses seven fully connected residual layers with ELUs and no dimensionality reduction, with dataset-dependent minibatch counts.Training uses 20K, 100K, or 1K minibatches for FAUST synthetic, FAUST real scans, and SHREC’16, respectively.
- Upscaling: The network predicts a k × k functional map on down-sampled shapes, which cannot directly recover a pointwise map on the corresponding full-resolution shapes.An explicit upscaling stage is therefore required.
- Evaluation: Siamese descriptors improve proximity-based CMC but do not necessarily induce good functional correspondence, whereas FMNet yields a noticeable accuracy gain on that task.The comparison is made using CMC and geodesic error.
- Upscaling: Upscaling transfers low-resolution points by nearest-neighbor searches in R3 and composes the resulting map with projections between low- and full-resolution shapes.The composed map is initially sparse on the full-resolution source shape.
- Upscaling: A convex, nonsmooth ℓ2,1 minimization discards potential mismatches before nearest-neighbor recovery of a dense full-resolution point-to-point map.The optimization uses Fourier coefficients of corresponding delta functions and can be solved globally with ADMM-like techniques.
6. Results
FMNet is evaluated through qualitative and quantitative comparisons across synthetic, real-scan, cross-dataset, and partial-shape correspondence benchmarks. The experiments show strong correspondence quality, generalization, and global behavior across varied matching scenarios.
- Experimental setup: Experiments compare FMNet qualitatively and quantitatively with state-of-the-art methods across multiple matching scenarios.The evaluation includes recent benchmarks covering different datasets and correspondence settings.
- Metric learning: The structured prediction model learns descriptors that are optimized for dense correspondence, outperforming metric-learning descriptors despite their ability to attract matches and repel mismatches.Metric-learning descriptors perform well for descriptor retrieval but not when used to compute dense correspondence through the functional-map pipeline.
- Real scans: On FAUST real scans, FMNet considerably improves upon a pose-prior method while using far less training data than a slightly better human-specific model.The competing stitched-puppet model uses 6000 meshes, whereas FMNet is trained on almost two orders of magnitude less data and is applicable to any shape category.
- Transfer: FMNet generalizes from FAUST training data to SCAPE, where it is compared against methods trained directly on SCAPE.This cross-dataset evaluation tests transfer beyond the training benchmark.
- Synthetic shapes: On FAUST synthetic shapes, FMNet significantly outperforms MoNet after refinement, producing almost ideal correspondence with zero error.Although MoNet has higher raw accuracy at zero, FMNet’s error curve shows better global behavior after refinement.
- Partial non-human shapes: FMNet also handles partial non-human shapes when partiality is represented in the training data, as demonstrated on the SHREC’16 benchmark.The dog class from the holes sub-challenge includes missing parts of varied forms and sizes, with qualitative results reported for the test shapes.
7. Discussion and conclusions
The method directly estimates dense correspondences within a functional-maps framework and performs well across challenging shape categories and datasets. Its main limitations are sensitivity to topological changes and failure under extreme partiality.
- Discussion and conclusions: The proposed neural network directly estimates dense shape correspondences within the functional maps framework, avoiding correspondence-agnostic descriptor learning.The method is structured for end-to-end correspondence estimation rather than relying on post-processing after descriptor learning.
- Discussion and conclusions: The approach performs well across synthetic models, real scans with acquisition artifacts, partiality, different shape categories, and cross-dataset generalization.The paper reports adaptation to categories such as dogs and generalization between different datasets.
- Discussion and conclusions: Laplacian eigenfunctions are sensitive to topological changes, making such examples more challenging for the method.The authors suggest that choosing a different basis may mitigate this limitation.
- Discussion and conclusions: The method recovers most pairs with missing parts but fails under extreme partiality.The authors propose incorporating partiality priors into the structured prediction model to address this failure.