PaperlayerView exact revision on arXiv

Source-linked AI summary

TokenMatch: 3D Mesh Correspondence Transformer with Curvature-Guided Tokenisation

Adeela Islam, Zorah Lähner, Vittorio Murino, Vladislav Golyanik

arXiv:2609.04202v1cs.CV

TL;DR

TokenMatch tackles the difficulty of matching partially observed, strongly non-isometric 3D shapes, where existing approaches face representation, generalisation, and inference limitations. It uses curvature-guided mesh patches with self- and cross-attention in a feed-forward functional-map model, and reports strong performance across partial and full benchmarks, including full-shape matching after partial-only training. The paper also identifies geodesic computation at high mesh resolutions as a practical limitation.

  • Problem

    Robust 3D correspondence remains challenging under partial observations and strong non-isometric deformations, while existing methods often rely on restrictive representations or have limited generalisation and efficiency.

  • Method

    TokenMatch uses curvature-guided overlapping mesh patches as tokens, transformer self- and cross-attention, functional maps, and overlap prediction in a unified feed-forward model.

  • Results

    TokenMatch consistently improves over prior methods across standard partial and full matching benchmarks, generalises from partial-only training to full-shape matching, and runs in a fraction of a second per shape pair.

  • Takeaways & Limitations

    Curvature-guided geometry-native representations combined with transformer global reasoning support a unified, transferable approach to partial and full 3D shape correspondence.

  • Takeaways & Limitations

    Geodesic computations can become expensive at mesh resolutions higher than those common in the current datasets.

Abstract

from arXiv · show

While data-driven 3D shape correspondence estimation has recently seen substantial progress, robust matching under partial observations and strong non-isometric deformations remains challenging. Existing learning-based approaches often rely on hand-crafted descriptors or template-based representations, whereas recent generative models over functional maps suffer from high inference cost, limited interpretability, and poor generalisation to partial shapes. In response to these limitations, this paper introduces TokenMatch, a new transformer-based unified model for estimating 3D shape correspondences. Our feed-forward approach trained exclusively on BeCoS, a challenging non-isometric partial-to-partial shape-matching dataset, can generalise to matching full shapes without retraining or fine-tuning. TokenMatch uses self- and cross-attention mechanisms to efficiently learn patch-level and point-level relations as well as dense correspondences between shape pairs. Our core insight is that meshes can be adaptively tokenised into patches using shape curvature guidance, enabling effective learning of shape-specific geometric descriptors for correspondence estimation. We evaluate TokenMatch on standard benchmarks for partial and full shape matching, including CP2P, PSMAL, BeCoS, FAUST, SCAPE, and SHREC'19. Our method achieves consistently high performance, in most cases outperforming existing methods for partial and full shape matching in the mean geodesic error and intersection-over-union metrics, while also running faster at sub-second inference speeds.

1 Introduction

TokenMatch addresses partial and non-isometric 3D shape correspondence with a unified transformer model built around curvature-guided mesh tokenisation. Trained on diverse partial data, it targets cross-setting generalisation and efficient correspondence estimation.

  • Motivation: Partial-shape correspondence remains difficult because missing geometry enlarges the solution space, requires overlap identification, and exposes weaknesses in complete-shape or near-isometric training assumptions.These constraints also limit robustness to unseen deformations, artefacts, and transfer between partial and full matching.
  • TokenMatch: TokenMatch combines a transformer encoder with self-attention, cross-attention, functional maps, and overlap prediction for unified 3D correspondence estimation.The model is designed to reason over variable-sized shape elements and dense inter-shape relations.
  • TokenMatch: Curvature-guided tokenisation represents meshes as overlapping, geometry-aware surface patches that capture local detail and intrinsic shape structure.These patches serve as transformer tokens and are intended to preserve geometric cues under irregular sampling.
  • Training and scope: Training on the diverse BeCoS dataset supports template-free, cross-category generalisation across partial and full matching settings.The paper reports evaluation and cross-category zero-shot generalisation across more than 2500 unique shapes.
  • Efficiency and scope: TokenMatch is a feed-forward model with sub-second inference and is presented as applicable across diverse shape categories without category-specific assumptions.Its variable token count allows full-shape matching without necessarily requiring full-shape retraining or fine-tuning.

2 Related Work

Prior correspondence methods span functional maps, combinatorial and unsupervised partial matching, transformers, and generative models. TokenMatch positions itself as a template-free, curvature-aware feed-forward alternative that combines spectral structure with global transformer interactions.

  • Functional maps: Functional-map methods provide compact spectral representations, but their global low-rank structure can limit partial-shape correspondence.The supplied related-work passage introduces this limitation before discussing partial-shape extensions.
  • Partial-shape matching: Partial matching requires both pointwise correspondence estimation and identification of overlapping regions, while combinatorial methods scale poorly and unsupervised methods remain sensitive to topological noise.Functional-map extensions address efficiency but inherit challenges associated with partial geometry.
  • Transformers: Transformers model long-range dependencies and global structure through token interactions, making them attractive for variable and geometrically diverse 3D inputs.Existing correspondence models may use vertex tokens or learned latent spaces that obscure fine-grained geometry under partial or non-uniform sampling.
  • TokenMatch positioning: TokenMatch uses curvature-driven mesh tokenisation to preserve intrinsic geometric cues while combining functional maps with transformer-based cross-shape interactions.The paper presents this combination as a unified framework for full and partial correspondence.
  • Generative models: Unlike diffusion-based correspondence methods, TokenMatch uses an efficient feed-forward transformer and is described as orders of magnitude faster than slow guided DDPM generation.The comparison also identifies TokenMatch as template-free.

3 Definitions and Background

The paper represents pointwise mesh correspondences through compact functional maps in Laplace–Beltrami spectral bases. Learned descriptors are projected into that basis, and the map is optimized with spectral consistency constraints.

  • Definitions: The target is a pointwise mapping between triangular meshes, but direct binary correspondence estimation is computationally expensive, motivating a reduced spectral representation.In partial matching, the mapping is defined only on a subset of the source shape.
  • Functional maps: For each shape, the method computes Laplace–Beltrami eigenvectors and eigenvalues, using the first k eigenvectors as the spectral basis.The eigenvectors satisfy LSΦS = ΦSΛS.
  • Functional maps: Learned feature descriptors are projected into the spectral domain to obtain the coefficient representations used by the functional map.The projection uses the Moore–Penrose pseudoinverse of the basis matrix.
  • Functional maps: The functional map is learned through an optimization objective that includes a Frobenius-norm term and a commutativity constraint with the Laplace–Beltrami operators.The stated goal is to learn descriptors that induce accurate functional maps.

4 TokenMatch: Our 3D Mesh Correspondence Transformer

TokenMatch directly tokenises meshes into curvature-guided, overlapping geometry patches and processes them with transformer attention for unified shape correspondence estimation. Its pipeline combines adaptive token construction, masked pre-training, contextual feature learning, and cross-shape overlap reasoning.

  • 4 TokenMatch: Our 3D Mesh Correspondence Transformer: TokenMatch directly processes tokenised mesh geometry in a transformer framework for unified partial and full 3D shape correspondence estimation.The design avoids fixed spectral representations and handcrafted descriptors while modelling local surface geometry and global shape context.
  • 4.1 Geometry-Aware Curvature-Guided Mesh Tokenisation: Curvature-guided sampling combines local and spectral geometry to place token centres using signal-weighted geodesic farthest point sampling.The spectral signal uses mid-frequency eigenfunction energy alongside absolute mean curvature, with β = 0.5 controlling geometry-signal influence.
  • 4.1 Geometry-Aware Curvature-Guided Mesh Tokenisation: Soft geodesic assignments create overlapping token regions whose weighted aggregation improves robustness to varying mesh resolution, sampling density, irregular tessellations, and discretisation noise.The bandwidth σ controls assignment diffuseness, while increasing the token count reduces centre spacing and enhances overlap.
  • 4.2 Transformer Backbone: Token features combine softly aggregated geometric descriptors with centre-coordinate embeddings before transformer layers produce contextualised shape representations.Each token representation is formed as x_i = e_i + p_i, with p_i obtained by applying an MLP to the sampled centre coordinate.
  • 4.2 Transformer Backbone: Masked auto-encoding pre-training reconstructs missing geometric structures from visible tokens, encouraging robust representations that support partial-to-partial, full-to-full, and partial-to-full matching.The decoder reconstructs masked geometric embeddings using geometric and feature-level reconstruction objectives.
  • 4.4 Shape Overlap Prediction: Cross-attended features support overlap prediction by using contextual interactions between the two shapes rather than independently extracted point features.This module leverages cross-shape information to estimate shared regions, complementing the correspondence-aware transformer pipeline.

5 Experimental Evaluation

TokenMatch is evaluated across partial and full 3D shape correspondence benchmarks, including challenging non-isometric, anisotropically remeshed, and cross-category settings. It achieves strong accuracy and generalisation while retaining sub-second inference.

  • Partial-shape matching: TokenMatch outperforms all baselines across the CP2P, PSMAL, and BeCoS partial-to-partial benchmarks.The improvements are especially pronounced on BeCoS, where geometric variability and missing regions make matching difficult.
  • Full-shape matching: TokenMatch achieves consistently strong full-shape performance on FAUST, SCAPE, and SHREC’19, with particularly notable gains on SHREC’19.It remains competitive with the strongest methods while generalising from partial-to-partial training without additional training.
  • Robustness to anisotropic remeshing: TokenMatch remains stable and consistently accurate under anisotropic remeshing, whereas most competing methods degrade with irregular discretisation.The authors associate this robustness with curvature-guided tokenisation and its geometry-independent handling of sampling variation.
  • Generalisation across categories: Despite training exclusively on partial-to-partial correspondences, TokenMatch handles full-to-full and partial-to-full BeCoS matching under severe partiality and deformation.The model achieves the lowest mean geodesic error across all but one SMAL category.
  • Training and inference time: TokenMatch requires approximately six hours of BeCoS training and 0.16 seconds per shape pair at inference, faster than DPFM, EchoMatch, and optimisation-based methods.The reported inference time includes overlap prediction, functional map recovery, and nearest-neighbour point-match recovery.
  • Masked pre-training and robustness to noise: Masked pre-training improves matching under incomplete geometry, while TokenMatch remains robust to moderate inference noise without noise augmentation during training.These findings are reported in the appendix robustness studies.

6 Conclusion

TokenMatch provides a unified, efficient framework for partial and full 3D shape matching, with strong robustness across challenging settings. Its curvature-guided tokenisation is central to the reported accuracy, while geodesic computation remains a scalability constraint.

  • TokenMatch consistently improves over prior methods across standard partial and full-shape benchmarks while remaining robust to partiality, non-isometric deformations, and geometric complexity.The method is reported to outperform local descriptor-based approaches particularly in challenging partial correspondence scenarios.
  • Training exclusively on partial correspondences supports strong full-shape matching at inference without architectural modification or retraining.Full-to-full matching is treated as a special case of partial-to-partial correspondence estimation.
  • Geodesic computations can become expensive at mesh resolutions higher than those common in current datasets.Future work targets scalable alternatives, noisy unstructured geometry, and larger real-world scan collections.

Supplementary Material

The supplementary material documents evaluation protocols, tokenisation ablations, hyperparameter sensitivity, noise robustness, and additional qualitative results. These analyses extend testing across partiality, non-isometric deformation, and incomplete geometry.

  • Appendix A details dataset preprocessing and evaluation protocols for partial and full-shape matching benchmarks.
  • Supplementary experiments analyze tokenisation alternatives, cross-attention ablation, token overlap, token count, masking ratio, and noisy inputs.
  • Additional qualitative results show geometrically consistent correspondences under limited overlap, incomplete local structure, noise, and missing regions.

A Dataset Split Details

The evaluation spans partial-to-partial benchmarks CP2P, PSMAL, and BeCoS, plus full-shape datasets FAUST, SCAPE, and SHREC’19. The appendix also compares mesh tokenisation alternatives as supplementary experimental context.

  • Partial matching is evaluated on CP2P, PSMAL, and BeCoS, while full matching additionally uses FAUST, SCAPE, and SHREC’19.
  • CP2P24 evaluates 100 test pairs sampled from 153 shapes in the CUTS24 test split, with 120 SHREC16 CUTS training shapes.
  • PSMAL uses species-disjoint splits over 49 shapes, producing 273 training pairs and 100 test pairs across eight animal categories.
  • BeCoS provides 10,185 training instances, 137 validation instances, and 142 test instances for realistically scaled non-isometric partial shapes.
  • FAUST uses 100 human meshes with an 80/20 training-testing split, while SCAPE uses 51 training and 20 testing meshes.
  • SHREC’19 is test-only, evaluating all available shapes except non-watertight shape 40.

B.1 Tokenisation Design Exploration

The design exploration compares global, intrinsic, local, and geodesic tokenisation strategies before selecting curvature-guided overlapping patches. Results indicate that curvature-aware mesh tokens and overlapping context improve correspondence performance over alternatives and point-cloud representations.

  • B.1 Tokenisation Design Exploration: Spectral clustering captures global structure but may disconnect regions, HKS produces diffuse patches, and hierarchical splitting lacks global shape awareness.
  • B.1 Tokenisation Design Exploration: The proposed curvature-guided overlapping tokenisation consistently outperforms alternative schemes across CP2P24, PSMAL, and BeCoS with the architecture and learned features fixed.The experiment isolates tokenisation as the varying component.
  • B.1 Tokenisation Design Exploration: Curvature-aware tokenisation concentrates token centres around articulations and boundaries, whereas point clouds provide less structured coverage of high-curvature regions.
  • B.1 Tokenisation Design Exploration: Curvature-guided tokens allocate capacity to geometrically informative regions, while overlap preserves contextual continuity across neighbouring patches.This design reduces sensitivity to token boundaries and aligns token placement with local geometric variation.
  • B.1 Tokenisation Design Exploration: Point-cloud inputs consistently reduce performance relative to meshes, including CP2P24 accuracy falling from 85.56% to 81.05% and PSMAL accuracy from 85.21% to 80.12%.The point-cloud variant uses the same transformer architecture, training protocol, and correspondence objective.

B.2 Effect of Cross-Shape Interaction

Cross-attention improves correspondence by enabling direct interaction between source and target tokens, while tokenisation choices control the balance between locality, overlap, and geometric detail. Qualitative results further show transfer across partiality, shape completeness, and challenging deformations.

  • B.2 Effect of Cross-Shape Interaction: Accuracy falls from 85.56% to 83.94% on CP2P24, from 85.21% to 83.01% on PSMAL, and from 65.25% to 63.96% on BeCoS without cross-attention.The ablation identifies explicit source–target interaction as important for correspondence estimation.
  • B.2 Effect of Cross-Shape Interaction: Cross-attention exchanges source and target features, enabling joint correspondence hypotheses that disambiguate partial or ambiguous matches.Self-attention supports intra-shape reasoning but does not model all cross-shape correspondence associations.
  • Tokenisation: Moderate token overlap, achieved with σ = 0.15, yields the best performance, whereas very small or large bandwidths respectively cause brittle assignments or over-smoothed geometry.The bandwidth controls the spatial extent and overlap of soft geodesic token regions.
  • Tokenisation: Increasing the token count g improves fine-detail coverage, but gains saturate at higher resolutions, exposing a granularity–redundancy trade-off.The evaluation considers g ∈ {32, 64, 128, 256}.
  • Pre-training: Moderate masking of 50% provides the best masked-autoencoder pre-training performance by balancing reconstruction difficulty and available geometric context.The pre-training strategy encourages geometry-aware features that tolerate noise, partiality, and varying sampling densities.
  • Qualitative Results: Qualitative evaluations show consistent correspondences across noisy, partial, full, and partial-to-full settings, while reconstructions recover occluded structures and preserve global shape consistency.The results span CP2P24, PSMAL, and BeCoS scenarios.

B.6 Robustness to Noise

TokenMatch is evaluated with synthetic Gaussian vertex noise added only at test time, and its performance degrades only slightly as perturbation increases. Qualitative results likewise show robustness under mild and moderate noise, with expected degradation under severe perturbations.

  • B.6 Robustness to Noise: Performance decreases by only 0.52 points under the evaluated Gaussian-noise perturbations.Noise is scale-normalised using each mesh’s bounding-box diagonal and is introduced only during evaluation.

C Further Qualitative Results

Further qualitative evaluations indicate that TokenMatch maintains accurate, geometrically consistent, and semantically aligned correspondences across incomplete shapes, strong deformations, and realistic BeCoS scenarios. These results support transfer beyond the training distribution.

  • C Further Qualitative Results: TokenMatch produces accurate and geometrically consistent correspondences on CP2P24 and PSMAL despite limited overlap, partiality, and non-isometric deformation.The visualisations cover challenging partial-to-partial settings with incomplete local structure.
  • C Further Qualitative Results: On BeCoS, predicted correspondences remain stable across large geometric variations, missing regions, noise, and varying degrees of partiality while preserving semantic alignment.The evaluation includes full shapes and settings with substantial deformation.
  • C Further Qualitative Results: Together, the qualitative results indicate that TokenMatch learns transferable geometric representations that generalise beyond its training distribution.This conclusion follows from the cross-dataset and cross-setting evaluations.
Loading 2609.04202v1…