PaperlayerView exact revision on arXiv

Source-linked AI summary

SemAnCorr: Semantic Anchored Correspondence for Zero-Shot Manipulation Skill Transfer

Xiaoxiang Dong, William Baron, Hongyi Chen, Uksang Yoo, Jeffrey Ichnowski, Weiming Zhi

arXiv:2607.28382v1cs.RO

TL;DR

Manipulation skills must transfer between functionally similar objects despite geometric differences, but existing matching can miss the geometric frames needed for execution. SemAnCorr anchors semantically consistent regions and propagates them with functional maps; it reports stronger benchmark correspondence and more reliable zero-shot transfer on unseen objects.

  • Problem

    Existing correspondence methods can identify interaction regions yet miss skill sequence and articulated structure, while effective transfer requires both semantic interaction locations and geometrically coherent execution frames.

    Source: §I · p. 1

  • Method

    SemAnCorr is a training-free framework that selects semantically meaningful anchors and propagates their constraints across object surfaces using functional maps.

    Source: §I · pp. 1–2

  • Results

    SemAnCorr achieved 90.8% average semantic accuracy across seven categories, 6.2% above D3Fields, and improved zero-shot transfer reliability on previously unseen objects.

    Source: §I · p. 2, D. Inner-Category Generalization · p. 5

  • Takeaways & Limitations

    Semantic consistency determines where to interact, while geometric coherence determines how, and both are necessary for reliable skill transfer across geometrically diverse instances.

    Source: §VI · p. 8

Abstract

from arXiv · show

Transferring manipulation skills across object instances that share functionality but differ in geometry remains a fundamental challenge in robot learning. While recent correspondence methods leverage dense visual descriptors and 3D feature fields, nearest-neighbor feature matching often produces spatially incoherent correspondences that fail to recover the local geometric frames required for reliable skill transfer. We introduce SemAnCorr, a training-free framework that establishes dense correspondence by selecting semantically consistent anchor regions through joint pose-correspondence optimization and propagating these constraints over the object surface using functional maps. The resulting correspondences preserve both semantic consistency and geometric coherence, enabling object-centric manipulation skills to transfer across geometrically diverse instances. We evaluate SemAnCorr on a dense correspondence benchmark built on PartNet-Mobility, achieving 90.8% semantic accuracy in our benchmark evaluation while improving geometric coherence over recent state-of-the-art baselines. Finally, we show that these improvements translate directly into real-world manipulation performance: using a single demonstration, SemAnCorr enables substantially more reliable zero-shot manipulation skill transfer to previously unseen objects than existing correspondence methods. Videos and additional visualizations are available at [https://semancorr.github.io](https://semancorr.github.io) .

I. INTRODUCTION

The introduction frames skill transfer as requiring correspondences that preserve interaction semantics and local geometric frames, then presents SemAnCorr and its benchmark and manipulation contributions. Related work is positioned against methods with limitations in geometric coherence, 3D orientation, or dense functional structure.

I. INTRODUCTION

  • Existing approaches can locate approximate interaction regions but often miss the skill’s sequence and relationship to articulated structure.The paper argues that effective transfer needs semantic correspondence for where to interact and geometrically coherent local frames for how to execute the interaction.Source: §I · p. 1
  • SemAnCorr’s improved geometric correspondence translates into more reliable zero-shot skill transfer across previously unseen object instances.The paper also presents an object-centric pipeline for transferring articulated manipulation skills from a single demonstration.Source: §I · p. 2
  • SemAnCorr is a training-free dense correspondence framework that selects semantic anchors and propagates them over object surfaces with functional maps.The framework aims to preserve semantic consistency and geometric coherence across geometrically diverse objects.Source: §I · pp. 1–2

II. RELATED WORK

  • Classical functional maps rely on intrinsic geometry, deep functional maps require category-specific training, and DenseMatcher uses learned feature refinement.SemAnCorr retains the functional-map backbone but uses pretrained-feature semantic anchors instead of learned refinement, at the cost of per-object-pair optimization.Source: §II · p. 2
  • Prior 2D correspondence approaches neglect 3D interaction orientation, while 3D descriptor fields can produce spatially incoherent matches through pointwise nearest-neighbor matching.SemAnCorr adds functional-map propagation to enforce global surface smoothness.Source: §II · p. 2
  • Affordance methods often require category-specific training, while keypoint and retrieval methods cannot recover local geometric frames from sparse or coarse representations.SemAnCorr instead establishes dense surface correspondence to transfer complete functional signatures without task-specific training.Source: §II · p. 2

III. PROBLEM STATEMENT

The problem is to compute a dense vertex-level map that matches semantically similar parts while remaining smooth over the object surface. The method overview describes identifying parts, selecting semantically and geometrically compatible pairs, and propagating constraints across the surface.

III. PROBLEM STATEMENT

  • The target is a dense vertex-level correspondence that matches semantically similar parts and remains smooth across the full object surface.The formulation assumes each mesh is a single connected component and that corresponding semantic region pairs exist to anchor the map.Source: §III · p. 2
  • Object-relative quantities defined on one mesh can be transferred to the other by applying the correspondence.The map is constrained to approximate the semantic anchor pairs while varying smoothly beyond those regions.Source: §III · p. 2

IV. METHOD: SEMANTIC ANCHORED CORRESPONDENCE

  • The method identifies meaningful surface parts, selects pairs consistent in meaning and geometry, then propagates a dense map using those pairs as constraints.These stages are presented as the high-level SemAnCorr pipeline.Source: §IV · p. 2

A. 3D Semantic Acquisition and Clustering

SemAnCorr builds a semantic point cloud from multiview rendered observations, then clusters the surface into semantically coherent parts. The clusters provide the units used by later anchor selection.

A. 3D Semantic Acquisition and Clustering

  • Pretrained SigLip2Vision embeddings are extracted from rendered views, lifted to 3D using camera and depth information, and aggregated at nearby mesh points.The patch embeddings average the model’s last four hidden states.Source: A. 3D Semantic Acquisition and… · p. 3
  • The framework clusters points using reduced semantic embeddings concatenated with normalized 3D positions.PCA reduces embedding dimensionality before k-means forms an initial partition.Source: A. 3D Semantic Acquisition and… · p. 3
  • Spatially disconnected clusters are split and small fragments are merged with neighboring clusters to produce contiguous semantic parts.These parts are the semantic clusters used in the framework overview before anchor-region selection.Source: A. 3D Semantic Acquisition and… · p. 3, Figure 2 · p. 3

B. Joint Alignment and Anchor Selection

SemAnCorr selects anchors by jointly considering semantic similarity and geometric compatibility under a rigid alignment. It refines initial semantic matches into mutually exclusive, high-confidence pairs for the downstream functional map.

B. Joint Alignment and Anchor Selection

  • Bilateral margin confidence measures whether each cluster prefers its proposed match over alternatives from both sides of the pair.The method greedily selects the highest-confidence pairs subject to a bijectivity constraint, so no cluster is reused.Source: B. Joint Alignment and Anchor Selection · p. 3
  • Joint pose-correspondence optimization addresses semantically similar clusters that may be geometrically incompatible across objects.The objects are first normalized to the unit sphere, and soft cluster assignments are co-updated with rigid pose alignment.Source: B. Joint Alignment and Anchor Selection · pp. 3–4
  • The joint score combines geometric compatibility under pose with semantic similarity to evaluate candidate cluster pairs.Geometric compatibility is based on chamfer similarity between cluster point clouds.Source: B. Joint Alignment and Anchor Selection · pp. 3–4
  • The semantic term initially guides alignment, while later optimization shifts weight toward geometric compatibility to refine the pose.The best initialization’s joint scores become confidence scores for final anchor selection.Source: B. Joint Alignment and Anchor Selection · p. 4

C. Semantic Anchored Functional Map

SemAnCorr uses semantic anchor regions to constrain a functional map, then refines the resulting dense correspondence to preserve semantic grounding and geometric smoothness.

C. Semantic Anchored Functional Map

  • A low-dimensional functional map encodes surface-to-surface function transforms in a spectral basis, with semantic anchors constraining the map.The first k Laplace-Beltrami eigenfunctions provide a smoothness prior; the dense pointwise map is recovered by nearest-neighbor search in the induced spectral embedding.Source: C. Semantic Anchored Functional Map · p. 4
  • Anchor-region correspondences are found after global mesh registration, using per-region centering and per-axis scaling before nearest-neighbor matching.The matching is performed within each semantically corresponding anchor pair.Source: C. Semantic Anchored Functional Map · p. 4
  • An initial functional map is fit from keypoints subsampled per anchor cluster to avoid overconstraining the spectral basis.The passage describes this as avoiding collapse of the pointwise map into a Voronoi partition.Source: C. Semantic Anchored Functional Map · p. 4
  • A geometry-constrained ZoomOut refinement increases basis dimension while alternating map and correspondence updates, limiting searches to spatial neighbors and re-pinning anchor interiors.This refinement is intended to prevent large jumps and anchor drift while keeping correspondences semantically grounded and geometrically smooth.Source: C. Semantic Anchored Functional Map · p. 4

V. EMPIRICAL EVALUATIONS

The evaluation asks whether SemAnCorr preserves semantic and geometric correspondence across geometrically diverse objects and whether correspondence quality relates to zero-shot skill-transfer reliability.

V. EMPIRICAL EVALUATIONS

  • The experiments assess whether improved correspondence quality translates into more reliable zero-shot manipulation skill transfer.The stated evaluation spans a manipulation dataset and real-world robotic settings.Source: §V · p. 4
  • The evaluation tests whether dense correspondence preserves semantic consistency and geometric coherence across geometrically diverse object instances.The paper also asks whether improved correspondence quality translates into more reliable zero-shot manipulation skill transfer.Source: §V · p. 4
  • Correspondence visualizations transfer a smooth color gradient from source to target, with smooth transitions indicating coherent mappings and fragmented patterns indicating spatial inconsistency.Source: §V · p. 4

A. Dataset and Implementation

The evaluation uses PartNet-Mobility object pairs and reports semantic accuracy alongside complementary measures of geometric correspondence quality.

A. Dataset and Implementation

  • Seven PartNet-Mobility categories comprising over 100 rigid and articulated objects are evaluated using all within-category object pairs.Each category reports the mean and standard deviation of each metric; the implementation fixes K = 6 clusters, α = 3 anchors, and k = 30 basis functions.Source: A. Dataset and Implementation · p. 4
  • On an RTX 4090 GPU, SemAnCorr computes dense correspondence for an object pair in approximately 6 s on average.The reported time includes semantic feature extraction, anchor optimization, and functional map estimation.Source: A. Dataset and Implementation · p. 4

1) Semantic Accuracy.

  • Semantic accuracy (sAcc) measures the fraction of target vertices with valid semantic labels whose transferred labels match ground truth.Each vertex receives the label of its corresponding source vertex before comparison.Source: 1) Semantic Accuracy · p. 4

2) Geometric Coherency.

  • Geometric coherency is assessed through continuity and coverage: continuity checks local-neighborhood preservation, while coverage measures the fraction of source vertices receiving a correspondence.Continuity uses a tolerance of τ = 2 source edge lengths; low coverage means parts of the demonstrated object may lack counterparts on the novel object.Source: 2) Geometric Coherency · pp. 4–5
  • The Geometric Coherence Score (GCS) combines continuity and coverage using their harmonic mean, emphasizing the weaker component.The two measures address local direction correctness and sufficient mapped spatial extent for recovering interaction frames.Source: 2) Geometric Coherency · p. 5

C. Baselines

The evaluation compares four representative geometry- and semantics-driven correspondence baselines.

C. Baselines

  • The comparison includes four baselines spanning geometry- and semantics-driven correspondence approaches.Source: C. Baselines · p. 5
  • FM-WKS uses Wave Kernel Signature descriptors and intrinsic geometry without semantic information.It serves as a lower bound on the contribution of semantic features.Source: C. Baselines · p. 5
  • Robo-ABC matches pretrained visual features in rendered 2D views and lifts the highest-confidence match to 3D.Eight viewpoints are rendered per mesh.Source: C. Baselines · p. 5
  • D3Fields matches per-vertex descriptors formed by lifting DINOv2 features from multi-view renders, isolating functional-map propagation against nearest-neighbor matching.Source: C. Baselines · p. 5
  • DenseMatcher combines SD-DINO features with a learned DiffusionNet refiner and functional maps.Its generalization is evaluated on the paper's PartNet-Mobility categories.Source: C. Baselines · p. 5

D. Inner-Category Generalization

SemAnCorr leads semantic accuracy across seven categories and also yields markedly stronger geometric coherence than competing methods.

D. Inner-Category Generalization

  • 90.8% average semantic accuracy, 6.2% above D3Fields, with SemAnCorr outperforming all baselines across seven categories.Semantic Accuracy is reported in Table I.Source: D. Inner-Category Generalization · p. 5
  • Within-category pairs score highest in anchor confidence, while functionally related categories score higher than less similar categories.Source: Figure 5 · p. 6
  • SemAnCorr achieves a GCS more than twice that of the next-best method while maintaining continuity and meaningful surface coverage.FM-WKS attains high continuity by collapsing correspondences onto few vertices, whereas D3Fields has fragmented local mappings.Source: D. Inner-Category Generalization · pp. 5–6

E. Cross-Category Generalization

SemAnCorr is evaluated on cross-category correspondence, component ablations, and real-world skill transfer, with results highlighting the need for both semantic and geometric correspondence.

E. Cross-Category Generalization

  • SemAnCorr achieves the highest semantic accuracy on Scissors→Pliers and Kettle→Bottle, with smooth correspondences despite noticeable geometric differences.Source: E. Cross-Category Generalization · p. 6
  • Anchor confidence peaks within categories and remains relatively high for functionally related groups, but is not a direct measure of manipulation success.The score is proposed as a lightweight compatibility prior.Source: E. Cross-Category Generalization · p. 6

F. Ablation Study

  • Ablations show complementary roles: removing semantic clustering degrades geometric coherence, removing anchors most harms semantic accuracy, and removing propagation reduces both.The ablation results are summarized in Table IV, with qualitative examples in Fig. 6.Source: F. Ablation Study · p. 6, Table IV · p. 6, Figure 6 · p. 6

G. Real-World Manipulation Skill Transfer

  • The real-world evaluation compares SemAnCorr chiefly with D3Fields across five skill-transfer tasks, running each task in 10 trials with varied object poses and viewpoints.Source: G. Real-World Manipulation Skill… · pp. 6–7, Table V · p. 7
  • Skills are encoded as contact keyframes and relative motion waypoints; transfer uses Procrustes alignment to guide GPG grasp selection, while motion segments transfer unchanged.GPG ranks candidate grasps by quality and proximity to the transferred prior pose.Source: G. Real-World Manipulation Skill… · p. 7
  • On complex Tasks 3–5, D3Fields often finds the correct semantic part but fails from incoherent local frames; SemAnCorr's remaining failures mainly reflect mesh-to-workspace alignment.The methods perform comparably on simpler Tasks 1 and 2; the real-world evaluation reports task success over 10 trials.Source: G. Real-World Manipulation Skill… · pp. 6–8, Table V · p. 7, Figure 7 · p. 7

VI. CONCLUSION AND FUTURE WORK

The paper concludes that reliable transfer across geometrically diverse objects requires both semantic consistency and local geometric coherence. It also identifies policy-data bootstrapping as a potential use and bimanual or dexterous manipulation as future scope.

VI. CONCLUSION AND FUTURE WORK

  • Reliable skill transfer across geometrically diverse objects requires semantic consistency for where to interact and geometric coherence for how.Source: §VI · p. 8
  • Future work includes extending the formulation to bimanual and dexterous manipulation, where multiple contact points must remain geometrically consistent.Source: §VI · p. 8
  • Successful executions could be transferred to unseen objects to generate demonstrations for policy fine-tuning, reducing repeated human data collection.Source: §VI · p. 8
Loading 2607.28382v1…