PaperlayerView exact revision on arXiv

Source-linked AI summary

SemGeo-Gen: Unsupervised Generation of Approximate Cross-Instance Semantic-Geometric Correspondences

Roy Amoyal, Shira Ifergane, Oren Freifeld

arXiv:2610.32663v1cs.CVcs.AI

TL;DR

Exact cross-instance point correspondences can be ambiguous, and manual annotations are costly to scale. SemGeo-Gen generates approximate semantic-geometric supervision from weakly aligned 3D objects; its use for synthetic pretraining improves SPair-71k results after real-data fine-tuning.

  • Problem

    Exact point-level correspondences across instances are often ambiguous, while manual annotation is costly and difficult to scale.

    Source: §1 · p. 1

  • Method

    SemGeo-Gen lifts multi-view DINOv2 features into 3D, aligns weakly aligned object instances with continuous piecewise-affine registration, and prunes matches for semantic consistency.

    Source: §1 · p. 2

  • Results

    Up to 5 PCK points of gains on SPair-71k after real-data fine-tuning accompany 93% PCK@0.10 and 97% PCK@0.15 agreement with sparse KeypointNet annotations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Approximate correspondences generated from 3D assets can complement manual annotations and improve cross-instance correspondence learning.

    Source: §5 · p. 10

    The method assumes rigid, approximately aligned 3D objects, and its CPA deformation may not fully capture large cross-instance structural differences without supervision.

    Source: §4.1 · p. 7, §4.6 · p. 10

Abstract

from arXiv · show

We introduce SemGeo-Gen, an unsupervised method for generating approximate cross-instance semantic-geometric supervision from weakly aligned 3D object collections. Our goal is not to recover exact point-level correspondences, which are often ambiguous across different instances and would require expensive, impractical manual supervision at scale. Instead, we automatically produce approximate correspondences that are semantically meaningful, geometrically consistent, and diverse enough to train modern correspondence models. Given only coarse category-level rotation alignment, SemGeo-Gen lifts multi-view DINOv2 features to 3D, aligns object instances using continuous piecewise-affine registration, and prunes candidate matches using semantic consistency. The resulting 3D correspondences can be projected into rendered views, yielding scalable 3D-3D, 3D-2D, and 2D-2D supervision without manual annotation. We validate the generated correspondences against sparse human annotations from KeypointNet, obtaining 93% PCK@0.10 and 97% PCK@0.15. More importantly, we show that using the generated correspondences for synthetic pretraining improves recent state-of-the-art semantic correspondence models on SPair-71k after fine-tuning on real data, with gains of up to 5 PCK@0.10 points. Together, these results indicate that approximate semantic-geometric supervision generated from 3D assets can improve real-image correspondence learning and serve as a scalable alternative to costly manual annotation. Project page: https://bgu-cs-vil.github.io/SemGeo-Gen/.

1 Introduction

Exact point matches across object instances can be ambiguous, while manual correspondence annotations are costly and hard to scale. SemGeo-Gen generates approximate semantic-geometric matches from weakly aligned 3D collections and uses them to improve correspondence learning.

1 Introduction

  • Cross-instance correspondence is harder than same-instance matching because objects vary in geometry, proportions, structure, texture, and appearance while sharing only some semantic parts.The target matches should preserve semantic identity and geometric role when those distinctions are meaningful.Source: §1 · p. 1
  • Existing supervision relies heavily on manual annotation, which is costly, sparse, and difficult to scale, while exact point-level matches are often ambiguous across instances.The paper asks whether approximate matches that are semantically meaningful, geometrically consistent, and sufficiently diverse can supervise learning at scale.Source: §1 · pp. 1–2
  • SemGeo-Gen combines lifted multi-view semantic features, continuous piecewise-affine registration, and semantic pruning to produce matches constrained by geometry and semantics.It generates correspondences from weakly aligned 3D object collections without point, part, or correspondence supervision.Source: §1 · p. 2
  • Generated 3D correspondences can be projected into rendered views to provide scalable 3D–3D, 3D–2D, and 2D–2D supervision for image-based models.Each accepted object pair can yield dense 3D matches and many rendered 2D–2D training pairs.Source: §1 · p. 2
  • Up to 5 PCK points of gains on SPair-71k after real-data fine-tuning accompany 93% PCK@0.10, 97% PCK@0.15, and 99% PCK@0.20 agreement with sparse KeypointNet annotations.The results evaluate both synthetic pretraining for real-image models and the quality of generated 3D correspondences.Source: §1 · p. 2

2 Related Work

Prior correspondence resources provide either sparse image annotations or 3D datasets with limited category coverage, appearance, or fine-grained point supervision. SemGeo-Gen contrasts with these resources by generating approximate cross-instance correspondences directly between 3D instances and projecting them across modalities.

2D and 3D Cross-Instance Correspondence Datasets

  • Image-based benchmarks such as PF-PASCAL, SPair-71k, and CUB provide sparse annotations, while other datasets derive supervision from pose data; these resources remain image-based and human-annotated.The passages identify bibliography markers [11], [23],, [37], [19], [5], [34], and [22] for the cited datasets and pose resources.Source: §2 · p. 2
  • Existing 3D resources often focus on geometry, a single category, or articulated shapes, and commonly lack appearance information or scalable fine-grained point-to-point supervision.KeypointNet covers 16 rigid-object categories with sparse annotations averaging roughly 12 keypoints per object.Source: §2 · p. 2
  • Existing datasets rely on expensive, sparse human annotations, whereas the proposed approach automatically generates approximate correspondences across diverse rigid-object categories and supports 3D–3D, 3D–2D, and 2D–2D tasks.The accompanying figure illustrates these three correspondence modalities between airplane instances.Source: §2 · p. 3, Figure 1 · p. 3

2D Semantic-Geometric Correspondences

  • Semantic-only matching may treat geometrically symmetric points as equivalent, whereas standard semantic correspondence without annotation flips requires resolving orientation when distinctions are well defined.For symmetric categories, geometric distinctions can remain inherently ambiguous; GeoAware-SC [39] highlights cases where appearance alone is insufficient and existing methods often underperform.Source: §2 · p. 3

3D Semantic Keypoints and Correspondence Methods

  • DenseMatcher uses projected foundation-model features and functional maps but relies on costly part annotations; UKPGAN and Skeleton Merger [28] focus on sparse keypoints rather than dense cross-instance correspondences.The related-work discussion says none provides scalable, annotation-free dense supervision across diverse rigid-object categories.Source: §2 · pp. 3–4

3 Method

SemGeo-Gen generates approximate cross-instance correspondences from weakly aligned rigid-object pairs without point-, part-, or correspondence-level supervision. It combines lifted multi-view semantics, continuous piecewise-affine registration, and semantic filtering, then projects the resulting matches into rendered views.

3 Method

  • SemGeo-Gen takes approximately rotation-aligned rigid-object pairs and produces approximate 3D–3D correspondences that can be projected into 3D–2D and 2D–2D supervision.The weak alignment provides a category-level geometric prior without point-, part-, or correspondence-level supervision; semantic-coverage filtering is optional.Source: §3 · p. 4

3.1 Object Representation and Feature Lifting

  • Multi-view DINOv2 features are extracted from rendered views, lifted to visible 3D points, and averaged into a semantic descriptor for each point.The objects are represented as point clouds with associated semantic feature maps.Source: §3.1 · p. 4

3.2 Semantic Similarity

  • Semantic compatibility is measured by cosine similarity between DINOv2 descriptors jointly reduced to 64 dimensions with PCA.The implementation uses a semantic threshold of 0.575, corresponding to a squared Euclidean distance threshold of 0.85 between L2-normalized descriptors.Source: §3.2 · p. 5

3.3 Optional Semantic Coverage Pair Filtering

  • Optional bidirectional semantic-coverage filtering retains object pairs only when each direction meets the coverage condition.Coverage measures the fraction of source points with at least one semantically compatible target point; filtering reduces retained pairs while improving their quality and the resulting correspondences.Source: §3.3 · p. 5

3.4 Continuous Piecewise-Affine Registration

  • Continuous piecewise-affine registration deforms the source toward the target using a tetrahedral control grid and barycentric interpolation.Smoothness regularization encourages locally coherent deformations while allowing local shape variation; optimization uses Chamfer distance with smoothness regularization.Source: §3.4 · pp. 5–6

3.5 Approximate Correspondence Extraction

  • After registration, each transformed source point is assigned its nearest target point to form candidate correspondences.The stored match links the original source point to the target point, so it is an approximate correspondence induced by deformation-based alignment rather than exact point-level annotation.Source: §3.5 · p. 6

3.6 Semantic Consistency Pruning

  • Semantic pruning removes geometrically nearby candidate matches that are semantically inconsistent according to lifted DINOv2 features.The remaining matches form the final approximate semantic-geometric correspondence set.Source: §3.6 · pp. 6–7

3.7 Projection to 2D Correspondences

  • Calibrated camera projections convert 3D correspondences into 3D–2D or 2D–2D supervision, retaining only endpoints visible in their rendered views.Rendering multiple views of each object pair yields automatically generated training data for image-based semantic correspondence models.Source: §3.7 · p. 7

4 Experiments

The experiments test whether synthetic correspondences from 3D assets improve real-image matching and agree with sparse human annotations. They find improvements after synthetic pretraining and real-data fine-tuning, while also identifying alignment, deformation, feature, and domain-gap constraints.

4.1 Experimental Data

  • The experiments use KeypointNet for sparse human-annotation evaluation and Objaverse-OA for broader shape diversity; SPair-71k synthetic pretraining uses 12K pairs per category from nine rigid categories.KeypointNet annotations are used only for evaluation, while Objaverse-OA has over 100 curated rigid categories without correspondence annotations.Source: §4.1 · p. 7, §4.2 · p. 7

4.2 Results on SPair-71k

  • GeoAware-SC mean PCK rises from 79.36 to 83.73 (+4.37), and MARCO rises from 86.34 to 89.30 (+2.96), with synthetic pretraining followed by SPair-71k fine-tuning.Naive joint training gives mixed results; the gains require no architectural modifications.Source: §4.2 · p. 8
  • On all 18 SPair-71k categories, Rigid-9 synthetic pretraining improves GeoAware-SC PCK@0.10 by 3.72 on rigid categories, 1.29 on non-rigid categories, and 2.39 overall versus real-only training.The gains extend to categories excluded from synthetic pretraining.Source: §4.2 · p. 8, Table 2 · p. 8

4.3 Evaluation of Annotation Quality

  • KeypointNet evaluation yields 93% PCK@0.10, 97% PCK@0.15, and 99% PCK@0.20, with a median GT-used rate of 57.6%.Evaluation retains pairs passing match-coverage and feature-similarity thresholds; semantic pruning reduces coverage by excluding uncertain matches between structurally different instances.Source: §4.3 · p. 9, Table 3 · p. 9
  • Projected correspondences achieve 94.2% PCK@0.10 in 2D across 16 categories, with an average GT-used rate of 45.3%; 3D PCK@0.10 on the same visible correspondences is 95.1%.The authors note possible projection errors or visibility inconsistencies from approximate 3D correspondences.Source: §4.3 · p. 9

4.4 Ablations and Hyperparameter Sensitivity

  • Semantic filtering improves PCK@0.10 from over 84% with geometric registration alone to about 91% with correspondence-level pruning and about 93% with pair-level filtering; CPA is about 3× faster than TPS.Performance is stable across reasonable smoothness and semantic-threshold ranges; τ_sem = 0.575 balances accuracy and coverage, while λ_smooth = 0.3 better preserves structure while allowing local deformation.Source: §4.4 · pp. 9–10

4.5 Discussion / 4.6 Limitations

  • The approach scales across aligned rigid 3D collections and rendered view pairs, but assumes approximate alignment and has limitations from CPA deformation, DINOv2 features, non-rigid pose variation, and synthetic-to-real domain gap.The authors suggest pose clustering for non-rigid variation and note that real, canonically aligned 3D assets could reduce the domain gap.Source: §4.5 · p. 10, §4.6 · p. 10

5 Conclusion

The paper concludes that its unsupervised pipeline generates scalable cross-instance correspondences from 3D assets without manual annotation. The correspondences align well with human annotations and provide effective pretraining for real-image semantic correspondence models.

5 Conclusion

  • The pipeline generates cross-instance semantic-geometric correspondences from 3D assets without manual annotation.The method is presented as scalable and unsupervised.Source: §5 · p. 10
  • The generated correspondences are approximate but align well with human annotations.Source: §5 · p. 10
  • The correspondences provide an effective pretraining signal for real-image semantic correspondence models.The paper concludes that scalable semantic-geometric data can complement manual annotations and improve cross-instance correspondence learning.Source: §5 · p. 10

Correspondences

The supplementary material adds qualitative transfer comparisons for MARCO and GeoAware-SC, alongside examples of correspondences on rigid categories beyond existing benchmarks. These examples are drawn from Objaverse-OA.

Correspondences

  • The supplement provides additional qualitative results, extended ablations, and detailed per-label analysis.Source: Correspondences · p. 14

A.1 SPair-71k Transfer Results

A.2 Correspondence Examples on New Categories

  • Examples show approximate semantic-geometric correspondences on diverse Objaverse-OA rigid categories beyond SPair-71k, KeypointNet, and AP-10K coverage.Source: §A.2 · p. 17, Figure 6 · p. 17

A.3 Qualitative Results on KeypointNet

The KeypointNet examples compare predicted target points with sparse human-annotated semantic keypoints, including cases of close agreement. The supplement also describes additional quantitative comparisons on SPair-71k and KeypointNet.

A.3 Qualitative Results on KeypointNet

  • Predictions generally align well with human annotations across categories, and selected examples show close overlap with sparse semantic keypoints.Source keypoints serve as queries for predictions on target objects; red lines connect predictions to their ground-truth annotations.Source: §A.3 · p. 18, Figure 7 · p. 18, Figure 8 · p. 19

B Additional Quantitative Results

  • Additional quantitative evaluations examine SPair-71k thresholds and training strategies, and compare KeypointNet correspondences with the Back to 3D few-shot detector.Source: B Additional Quantitative Results · p. 20

B.1 Additional Results on SPair-71k

Across evaluated models, synthetic pretraining followed by real-data fine-tuning consistently outperforms real-only training at practical thresholds, with gains also extending to categories excluded from synthetic pretraining. The appendix further compares SemGeo-Gen with few-shot keypoint detection on KeypointNet and summarizes additional ablations.

B.1 Additional Results on SPair-71k

  • Synth→Real improves GeoAware-SC across all evaluated thresholds, while MARCO’s gain reverses at PCK@0.01 (−2.17) but holds at more relaxed thresholds.For GeoAware-SC, joint training slightly exceeds Real-only but remains below Synth→Real; MARCO joint training underperforms Real-only.Source: §B.1 · p. 20
  • +4.37 PCK for GeoAware-SC (79.36 to 83.73) and +2.96 for MARCO (86.34 to 89.30) at PCK@0.10 with Synth→Real versus Real-only.Synth→Real is the strongest strategy overall; synthetic-only is insufficient for high performance, and naive joint training is suboptimal.Source: §B.1 · p. 20
  • The pretraining gains span GeoAware-SC, which uses DINOv2 and Stable Diffusion features, and MARCO, which uses DINOv2 without Stable Diffusion.The authors describe the benefit as architecture-agnostic and not tied to a specific feature family.Source: §B.1 · p. 20

B.2 Comparison with Back to 3D on KeypointNet

  • SemGeo-Gen achieves 86.5% PCK@0.10 on KeypointNet, exceeding Back to 3D’s 1-shot (63.2%) and 3-shot (64.1%) results without keypoint supervision.Back to 3D is a few-shot keypoint detection method, not a cross-instance correspondence method.Source: §B.2 · p. 21, Table 4 · p. 21

C Extended Ablation Study

  • The appendix adds ablations of registration, semantic filtering, and sensitivity to imperfect canonical alignment to characterize accuracy, efficiency, and robustness.The supplied passage introduces these studies but does not report their findings.Source: C Extended Ablation Study · p. 22

C.1 Registration Method Ablation

The ablations show that non-rigid registration and semantic pruning both matter for accurate correspondences, while the semantic threshold trades accuracy against coverage. Accuracy also depends on approximate canonical alignment.

C.1 Registration Method Ablation

  • CPA and TPS achieve similar correspondence accuracy, but CPA is about 3.3× faster per pair (1.13s vs. 3.71s); affine registration performs worse, especially at tighter PCK thresholds.CPA achieves the best average performance at four of five PCK thresholds, while TPS is slightly better at PCK@0.05.Source: §C.1 · pp. 22–23, Table 5 · p. 23

C.2.1 No Semantic Filtering

  • With no semantic filtering, geometric alignment alone yields lower accuracy, especially at PCK@0.05 and PCK@0.10, because it can match semantically inconsistent regions.The study compares removal of filtering, correspondence-level filtering alone, and threshold choices with pair-level filtering.Source: §C.2.1 · p. 23, §C.2 · p. 23, Table 6 · p. 24

C.2.2 Correspondence-Level Semantic Filtering Only

  • Mean PCK@0.10 rises from 0.84 without filtering to 0.91 with correspondence-level semantic filtering alone.The gain supports the role of pruning geometrically plausible but semantically inconsistent matches.Source: §C.2.2 · p. 23, Table 7 · p. 24

C.2.3 Correspondence-Level Semantic Threshold

  • The GT-used coverage measure is conservative because sparse KeypointNet keypoints can miss valid dense surface correspondences outside the 0.05 evaluation radius.The method generates dense correspondences across the object surface, whereas annotations typically mark salient corners or extreme parts.Source: §C.2.3 · p. 23
  • At τsem = 0.75, accuracy reaches 75.8% PCK@0.05 but GT-used coverage is 26.0%; at τsem = 0.25, coverage is 91.5% with 69.9% PCK@0.05 and 89.6% PCK@0.10.The default τsem = 0.575 retains 57.6% of keypoints with almost no accuracy loss versus the strict setting, more than doubling the 26.0% rate.Source: §C.2.3 · pp. 23–25, Table 8 · p. 25

C.3 Sensitivity to Rotation Misalignment

  • Mean PCK@0.10 falls from 0.93 for aligned objects to 0.89 at 5° and 0.85 at 10°; performance drops substantially at 30° and beyond.Approximate canonical alignment remains an important prerequisite for accurate correspondence generation.Source: §C.3 · p. 25, Table 9 · p. 25

D Qualitative Analysis of DINOv2 Based Semantic Filtering

The qualitative analysis illustrates how lifted DINOv2 features support pair-level semantic coverage checks and correspondence-level pruning. It shows that semantic filtering removes unmatched or semantically inconsistent regions, including regions that geometric deformation might otherwise align.

D Qualitative Analysis of DINOv2 Based Semantic Filtering

  • After registration, red points are removed, and correspondence-level pruning further rejects geometrically matched points whose assigned matches are semantically inconsistent.Pair-level coverage filtering and correspondence-level pruning serve distinct roles in the pipeline.Source: D Qualitative Analysis of DINOv2 Based… · p. 26
  • Figure 9 colors points green when the paired object contains a semantically similar point and red when it does not, alongside the rendering and joint-PCA feature visualization.The semantic similarity threshold is 0.575; coverage is used to filter pairs lacking enough shared semantic structure.Source: Figure 9 · p. 26, D Qualitative Analysis of DINOv2 Based… · p. 26
  • Figure 10 shows sampled post-pruning matches across views; exposed car seats, a steering wheel, and a roof region receive no correspondences when semantically unmatched.The roof is removed even though CPA deforms it geometrically toward other object parts.Source: D Qualitative Analysis of DINOv2 Based… · p. 26, Figure 10 · p. 27

E 3D–2D Correspondence Evaluation on KeypointNet

The KeypointNet evaluation tests whether approximate 3D–3D matches remain accurate when projected to visible target views. On found visible keypoints, 3D–2D accuracy is close to 3D accuracy, while coverage reflects both correspondence availability and visibility.

E 3D–2D Correspondence Evaluation on KeypointNet

  • The evaluation retains only visible ground-truth keypoints and correspondences with visible target points, then selects the nearest source-point match within 0.05 times the target 3D bounding-box diagonal.Keypoints without such a match are counted as not found; PCK is reported on found keypoints.Source: E 3D–2D Correspondence Evaluation on… · p. 27
  • 94.2% PCK@0.10 is achieved for 3D–2D correspondences across 16 categories, versus 92.7% in the original 3D–3D evaluation over all found keypoints.On the same retained keypoints, 3D PCK is 95.1% versus 94.2% in 2D; the authors note projection inaccuracies may contribute to the difference.Source: E 3D–2D Correspondence Evaluation on… · p. 28, Table 10 · p. 28
  • Average coverage is 45.3% of visible ground-truth keypoints, combining correspondence availability with visibility filtering.The study renders each target from four random viewpoints and reports stable results across sampled viewpoint sets, varying by at most ±0.1 percentage points.Source: E 3D–2D Correspondence Evaluation on… · pp. 27–28

F Detailed PCK Results by Semantic Label and KeypointNet Category

The per-keypoint evaluation reports results across 15 categories using the full pipeline and filtered KeypointNet annotations. Its sparse annotations make the results a conservative estimate of quality near annotated locations.

F Detailed PCK Results by Semantic Label and KeypointNet Category

  • 69,058 source-target pairs were evaluated across 15 categories, and 46,849 passed the pair-level filter.The analysis uses a subset of up to 100 objects per category, including 38 for Cap and 90 for Helmet.Source: F Detailed PCK Results by Semantic… · p. 28
  • A keypoint is evaluable when a predicted correspondence remains within 0.05 of the object bounding-box diagonal from its ground-truth location; results are reported as raw correct/total counts.The evaluation uses correspondence-level semantic filtering at τsem = 0.575 and pair-level filtering.Source: F Detailed PCK Results by Semantic… · p. 28
  • Tables 11–25 report per-keypoint PCK results as correct/total for 15 KeypointNet categories.The annotations are sparse and usually mark salient corners or extreme parts, while the method produces dense surface correspondences; the reported quality is therefore conservative around annotated locations.Source: F Detailed PCK Results by Semantic… · p. 28
  • Tables 26–29 report PCK averaged over seeds at semantic thresholds 0.625, 0.575, 0.525, and 0.25, respectively.The supplied captions identify the thresholds but do not provide the table values.Source: Table 26 · p. 35, Table 27 · p. 35, Table 28 · p. 36
Loading 2610.32663v1…