Source-linked AI summary
Geometry Matters: 3D Foundation Priors for Learning Semantic Correspondence
Artur Jesslen, Olaf Dünkel, Adam Kortylewski
TL;DR
Semantic correspondence is hindered by 2D foundation features that confuse symmetric, repeated, and visually similar parts with different 3D structure. The paper adds reconstructed instance-specific geometry, rendered PartField descriptors, and geodesic filtering to train an adapter without manual pose annotations. The framework yields consistent benchmark improvements over prior methods, while its performance remains sensitive to pose and shape-estimation errors and PartField’s coarse localization.
Problem
2D foundation features lack explicit 3D awareness and cannot reliably distinguish visually similar structures that are geometrically distinct.
Method
SAM3D geometry and pose estimates are refined by render-and-compare optimization; rendered PartField descriptors and geodesic filtering guide adapter training on DINOv2 and Stable Diffusion features.
Results
The framework achieves consistent improvements over prior methods on standard benchmarks while using less manual supervision.
Takeaways & Limitations
Instance-specific 3D structure can guide both geometry-aware feature construction and pseudo-label filtering for semantic correspondence.
Takeaways & Limitations
Performance depends on SAM3D pose and shape estimates, while PartField supplies coarse regional rather than precise within-part localization.
Abstract
from arXiv · showhide
Foundation features from self-supervised vision models and text-to-image diffusion models have proven effective for semantic correspondence estimation. However, because these features are learned primarily from 2D image objectives, they lack explicit 3D awareness and often confuse symmetric object sides, repeated parts, and visually similar structures that are distinct in 3D. We introduce a 3D-aware post-training framework that goes beyond available 2D foundation features by incorporating priors from 3D foundation models. Given an image, our method uses SAM3D to estimate object geometry and pose, and refines the pose through render-and-compare optimization. Subsequently, we render PartField descriptors from the reconstructed geometry into the image plane based on the estimated object pose. The resulting geometry-aware feature maps complement DINO and Stable Diffusion features, while geodesic distances on the reconstructed shapes enable reliable filtering of candidate correspondences. We use the filtered matches as supervision to train a lightweight adapter on top of DINO and Stable Diffusion for semantic correspondence. In contrast to prior post-training approaches that require pose annotations and rely on coarse spherical geometry, our method automatically obtains instance-specific 3D structure and uses it to guide correspondence learning. Experiments show that our approach improves semantic correspondence over the prior methods while reducing manual geometric supervision. Code and model can be found at https:/github.com/GenIntel/3D-SC.
1 Introduction
Semantic correspondence remains difficult because 2D foundation features confuse visually similar but geometrically distinct parts. The paper introduces an annotation-free 3D-aware framework that combines reconstructed geometry, PartField descriptors, and geodesic filtering to improve correspondence quality.
- Semantic correspondence must handle changes in appearance, viewpoint, articulation, shape, and background while matching semantically equivalent parts.
- DINOv2 and Stable Diffusion provide strong transferable features, but their 2D training objectives lack explicit 3D awareness.
- Prior weak-3D methods require human pose annotations and coarse spherical geometry, limiting scalability and fine distinctions between symmetric or articulated parts.
- The proposed framework estimates geometry and pose with SAM3D, refines pose by render-and-compare optimization, and renders PartField descriptors into the image plane.
- Geodesic distances on reconstructed shapes filter inconsistent candidate matches and provide higher-quality pseudo-labels for adapter training.
- The resulting geometry-aware features achieve state-of-the-art semantic correspondence with less manual supervision than prior methods.
2 Related Work
Related work progresses from image-based semantic matching toward geometric priors and 3D-aware representations. These approaches improve structural reasoning but often depend on templates, accurate pose, or category-level reconstruction.
- Semantic correspondence methods evolved from hand-crafted descriptors and learned matchers toward weak supervision, cycle consistency, and pseudo-label expansion.
- DINOv2 and diffusion features provide complementary semantic and spatial cues for dense matching.
- Geometric approaches use CAD surfaces, category templates, atlases, and learned 3D representations to establish correspondences in shared geometric frames.
- Existing 3D-aware methods typically require mesh templates, precise pose, or category-level reconstruction pipelines.
3 Method
3D-SC reconstructs and canonicalizes an object-centric mesh, combines rendered PartField descriptors with DINOv2 and Stable Diffusion features, and filters candidate matches using geometry.
- 3D reconstruction and canonicalization: The pipeline reconstructs and canonicalizes an object-centric mesh for each instance before correspondence estimation.It refines scale, translation, and orientation using foundation-model predictions and render-and-compare optimization.
- Feature fusion and candidate generation: Rendered PartField descriptors encode geometric and part-level structure in image space and complement DINOv2 and Stable Diffusion features.PartField can distinguish visually similar but geometrically distinct parts, such as front versus rear wheels or left versus right legs.
- 3D reconstruction and canonicalization: Residual four-fold yaw ambiguity is resolved by rendering eight known orientations and selecting a correction with OrientAnything V2 and majority voting.The selected correction rotates each mesh into a consistent canonical orientation.
- 3D reconstruction and canonicalization: Pose refinement first uses distance-transform attraction and then differentiable soft-IoU optimization after the rendered and observed masks overlap.The distance-transform stage tolerates coarse boundaries, while soft-IoU tightens the resulting alignment.
- Feature fusion and candidate generation: Candidate matches are proposed by nearest-neighbor search over fused DINO, Stable Diffusion, and PartField features with relaxed cyclic consistency.The three feature sources are independently L2-normalized, concatenated with category-agnostic weights, and searched in the fused space.
- Geometric verification and training: Surviving candidates are lifted onto reconstructed meshes and rejected when their geodesic error exceeds a threshold, producing pseudo-labels for a lightweight adapter.The adapter is trained on top of frozen DINOv2 and Stable Diffusion features using the retained pseudo-labels.
4 Experiments
The experiments evaluate 3D-SC across four semantic-correspondence benchmarks and test its components through qualitative comparisons and ablations. Results show strong gains, especially for geometry-sensitive correspondences, while geodesic filtering and feature choices improve supervision and performance.
- Qualitative results: 3D-SC produces denser, well-distributed, and geometrically consistent pseudo-annotations that avoid left-right ambiguities.The qualitative comparison is against DIY-SC and attributes consistency to instance-specific 3D geometry.
- Evaluation on SPair-71k: 3D-SC reaches 73.0 PCK@0.1 on SPair-71k, improving 3.4 points over DIY-SC+OriAny among methods with the same supervision regime.Gains concentrate in symmetric rigid categories, including bus (+10.8), tv monitor (+9.8), car (+6.9), and motorcycle (+5.1).
- Evaluation on SPair-71k: 3D-SC achieves 70.8 PCK@0.1 on SPair-Geo-Aware, surpassing existing weakly supervised approaches on geometry-sensitive correspondences.The subset emphasizes symmetric or repeated parts and tests object orientation and geometry.
- Evaluation on SPairU: 3D-SC obtains 67.3 PCK@0.1 on SPairU, the best result without human annotations and only 0.6 points below DIY-SC.The method improves over DIY-SC+OriAny by 1 point, despite PartField being less suited to keypoints within the same part.
- Evaluation on AP-10K: 3D-SC achieves 69.6/68.5/56.9 PCK@0.1 on AP-10K intra-species, cross-species, and cross-family splits, outperforming the strongest no-human-annotation baseline on all three.The results extend beyond rigid objects to articulated and shape-diverse animal poses.
- Ablations: Adding PartField to SD+DINO lowers unfiltered-candidate FPR while increasing the average number of retained candidates per pair.This indicates suppression of incorrect matches alongside recovery of additional correct matches involving geometrically distinct regions.
- Ablations: Replacing DINOv2 with DINOv3 improves performance by 0.9 PCK@0.1, while 0.5-0.9 point gains also appear for DIY-SC variants.3D-SC remains stronger than both DIY-SC variants under either backbone setting.
5 Limitations and Future Work
The pipeline remains limited by geometry-estimation and feature-resolution issues, while cross-mesh matching currently uses a relatively simple alignment strategy. Future improvements target stronger deformable-category features and denser registration.
- Estimation and filtering: Errors in SAM3D pose and shape estimates propagate through reprojection and can degrade geodesic consistency, although filtering removes most false positives.This constrains reliability when the reconstructed geometry or pose is inaccurate.
- Feature resolution: PartField provides coarse regional cues rather than precise within-part localization, motivating its relatively low fusion weight and limiting gains on SPairU.The paper suggests stronger 3D features tailored to deformable categories such as animals.
- Future work: Cross-mesh correspondence uses nearest-neighbor matching in PartField space, leaving denser registration with optimal transport or functional maps as future work.Such methods would trade additional computation for finer alignment.
6 Conclusion
The framework uses automatically obtained, instance-specific 3D structure to guide semantic correspondence through geometry-aware features and geodesic filtering. It improves over prior methods on standard benchmarks without human pose annotations.
- The method combines SAM3D geometry and pose estimation with render-and-compare refinement to obtain instance-specific 3D structure.
- Rendered PartField descriptors complement DINO and Stable Diffusion features with geometry-aware cues in the image plane.
- Geodesic distances on reconstructed shapes filter inconsistent candidate correspondences before adapter training.
- The filtered correspondences supervise a lightweight adapter that yields consistent improvements over prior methods on standard benchmarks.
- The results suggest that automatically obtained instance-specific structure can provide a more reliable geometric prior than coarse spherical proxies.
Correspondence
The supplement organizes additional material on pseudo-groundtruth, correspondence pseudo-annotations, results, visualizations, reproducibility, and LLM assistance. These topics are divided across Sections A–D.
- Section A covers implementation details for 3D reconstruction and pose canonicalization.
- Section B provides details on feature fusion, pseudo-label generation, and geodesic filtering.
- Section C reports per-category results and additional qualitative visualizations.
- Section D discusses reproducibility and the use of LLM assistance in writing the paper.
A Pseudo-groundtruth via foundation models
The pseudo-groundtruth pipeline reconstructs and canonicalizes object geometry, then projects PartField features into the image plane. These features are spatially coherent within parts and discriminative across repeated or symmetric structures.
- 3D reconstruction and canonicalization: Starting from one image, the pipeline extracts a 2D mask, reconstructs an object-centric mesh, refines its pose, and resolves residual yaw ambiguity.
- 3D reconstruction and canonicalization: Category prompts improve SAM3 mask quality and reduce downstream failure cases, although masks can also be obtained without prompts.
- Pose refinement: The interior-coverage reward prevents pose optimization from placing the rendered silhouette outside the image in strongly occluded cases.
- Yaw canonicalization: 79 of 1,319 instances required non-zero yaw correction, corresponding to 5.99% of the dataset.
- PartField features: PartField features remain spatially coherent within semantic parts while discriminating repeated or symmetric structures.
- PartField rasterization: Rasterization assigns each image pixel the PartField feature of its corresponding visible 3D mesh point.
B.2 Feature fusion
Feature-fusion weights are selected by validation-set grid search, and several combinations perform similarly. The chosen default is α = 1/2, β = 1/3, and γ = 1/6, with PartField contributing to downstream gains.
- Weight search: The fusion weights are chosen by sweeping α and β in increments of 1/6 on SPair-71k validation pseudo-correspondences.
- Weight search: Several weight combinations reach similar peak PCK@0.10 performance, indicating limited sensitivity to exact weighting.
- Default fusion: The default weights are α = 1/2 for Stable Diffusion, β = 1/3 for DINOv2, and γ = 1/6 for PartField.
- Square-root weighting: Independent L2 normalization makes fused-space cosine similarity equivalent to a weighted average of source-specific cosine similarities.
- Evaluation: Table C1 reports per-image PCK at multiple thresholds on SPair-71k, AP-10K, and SpairU.
C.1 Additional implementation details
The implementation is evaluated on a single NVIDIA L40 GPU, with reconstruction and full-dataset pseudo-labeling runtimes reported.
- Compute: 12.42 s per object is the average runtime for canonicalized 3D object reconstruction.The measurement uses a single NVIDIA L40 GPU with 40 GB of memory.
- Compute: Roughly 18 h are required to compute pseudo-labels for the approximately 53k SPair-71k training pairs.The end-to-end runtime includes SD, DINO, and PartField extraction, rasterization, cyclic consistency, and geodesic filtering.
- Compute: The pipeline is also compatible with smaller memory budgets.
C.2 Additional results
Additional experiments show that 3D-SC gains are concentrated in rigid, symmetric categories, while qualitative pseudo-annotations are denser and more geometrically consistent than DIY-SC outputs.
- Supplementary evaluation: The supplementary tables include supervised methods and per-category SPair-71k results, while Geo-SC is excluded because its normalization prevents reliable direct comparison.
- Per-category results: Gains are largest in rigid, man-made categories with strong geometric symmetry, including bus (+10.8), tv/monitor (+9.8), and bottle (+8.8).Other reported gains include car (+6.9), train (+6.2), motorcycle (+5.1), and chair (+4.0).
- Per-category results: Non-rigid categories show slight regressions, including sheep (−2.7), cat (−1.5), cow (−1.7), and potted plant (−0.6).The passage attributes these patterns to weaker PartField generalization on deformable shapes and merged pot-and-plant reconstructions.
- Per-category results: Table C2 reports per-category PCK@0.1 scores using per-keypoint normalization on SPair-71k.
- Qualitative comparison: 3D-SC produces denser pseudo-annotations covering more object surface while remaining geometrically consistent and free from left-right ambiguities.DIY-SC pseudo-labels are described as sparser and more prone to symmetric confusions because of its spherical geometric prior.
D Reproducibility and LLM assistance
The authors plan to release the complete code, data, and reproducibility materials, and report limited LLM use for writing and coding assistance.
- Reproducibility: The complete processing pipeline, dataset-preparation scripts, training and inference code, configurations, and reproduction instructions will be publicly released.The materials are to be provided in the GenIntel/3D-SC repository.
- LLM assistance: LLMs were used only for sentence refinement, paragraph condensation, and coding assistance.The authors state that technical contributions and final implementation decisions were their own.