PaperlayerView exact revision on arXiv

Source-linked AI summary

MARCO: Navigating the Unseen Space of Semantic Correspondence

Claudia Cuttano, Gabriele Trivigno, Carlo Masone, Stefan Roth

arXiv:2604.18267v1cs.CV

TL;DR

Semantic correspondence models achieve strong accuracy but often generalize poorly beyond annotated keypoints, while diffusion-based dual encoders are large and costly. MARCO builds on DINOv2 with coarse-to-fine localization and dense self-distillation that propagates sparse supervision across object surfaces. It achieves state-of-the-art performance across standard and generalization benchmarks while remaining substantially smaller and faster than dual-encoder methods.

  • Problem

    Dual-encoder correspondence models combining DINOv2 and diffusion backbones are accurate but generalize poorly beyond training keypoints, creating a gap between benchmark performance and real-world queries.

  • Method

    MARCO adds lightweight adapters and an upsampling head to DINOv2, combining coarse-to-fine supervision with dense self-distillation from reliable feature correspondences.

  • Results

    +8.9 PCK@0.01 on SPair-71k, +5.1 on SPair-U, and +5.6 on MP-100 are reported, while MARCO is 3× smaller and 10× faster than dual-encoder methods.

  • Takeaways & Limitations

    A single DINOv2 backbone can provide accurate pixel-level correspondences while generalizing to unseen landmarks and novel categories with minimal computational overhead.

Abstract

from arXiv · show

Recent advances in semantic correspondence rely on dual-encoder architectures, combining DINOv2 with diffusion backbones. While accurate, these billion-parameter models generalize poorly beyond training keypoints, revealing a gap between benchmark performance and real-world usability, where queried points rarely match those seen during training. Building upon DINOv2, we introduce MARCO, a unified model for generalizable correspondence driven by a novel training framework that enhances both fine-grained localization and semantic generalization. By coupling a coarse-to-fine objective that refines spatial precision with a self-distillation framework, which expands sparse supervision beyond annotated regions, our approach transforms a handful of keypoints into dense, semantically coherent correspondences. MARCO sets a new state of the art on SPair-71k, AP-10K, and PF-PASCAL, with gains that amplify at fine-grained localization thresholds (+8.9 PCK@0.01), strongest generalization to unseen keypoints (+5.1, SPair-U) and categories (+4.7, MP-100), while remaining 3x smaller and 10x faster than diffusion-based approaches. Code is available at https://github.com/visinf/MARCO .

1. Introduction

MARCO addresses the gap between strong benchmark accuracy and poor generalization beyond annotated keypoints by combining fine-grained localization with dense semantic supervision. Built on DINOv2, it uses lightweight architectural additions and complementary training objectives to improve precision, generalization, and efficiency.

  • Motivation: Semantic correspondence seeks pixel-level matches between equivalent object regions despite pose, texture, and viewpoint variation, with supervision typically limited to sparse locations.Accurate correspondences support image editing, pose estimation, style transfer, and affordance understanding.
  • Architecture: MARCO adds bottleneck adapters and a compact upsampling head to DINOv2, increasing parameters by less than 5% while restoring sub-patch resolution.The model follows a minimalist single-backbone design rather than combining DINOv2 with diffusion features.
  • Training objectives: The coarse-to-fine objective narrows a Gaussian target distribution during training, progressing from region-level alignment to subpatch-accurate localization.This improves standard thresholds and resolves finer subregions such as pupils rather than only coarse parts such as eyes.
  • Training objectives: MARCO mines reliable feature matches, densifies them with piecewise-affine interpolation over Delaunay triangles, and uses geometrically consistent matches to extend supervision across object surfaces.The resulting self-supervision expands a handful of annotated keypoints into dense correspondences while avoiding predefined 3D structures and category priors.
  • Results: +8.9 PCK@0.01 is achieved on SPair-71k, while MARCO improves over Jamais Vu by +5.1 on SPair-U and +5.6 on MP-100; it is also 3× smaller and 10× faster than dual-encoder methods.These results cover fine localization, unseen keypoints, unseen categories, and computational efficiency.

2. Related Work

Related work develops semantic correspondence through emergent foundation-model features, self-supervised dense matching, and benchmarks designed to test generalization beyond training landmarks and categories. MARCO follows the single-backbone direction while addressing limitations in existing generalization evaluations and category-specific approaches.

  • Emergent correspondences in foundation models: DINOv2 provides semantically rich features, while Stable Diffusion supplies fine-grained spatial detail, motivating dual-encoder correspondence models.MARCO instead enhances DINOv2 directly with lightweight adapters and an upsampling layer.
  • Self-supervised dense semantic correspondence: Self-supervised correspondence methods avoid dense annotation through cost volumes, spatial smoothness constraints, transformer attention, implicit neural fields, or refined 2D flow fields.These approaches target dense matching without relying exclusively on densely labeled correspondences.
  • Generalization in semantic correspondence: Standard benchmarks such as PF-PASCAL, AP-10K, and SPair-71k primarily evaluate landmarks seen during training, limiting assessment of generalization.SPair-U introduces a small number of unseen keypoints to address this limitation.
  • Generalization in semantic correspondence: Jamais Vu improves transfer to novel keypoints through category-specific 3D canonical representations, but remains tied to the training taxonomy.Its approach also depends on monocular depth estimation for 3D geometry.

3. Semantic Correspondence with MARCO

MARCO builds on DINOv2 with lightweight architectural refinements and training objectives that improve fine-grained localization while extending sparse correspondence supervision across object surfaces. Its pipeline mines reliable matches, densifies them into flow fields, filters them through flow clustering and ground-truth anchoring, and trains with self-distillation.

  • Semantic correspondence establishes pixel-level matches between semantically equivalent keypoints despite pose, texture, viewpoint, and sparse supervision challenges.The task supports applications including image editing, pose estimation, style transfer, and affordance understanding.
  • 3.1. Architecture: MARCO combines a frozen DINOv2 backbone with lightweight adapters and a compact upsampling layer for efficient feature refinement and sub-patch localization.The adapters refine high-level features with minimal parameter overhead, while upsampling increases feature resolution by ×4.
  • 3.2. Coarse-to-fine objective for precise localization: The coarse-to-fine objective replaces coordinate regression with Gaussian RBF distribution matching, narrowing the target bandwidth during training from broad alignment to precise localization.A wide kernel supports stable region-level matching, while progressively smaller bandwidth improves fine-grained localization.
  • 3.3. Dense self-distillation via flow anchoring: MARCO mines mutual nearest-neighbor matches from teacher features and combines them with annotated keypoints to form a reliable sparse seed set.The seed set is restricted to object regions using masks or bounding boxes before densification.
  • 3.3. Dense self-distillation via flow anchoring: Piecewise-affine warps over Delaunay triangles convert sparse seeds into a continuous dense correspondence field across the source object.Affine transformations between paired triangles define a continuous warp and its displacement map.
  • 3.3. Dense self-distillation via flow anchoring: Flow-space clustering and ground-truth anchoring retain coherent regions whose motion agrees with annotated correspondences, producing pseudo-labels for self-distillation.This filtering addresses erroneous matches caused by object symmetries, occlusions, and model inaccuracies.

4. Experiments

MARCO is evaluated on standard correspondence benchmarks, unseen-keypoint and unseen-category protocols, and component ablations. It achieves state-of-the-art accuracy while improving fine localization and generalization through lightweight architecture and dense self-supervision.

  • Benchmarks: MP-100 evaluates unseen keypoints and semantic categories across five macro-domains, while SPair-U evaluates novel keypoints on seen categories.
  • Standard benchmarks: MARCO reaches state-of-the-art results on SPair-71k, AP-10K, and PF-PASCAL, with especially strong gains at PCK@0.01.On SPair-71k, it reaches 87.2 PCK@0.10 and surpasses Geo-SC by +4.0 points.
  • Generalization: MARCO improves over Jamais Vu by +5.1 PCK@0.10 on SPair-U, reaching 67.5, and gains across all reported MP-100 domains.Reported MP-100 gains include +2.0 on Human face, +10.2 on Apparel items, +7.7 on Home furniture, +3.0 on Animal body, and +4.9 on Animal face.
  • Ablations: Adapters and the upsampling head substantially improve localization, with gains of +8.1@0.01 and +12.6@0.01, respectively.The upsampling head restores sub-patch structure, while together the lightweight components turn DINOv2 into a precise correspondence backbone.
  • Ablations: The coarse-to-fine schedule balances fine and coarse localization, reaching 27.0@0.01 and 87.2@0.10 without the collapses of fixed-width targets.Fixed σ = 1 maximizes fine localization but reduces coarse accuracy and generalization, whereas fixed σ = 3 has the opposite trade-off.
  • Ablations: Dense self-supervision is essential for generalization: removing it causes a -25.7 PCK@0.10 collapse on SPair-U, while Delaunay densification adds +12.2 points.The densified fields convert sparse matches into coherent, locally smooth correspondence fields.

5. Conclusion

The paper presents MARCO as a single-backbone approach that jointly addresses correspondence precision and generalization. It points toward efficient models that can handle unseen keypoints and taxonomies, while identifying reduced reliance on sparse supervision as future work.

  • MARCO uses a single DINOv2 backbone trained to progressively refine spatial detail for accurate pixel-level correspondences.
  • Reliable structure emerging in DINOv2 feature space supports generalization to unseen landmarks and novel taxonomies.
  • Future work should further reduce dependence on sparse supervision and exploit large-scale unconstrained data.

Supplementary Material

The supplementary material extends MARCO with analyses of applicability, pre-training, and additional experiments.

  • The appendix studies whether dense self-distillation produces coherent, generalizable representations when applied to previous state-of-the-art methods.
  • It also presents a variant pre-trained on AP-10K and reports additional experiments.

A. Broad Applicability of our Dense Self-Distillation via Flow Anchoring

Flow-anchoring dense self-distillation generalizes beyond MARCO’s architecture, improving Geo-SC on unseen keypoints without degrading its in-domain accuracy. The loss is integrated with MARCO’s coarse-to-fine supervised objective.

  • Dense self-distillation via flow anchoring mines reliable feature correspondences and propagates them across objects through piecewise-affine interpolation.
  • Adding the flow-anchoring loss improves Geo-SC on SPair-U from 56.9 to 63.4 PCK@0.10.
  • On SPair-71k, the same modification raises Geo-SC from 83.2 to 83.6 PCK@0.10 rather than reducing in-domain accuracy.

B. Pre-training on AP-10k

MARCO is evaluated with AP-10K pre-training to enable fair comparison with recent correspondence methods that use the same protocol.

  • AP-10K pre-training is a common component of recent semantic correspondence evaluation protocols.Geo-SC introduced this strategy and benchmark, and subsequent methods including Jamais Vu adopted it.
  • MARCO includes an AP-10K-pretrained variant for direct comparison with approaches using this setting.

C. Additional Ablations

Additional ablations show that parameter-efficient adaptation, coarse-to-fine supervision, and dense self-distillation jointly improve precision and generalization while reducing reliance on extra supervision.

  • 67.0 PCK@0.10 on SPair-71k versus 43.9 on SPair-U shows that full DINOv2 fine-tuning harms generalization.Keeping the backbone frozen yields 54.9 on SPair-U, while QKV-only fine-tuning reaches 59.6.
  • AdaptFormer provides the best trade-off between in-domain accuracy and generalization among the tested parameter-efficient strategies.
  • 26.8 PCK@0.01 and 85.6 PCK@0.10 result after replacing InfoNCE+ℓ2 with coarse-to-fine supervision.The progressive ablation starts from 6.3 and 53.9, reaching 20.0 and 78.9 after adding adapters, upsampling, and standard supervision.
  • 67.5 PCK@0.10 on SPair-U follows dense self-distillation, up from 42.0, while SPair-71k reaches 87.2.
  • 13k average pseudo-labels per object achieve 67.5 PCK@0.10 after GT anchoring, compared with 17k labels and 64.7 without anchoring.Performance remains stable through σ = 5 pixels and begins degrading around σ = 10 pixels.
  • Removing SAM-based masks changes accuracy only slightly, from 27.0 to 26.6 PCK@0.01 on SPair-71k and from 67.5 to 66.9 on SPair-U.Masks therefore act as a mild spatial prior during pseudo-label mining rather than essential supervision.

E. Computational Cost

MARCO achieves a substantially smaller computational footprint than diffusion-based dual-encoder approaches while following the same evaluation protocol.

  • 323M parameters versus 950M makes MARCO substantially smaller than Geo-SC and Jamais Vu.MARCO uses a single DINOv2 backbone with adapters, whereas the comparison methods combine Stable Diffusion and DINOv2.
  • 8.3 FPS versus 0.85 FPS gives MARCO roughly a 10× inference speedup on an RTX4090.All methods use 840p feature extraction, batched image pairs, and the same soft-argmax prediction.

G. Details on the MP-100 Benchmark

MP-100 is introduced as a broad benchmark for testing correspondence generalization to unseen keypoints and categories across diverse object domains. Results show consistent gains for MARCO, especially on unseen categories and highly variable classes.

  • Benchmark motivation: MP-100 contains 100 categories and 18k images, compared with 1.8k images in SPair-71k.The benchmark broadens evaluation beyond the narrower SPair-71k dataset.
  • Data curation: 2k image pairs are sampled within each of five domains using stratified sampling to avoid class imbalance.The domains are human face, apparel items, furniture, animal face, and animal body.
  • Evaluation splits: The benchmark separates unseen-keypoint evaluation within familiar categories from unseen-category evaluation under strict zero-shot conditions.Its five splits cover human face, apparel items, animal body, animal face, and home furniture.
  • Aggregate results: +4.3 %, +4.5 %, and +5.4 % are MARCO’s average gains over the strongest prior method on unseen categories at PCK@0.05, PCK@0.10, and PCK@0.15.The gains are especially pronounced on Home furniture and remain consistent on Animal body and Animal face.
  • Unseen-keypoint results: +1.5 %, +5.7 %, and +9.0 % are MARCO’s gains over SD+DINO on Apparel items at PCK@0.05, PCK@0.10, and PCK@0.15.For Human face, MARCO is slightly weaker at PCK@0.05 but improves over DIFT by +0.2 % at PCK@0.10 and +1.4 % at PCK@0.15.
  • Per-category results: +14.7 on bed and +11.3 on olive baboon are among MARCO’s strongest per-category gains over Jamais Vu.Difficult categories such as table, locust, fly, and polar bear remain relatively close across methods.

H. Qualitative Examples

MARCO is qualitatively evaluated across appearance changes, occlusions, viewpoint variations, unseen keypoints, and unseen categories. The examples emphasize accurate, spatially coherent, and stable correspondences beyond the training distribution.

  • MARCO produces accurate and spatially coherent correspondences under appearance changes, occlusions, and viewpoint variations on SPair-71k.
  • The MP-100 examples evaluate generalization to unseen keypoints and unseen categories.They focus on novel landmark definitions and object types with distinct shapes and geometries.
  • MARCO adapts to landmark definitions never used during training and remains stable on novel object types.
  • Qualitative results visually confirm the generalization encouraged by MARCO’s dense self-distillation framework.
Loading 2604.18267v1…