PaperlayerView exact revision on arXiv

Source-linked AI summary

Telling Left from Right: Identifying Geometry-Aware Semantic Correspondence

Junyi Zhang, Charles Herrmann, Junhwa Hur, Eric Chen, Varun Jampani, Deqing Sun, Ming-Hsuan Yang

arXiv:2311.17034v2cs.CV

TL;DR

Foundation-model features can struggle to distinguish geometry-ambiguous matches between semantically similar parts. The paper proposes methods for improving geometric awareness and builds an AP-10K-based benchmark for training and evaluation. On SPair-71k, the supervised method improves over existing works, while severe pose variations remain challenging.

  • Problem

    Foundation-model features underperform on geometry-aware semantic correspondences, where semantically similar parts have different relations to an instance's geometry.

    Source: §1 · p. 1

  • Method

    The paper improves geometric awareness with test-time pose alignment and trained post-processing, and constructs an animal-pose-based benchmark from AP-10K.

    Source: §1 · p. 2

  • Results

    11.0p improvement in the supervised category, from 74.6 to 85.6, with methods outperforming existing works across all 18 categories.

    Source: §5.1 · p. 7

  • Takeaways & Limitations

    The method improves overall semantic correspondence and narrows the performance gap between geometry-aware and standard subsets.

    Source: §6 · p. 8

    Pose variations with severe deformation remain challenging; future work may address them with advanced reasoning or geometric priors.

    Source: §5.2 · p. 8

Abstract

from arXiv · show

While pre-trained large-scale vision models have shown significant promise for semantic correspondence, their features often struggle to grasp the geometry and orientation of instances. This paper identifies the importance of being geometry-aware for semantic correspondence and reveals a limitation of the features of current foundation models under simple post-processing. We show that incorporating this information can markedly enhance semantic correspondence performance with simple but effective solutions in both zero-shot and supervised settings. We also construct a new challenging benchmark for semantic correspondence built from an existing animal pose estimation dataset, for both pre-training validating models. Our method achieves a PCK@0.10 score of 65.4 (zero-shot) and 85.6 (supervised) on the challenging SPair-71k dataset, outperforming the state of the art by 5.5p and 11.0p absolute gains, respectively. Our code and datasets are publicly available at: https://telling-left-from-right.github.io/.

1. Introduction

The paper uses semantic correspondence to examine what vision foundation-model features struggle with, focusing on geometry-ambiguous matches between semantically similar parts. It proposes methods to improve geometric awareness and introduces an animal-pose-based benchmark for training and evaluation.

1. Introduction

  • Foundation-model features underperform on geometry-aware correspondences, which make up nearly 60% of SPair-71k and can be up to 30% harder for state-of-the-art methods.These matches involve semantically similar parts with different relations to an object's geometry, such as left versus right paws.Source: §1 · p. 1
  • The proposed methods address geometric ambiguity through test-time viewpoint alignment and a trained post-processing module, with pose-variant augmentation and window soft-argmax.The paper reports a 15% improvement on standard benchmarks at 0.32% extra runtime.Source: §1 · p. 2
  • The new AP-10K-based benchmark supports training and evaluation, includes five times more training pairs than the largest existing benchmark, and evaluates cross-species and cross-family correspondence.The paper also reports that it can serve as a pre-training resource for geometry-aware semantic correspondence.Source: §1 · p. 2
  • 85.6 PCK@0.10 on SPair-71k: the method outperforms the state of the art by more than 15%.The paper reports improvements in both unsupervised and supervised settings and particularly on the geometry-aware subset.Source: §1 · p. 2

2. Related Work

Prior semantic-correspondence methods use feature extraction, cost-volume computation, and matching-field regression, while newer foundation-model features provide powerful descriptors. The paper identifies limitations in geometry awareness and in the scale and scope of existing benchmarks.

2. Related Work

  • Conventional semantic-correspondence pipelines extract features, compute a cost volume, and regress a matching field.Prior approaches address intra-class variation through matching uniqueness, spatial priors, end-to-end regression, and unsupervised or weakly supervised techniques.Source: §2 · p. 2
  • Earlier semantic-correspondence methods also struggle with large pose changes and non-rigid deformation because of limited feature capacity or strong spatial priors.Source: §2 · p. 2
  • Foundation-model features can outperform earlier matching-specific methods, but still struggle with intrinsic instance geometry; existing benchmarks also have limited scale or variation.The paper introduces an AP-10K-based benchmark with training annotations to support broader evaluation of geometric awareness.Source: §2 · p. 2

3. Geometric Awareness of Deep Features

Geometry-aware correspondences require matching semantically similar parts according to instance orientation, a challenge that affects a substantial share of SPair-71k keypoints. Foundation features perform worse on these cases, which are more sensitive to pose variation, even though some features encode global pose information.

3. Geometric Awareness of Deep Features

  • The section analyzes geometry-aware matching and evaluates both its performance and the pose information encoded in pretrained features.Source: §3 · p. 2

3.1. Geometry-Aware Semantic Correspondence

  • Geometry-aware correspondences involve semantically similar parts whose correct match depends on instance orientation, such as distinguishing left from right paws.The geometry-aware subset is formed by clustering keypoints by semantic part and collecting cases with geometry-ambiguous matches.Source: §3.1 · p. 3, §3.2 · p. 3

3.2. Evaluation on the Geometry-aware Subset

  • Across all methods, the geometry-aware subset shows a substantial performance gap versus the standard set: around 20% for zero-shot methods and 10% for supervised methods.Source: §3.2 · p. 3

3.3. Sensitivity to Pose Variation

  • Across categories, geometry-aware correspondence performance is more sensitive to pose variation than performance on the standard set.Pose variation is assessed across five subsets ordered by annotated azimuth difference.Source: §3.3 · p. 3, Figure 5 · p. 3

3.4. Global Pose Awareness of Deep Features

  • SD+DINO achieves near-perfect rough pose prediction, while DINOv2 struggles to distinguish left from right but excels at front-versus-back prediction.The study evaluates DINOv2, SD, and fused SD+DINO on manually labeled cat images.Source: §3.4 · p. 4

4. Improving Geo-Aware Correspondence

The paper addresses geometry-aware correspondence with test-time pose alignment and a trainable feature post-processor. Its training combines dense and sparse objectives with pose-variant augmentation, while window soft argmax improves correspondence by balancing sub-pixel reasoning with resistance to noisy responses.

4.1. Test-time Adaptive Pose Alignment

  • At test time, pose alignment selects the source-image augmentation with the smallest IMD to reduce pairwise pose variation without training.The strategy uses pose-variant augmentations such as flips and rotations.Source: §4.1 · p. 4

4.2. Dense Training Objective

  • The dense objective applies soft-argmax to similarity maps and penalizes predicted-to-target keypoint distance, complementing sparse keypoint-feature contrastive training.The final objective combines dense and sparse losses; input feature-map dropout and Gaussian perturbations of ground-truth positions are also used.Source: §4.2 · p. 5

4.3. Pose-variant Augmentation

  • Pose-variant augmentation reuses a feature map from one flipped image across double-flip, single-flip, and self-flip settings, with keypoints flipped to preserve part identity.The self-flip setting is intended to teach the model to distinguish concepts rather than match keypoints by appearance alone.Source: §4.3 · p. 5

4.4. Window Soft Argmax

  • Both proposed zero-shot and supervised methods outperform prior arts across all categories on SPair-71k.Source: Table 2 · p. 6
  • Window soft argmax first locates a target center with argmax, then applies soft-argmax locally to enable sub-pixel reasoning while limiting noisy responses.The authors report that it substantially improves correspondence performance on all metrics.Source: §4.4 · p. 6

5. Experimental Results

The experiments evaluate the methods on established benchmarks and the new AP-10K benchmark, reporting overall and geometry-aware results alongside qualitative comparisons and limitations.

5. Experimental Results

  • Features are extracted at 60 × 60 resolution, and a four-layer bottleneck post-processor has 5M parameters.The input images are resized to 9602 for SD features and 8402 for DINOv2 features.Source: §5 · p. 6
  • The AP-10K benchmark contains 261k training, 17k validation, and 36k test pairs, including intra-species, cross-species, and cross-family evaluation.It is five times larger than the largest existing benchmark.Source: §5 · p. 6

5.1. Quantitative Analysis

  • 11.0p improvement, from 74.6 to 85.6, as supervised methods outperform existing works across all 18 SPair-71k categories.Table 3 reports results across SPair-71k, AP-10K, and PF-Pascal; the zero-shot approach also gains over SD+DINO.Source: Table 3 · p. 7, §5.1 · p. 7
  • The methods improve results across most reported datasets and PCK thresholds, with notable gains at stricter thresholds and on cross-species and cross-family tests.Training uses AP-10K intra-species sets for these cross-species and cross-family evaluations.Source: Table 3 · p. 7, §5.1 · p. 7
  • A 3.86% relative gap on SPair-71k geometry-aware PCK@0.10, down from 9.38% for SD+DINO (S), with AP-10K pre-training adding 4.3p on the geometry-aware subset.Adaptive viewpoint alignment brings larger gains on the geometry-aware subset in both zero-shot and supervised settings.Source: Table 4 · p. 7, §5.1 · p. 7
  • Dense training, pose-variant augmentation, and window soft argmax improve geometry-aware results, while ground-truth perturbation and feature-map dropout improve overall correspondence.Window sizes are set to 15 × 15 for supervised and 11 × 11 for unsupervised settings.Source: Table 5 · p. 7, §5.1 · pp. 7–8

5.2. Qualitative Analysis

  • Under extreme viewpoint variation, the method produces geometrically correct matches where both zero-shot and supervised SD+DINO struggle with geometric ambiguity.For an unsupervised query point, the method identifies semantically and geometrically correct points where SD+DINO matches appearance-similar regions and SD+DINO (S) produces a noisy map.Source: Figure 10 · p. 8, §5.2 · p. 8, Figure 11 · p. 8
  • Small instances challenge the method because of raw-feature-map resolution limits, and it may fail with extreme pose variation combined with severe deformation.Advanced reasoning or geometric priors are suggested as possible future directions.Source: Figure 12 · p. 8, §5.2 · p. 8

6. Conclusion

The paper identifies geometry ambiguity in semantic correspondence and introduces techniques and a benchmark to address it. Experiments show improved overall correspondence and a narrower gap between geometry-aware and standard subsets.

6. Conclusion

  • The methods improve overall semantic correspondence and narrow the gap between geometry-aware and standard subsets.The authors connect these gains to downstream tasks and analysis of foundation-model representations.Source: §6 · p. 8

A. Further Implementation Details

The appendix describes feature extraction, pose augmentations, and training procedures.

A. Further Implementation Details

  • SD features come from SD-1-5 UNet decoder layers 2, 5, and 8 at timestep 50; DINOv2 features come from the 11th layer token facet.The SD extraction uses an implicit captioner.Source: A. Further Implementation Details · p. 11
  • Horizontal flips provide primary viewpoint augmentation; bottle instances also receive rotations of +90°, 180°, and -90°.Pose-variant augmentation is not applied on PF-Pascal because its pairs have similar poses.Source: A. Further Implementation Details · p. 11
  • Training runs for 100k steps on SPair-71k and 250k steps on AP10K and PF-Pascal, with pre-extracted foundation-model features and online post-processor training.The reported training duration is a few hours on one GPU.Source: A. Further Implementation Details · p. 12

B. Benchmarking AP-10K Dataset for Semantic Correspondence

The AP-10K benchmark filters ambiguous images and pairs, balances validation and test sampling across species, and includes within- and cross-species/family evaluation.

B. Benchmarking AP-10K Dataset for Semantic Correspondence

  • Images with fewer than three visible keypoints or multiple target instances are excluded before splitting.Three species with fewer than 50 filtered images are held out from training to assess generalization.Source: B. Benchmarking AP-10K Dataset for… · p. 12
  • Validation and test sets uniformly sample 20 and 30 images per species, respectively, while remaining images form the training set.The sampling aims to balance evaluation across species.Source: B. Benchmarking AP-10K Dataset for… · p. 12
  • After requiring at least three mutually visible keypoints per pair, intra-species sampling yields 260,950 training, 8,816 validation, and 20,630 testing pairs.Training pairs are capped to reduce imbalance from quadratic pairing growth.Source: B. Benchmarking AP-10K Dataset for… · p. 12
  • Cross-species and cross-family sampling yields 4,300 and 4,200 validation pairs, and 9,619 and 6,300 testing pairs, respectively.Cross-family pairs are sampled across the 21 families with reduced sampling to save computation.Source: B. Benchmarking AP-10K Dataset for… · p. 12

C. Details on Geo-Aware Correspondence

The geometry-aware subset is substantial for most categories, and performance on it lags the standard set. Pose variation sensitivity is also higher on this subset, while the reported per-category comparison shows our methods outperforming previous methods.

C. Details on Geo-Aware Correspondence

D.2. Detailed Analysis on Window Soft Argmax

Window soft argmax performance varies with window size, and a size of 15 is chosen as a balance across PCK thresholds. Across dataset-transfer tests and ablations, the method reports gains in generalizability and geometry-related error metrics, with adaptive pose alignment and window soft argmax improving unsupervised results.

D.2. Detailed Analysis on Window Soft Argmax

  • PCK performance initially rises and then falls as the window grows from 0 to 60; peak values occur at sizes 5, 11, and 17 for PCK@0.01, PCK@0.05, and PCK@0.1.A window size of 15 is selected for balance.Source: §D.2 · p. 14, Figure 18 · p. 15
  • Window soft argmax outperforms Gaussian-kernel argmax across the reported PCK settings, with the authors hypothesizing that equal scaling within its window avoids bias toward the hard-argmax location.Source: §D.2 · pp. 14–15, Table 7 · p. 15
  • Applying window soft argmax during training hurts performance at loose PCK thresholds but helps at the stricter PCK@0.01 threshold.The authors hypothesize that training with windows emphasizes local regions at the expense of global information.Source: §D.2 · p. 15, Table 8 · p. 15

D.3. Discussion on Generalizability

D.4. Additional Ablation Analysis under Supervised Setting

  • 37.1%, 36.5%, 36.0%, and 40.4% improvements are reported for Jitter, Miss, Swap, and SwapLR, respectively.The analysis attributes Jitter, Swap, and SwapLR gains to spatial context from dense training and window soft argmax; pose-variant pair augmentation particularly reduces SwapLR.Source: §D.4 · p. 16, Table 9 · p. 16
  • The dense training objective is also reported to reduce Miss errors, which the authors hypothesize reflects suppression of background noise.The breakdown metrics include Jitter, Miss, Swap, PCK†, and the added left/right confusion metric SwapLR.Source: §D.4 · p. 16, Table 9 · p. 16

D.5. Ablation Study under Unsupervised Setting

  • Adaptive pose alignment improves performance across all five inference settings, while window soft argmax boosts both geometry-aware and standard-set results with or without alignment.Source: §D.5 · p. 16

E.1. Alternative Metrics for Pose Alignment

Alternative pose-alignment metrics with relaxed requirements yield competitive results that surpass the baseline. Qualitative comparisons cover AP-10K intra-species, cross-species, and cross-family subsets, as well as SPair-71k, where the method establishes correspondences in challenging cases.

E.1. Alternative Metrics for Pose Alignment

  • Both querypoint-based segmentation and a mutual-nearest-neighbor distance metric yield competitive pose-alignment results that significantly surpass the baseline.The latter eliminates the need for masks; querypoint segmentation uses source-image keypoints to obtain an instance mask.Source: §E.1 · p. 17, Table 13 · p. 17

E.2. Qualitative Results on AP-10K

E.3. Additional Qualitative Results on SPair-71k

Related papers

Loading 2311.17034v2…