Source-linked AI summary
SPair-71k: A Large-scale Benchmark for Semantic Correspondence
Juhong Min, Jongmin Lee, Jean Ponce, Minsu Cho
TL;DR
Semantic correspondence remains difficult because existing benchmarks are small, limited in variation, or sparsely annotated. SPair-71k introduces 70,958 richly annotated image pairs and shows that viewpoint and scale changes substantially affect current methods.
Problem
Semantic correspondence lacks benchmarks with sufficient scale, viewpoint and scale diversity, direct correspondence annotations, and comprehensive evaluation labels.
Method
SPair-71k constructs 70,958 image pairs with keypoints, masks, bounding boxes, viewpoints, scale, truncation, and occlusion annotations across organized data splits.
Results
Current methods perform better with less variation, while viewpoint and scale changes significantly affect performance and truncation and occlusion also degrade it.
Takeaways & Limitations
SPair-71k provides a large, diverse, richly annotated benchmark for future research on semantic correspondence and related joint problems.
Takeaways & Limitations
For cylindrical categories, instances are treated as flat because front/back and left/right keypoint distinctions are difficult to discriminate.
Abstract
from arXiv · showhide
Establishing visual correspondences under large intra-class variations, which is often referred to as semantic correspondence or semantic matching, remains a challenging problem in computer vision. Despite its significance, however, most of the datasets for semantic correspondence are limited to a small amount of image pairs with similar viewpoints and scales. In this paper, we present a new large-scale benchmark dataset of semantically paired images, SPair-71k, which contains 70,958 image pairs with diverse variations in viewpoint and scale. Compared to previous datasets, it is significantly larger in number and contains more accurate and richer annotations. We believe this dataset will provide a reliable testbed to study the problem of semantic correspondence and will help to advance research in this area. We provide the results of recent methods on our new dataset as baselines for further research. Our benchmark is available online at http://cvlab.postech.ac.kr/research/SPair-71k/.
1. Motivation
Semantic correspondence is challenging because matching instances within the same category involves substantial intra-class variation, while existing benchmarks are too small, insufficiently varied, incompletely annotated, and inconsistently split. SPair-71k addresses these limitations with 70,958 image pairs, rich annotations, and clearly organized learning splits.
- Semantic correspondence establishes visual correspondences between different instances of the same object or scene category, but intra-class variation makes it notoriously challenging.
- PF-WILLOW and PF-PASCAL [4] provide ground-truth correspondences but contain only 900 and 1,300 image pairs, respectively, with keypoint annotations.
- Previous datasets are too small for large-model training, lack viewpoint, scale, occlusion, and truncation diversity, provide limited annotations, and lack clear train-validation-test splits.Their splits may share images across training and testing, and evaluations use different PF-PASCAL splits [5] [11].
- SPair-71k introduces 70,958 image pairs from PASCAL 3D+ [14] and PASCAL VOC 2012, with rich annotations and clearly organized learning splits.Annotations include semantic-part keypoints, segmentation masks, bounding boxes, viewpoint, scale, truncation, and occlusion differences.
2. Dataset generation and annotation
SPair-71k is generated from 1,800 PASCAL VOC images spanning 18 categories, with image- and pair-level annotations designed to capture viewpoint, scale, truncation, and occlusion variation. The resulting benchmark contains 70,958 disjoint image pairs across training, validation, and testing splits.
- Dataset generation and annotation: The dataset uses 1,800 images from 18 PASCAL VOC categories, including 1,000 images from 10 rigid PASCAL 3D+ categories.
- Dataset generation and annotation: Viewpoint and scale differences are categorized as easy, medium, or hard, while occluded or truncated keypoints are omitted and instances contain 3 to 30 annotated keypoints in practice.
- Dataset generation and annotation: Image-level splits are created before pairing, yielding 997 training, 322 validation, and 481 testing images distributed across quantized azimuth values.
- Dataset generation and annotation: Pair-level annotations combine common keypoint correspondences with azimuth, bounding-box, truncation, and occlusion information, excluding pairs without common keypoints.
- Dataset generation and annotation: 70,958 image pairs are obtained, comprising 53,340 training, 5,384 validation, and 12,234 testing pairs.
3. Baseline results on SPair-71k
The section establishes baseline evaluations of recent methods on SPair-71k using both authors’ original models and SPair-71k-finetuned versions. It also analyzes performance across viewpoint, scale, truncation, and occlusion variations, finding viewpoint and scale to be the most damaging factors.
- Baseline evaluation: Recent state-of-the-art methods [8] [9] [11] [12] are evaluated with both author-provided original models and models finetuned on the SPair-71k train/validation set.The original weakly supervised learning objective of one method appears fragile under large viewpoint differences, so its results require further investigation.
- Analysis by variation factors: Image alignment methods [9] [12] lose performance more quickly than region-matching methods as viewpoint variation increases.Performance drops from scale changes, truncation, and occlusion do not differ significantly across methods.
- Analysis by variation factors: Viewpoint and scale variations affect performance more strongly than truncation and occlusion, although both truncation and occlusion clearly degrade results.SPair-71k annotates viewpoint and scale as easy, medium, or hard, and truncation and occlusion by source, target, both, or none.
4. Conclusion
The paper introduces SPair-71k, a large-scale semantic correspondence benchmark with 71k image pairs, diverse real-world variations, and rich annotations. It is designed to generalize visual correspondence beyond prior datasets.
- 4. Conclusion: SPair-71k provides a large-scale benchmark dataset containing 71k image pairs for semantic correspondence.
- 4. Conclusion: The dataset includes diverse viewpoint, scale, truncation, and occlusion variations that reflect real-world scenarios.
- 4. Conclusion: SPair-71k offers rich annotations, including object bounding boxes, keypoint correspondences, and variation factors.