Source-linked AI summary
SCNet: Learning Semantic Correspondence
Kai Han, Rafael S. Rezende, Bumsub Ham, Kwan-Yee K. Wong, Minsu Cho, Cordelia Schmid, Jean Ponce
TL;DR
Semantic correspondence across different instances remains challenging because appearance and spatial layout can vary substantially, while prior methods emphasize hand-crafted appearance features or appearance-only learning. SCNet uses region proposals in an end-to-end CNN and incorporates geometric consistency into its loss. It substantially outperforms recent deep architectures and hand-crafted-feature methods on standard benchmarks.
Problem
Semantic correspondence must match different instances of the same object or scene category despite large appearance and spatial-layout variation.
Method
SCNet is an end-to-end CNN that uses object proposals as matching primitives and explicitly incorporates geometric consistency into its loss.
Results
SCNet substantially outperforms recent deep learning architectures and previous methods based on hand-crafted features on standard benchmarks.
Takeaways & Limitations
Learning both appearance and geometric terms yields state-of-the-art results across several semantic-correspondence benchmarks.
Takeaways & Limitations
The model assumes appearance matching is independent of geometry matching and the offset.
Abstract
from arXiv · showhide
This paper addresses the problem of establishing semantic correspondences between images depicting different instances of the same object or scene category. Previous approaches focus on either combining a spatial regularizer with hand-crafted features, or learning a correspondence model for appearance only. We propose instead a convolutional neural network architecture, called SCNet, for learning a geometrically plausible model for semantic correspondence. SCNet uses region proposals as matching primitives, and explicitly incorporates geometric consistency in its loss function. It is trained on image pairs obtained from the PASCAL VOC 2007 keypoint dataset, and a comparative evaluation on several standard benchmarks demonstrates that the proposed approach substantially outperforms both recent deep learning architectures and previous methods based on hand-crafted features.
1. Introduction
SCNet targets semantic correspondence across different instances of the same object or scene category, where appearance and spatial layout vary substantially. It learns matching from region proposals while incorporating geometric consistency, and outperforms recent deep and hand-crafted approaches on standard benchmarks.
- Motivation: Semantic correspondence matches different instances within the same object or scene category despite large appearance and spatial-layout changes.
- Prior approaches: Previous methods commonly combine spatial regularization with hand-crafted features such as SIFT, DAISY, or HOG.
- Approach: SCNet is a CNN architecture that uses object proposals as matching primitives and learns geometrically plausible correspondence.
- Approach: The model learns to match regions using both appearance and geometry, explicitly incorporating proposal geometric consistency into its loss.
- Results: Comparative evaluation shows substantial gains over recent deep architectures and methods based on hand-crafted features.
- Results: SCNet achieves state-of-the-art results on several benchmarks, demonstrating the advantage of learning both appearance and geometric terms.
2. Related work
Related work spans hand-crafted semantic-correspondence methods and CNN approaches developed for classical correspondence problems. Existing deep methods often address same-scene or same-object matching, while semantic correspondence requires handling different instances and limited ground-truth availability.
- Semantic correspondence: SIFT Flow and related methods establish semantic correspondences using dense descriptors, hierarchical optimization, or more efficient descriptor variants.
- Semantic correspondence: Classical semantic-correspondence approaches use descriptors such as SIFT or DAISY to capture information beyond raw color values.
- Deep learning for correspondence: CNNs have been applied to optical flow and stereo matching to learn feature descriptors or similarity functions.
- Deep learning for correspondence: These classical correspondence settings are inherently limited to matching images of the same physical object or scene.
- Deep learning for correspondence: Other semantic-correspondence methods use ImageNet-pretrained CNN features or 3D models to address the lack of datasets and ground truth.
3. Our approach
SCNet learns region correspondences by combining appearance similarity with geometric consistency over discretized offsets. Its trainable formulation uses sparse geometry-aware scoring and gradient computation to make learning practical.
- Region-based matching: SCNet matches object proposal boxes, covering arbitrary region positions and sizes in semantic correspondence.The model follows proposal flow and uses proposal boxes as its matching primitives.
- Appearance and geometry: The model combines appearance similarity with global geometric consistency by allowing matches to vote on potential offsets.The learned score rewrites probabilistic Hough matching in terms of parameterized similarities and geometric compatibility.
- Learning objective: SCNet learns its similarity function from true and false matches by minimizing a loss with a regularizer, using hinge loss and L2 regularization.At test time, each region is associated with the region receiving the highest learned score.
- Learned similarity: The architecture uses a trainable embedding for appearance similarity, with rectified cosine similarity setting negative values to zero.The embedding function can be implemented by a differentiable architecture such as a CNN.
- Geometry kernel: The geometry kernel groups potential matches into discretized offset bins and scores each match using other matches assigned to the same bin.Offsets encode position and scale changes, while the sparse kernel enables sharing across matches in a bin.
- Gradient-based learning: Sparse geometry grouping reduces gradient computation by restricting each match to a small set of matches in its offset bin instead of all matches.The full gradient can require up to n^2 evaluations, whereas the sparse kernel substantially reduces this cost.
4. SCNet architecture
SCNet is a CNN architecture that efficiently matches region proposals while learning appearance and geometric compatibility. Its variants compare shared or separate geometry embeddings, with training pairs and sampling defined using proposal overlap.
- Architecture: SCNet processes image pairs through shared CNN feature maps and parallel ROI pooling for region proposals.Convolutional layers are shared across all proposals, and ROI pooling produces same-sized proposal feature maps.
- Architecture: SCNet-AG embeds proposal features as normalized vectors and combines rectified appearance similarity with geometric consistency.The geometry term is enforced through the paper's voting-layer kernel.
- Architecture: SCNet-A removes the geometry term and voting layer, using appearance similarity directly as the match score.This variant provides an architecture without geometric modeling for comparison.
- Architecture: SCNet-AG+ learns a separate geometry embedding, allowing geometric scoring to differ from appearance scoring across rigid and deformable parts.The separate similarity function is intended to provide greater flexibility for geometric scoring.
- Training: Training samples positives with proposal IoU above 0.6 and negatives below 0.4, selecting high-appearance-similarity negatives among eligible proposals.For each proposal, the procedure constructs equal numbers of positive and negative pairs.
5. Experimental evaluation
SCNet is evaluated on multiple correspondence benchmarks using region proposals, dense-flow transfer, and several comparison methods. Results show consistent advantages for learned appearance and geometric matching, while proposal choice and evaluation setting affect performance.
- Experimental setup: PF-PASCAL contains 1300 image pairs from 20 object classes, split into 700 training, 300 validation, and 300 testing pairs.Testing uses 1000 region proposals per image, while training uses 500 for efficiency.
- Experimental setup: The evaluation uses PCK for dense flow, PCR for region matching, and mIoU@k for the quality of top-k proposal matches.PCK evaluates sparse keypoints, whereas PCR and mIoU@k evaluate region-based correspondences.
- Region matching: SCNet models outperform HOG-based proposal-flow methods on PF-PASCAL, and geometric variants SCNet-AG and SCNet-AG+ substantially outperform appearance-only SCNet-A.SCNet-AG+ slightly outperforms SCNet-AG; fully fine-tuning VGG16 provides only marginal improvement over partial learning.
- Region matching: Replacing HOG with SCNet features improves NAM, PHM, and LOM, although LOM is more than 10 times slower than SCNet-AG+.LOM takes 3.15s for matching after 0.21s of feature extraction, compared with 0.33s total for SCNet-AG+.
- Region matching: RP proposals perform best among RP, selective search, uniform sampling, and sliding-window proposals, while uniform sampling performs worst by a large margin.This result indicates that the region proposal process is an important factor for matching performance.
- Dense correspondence: On PF-PASCAL, the three SCNet variants achieve PCK of 66.3 or above and consistently outperform UCN and Proposal Flow methods.The raw network trained on a different PASCAL subset obtains mean PCK 36.0, versus 55.6 for the retrained UCN.
- Dense correspondence: On PF-WILLOW, all SCNet models significantly outperform UCN and other hand-crafted or learned-feature methods, while SCNet-AG slightly exceeds SCNet-A and SCNet-AG+.SCNet trained on PF-PASCAL is evaluated on PF-WILLOW to assess transferability across distributions.
- Additional benchmarks: SCNet achieves comparable results on Caltech-101 and outperforms all other methods in IoU on PASCAL Parts, although FCSS performs better in PCK there.SCNet-AG+ is the best-performing SCNet model on PASCAL Parts.
6. Conclusion
SCNet learns semantic correspondence by matching object proposals using both appearance and geometry. The model substantially outperforms recent deep architectures and hand-crafted-feature methods.
- SCNet uses object proposals as matching primitives and learns correspondence from both appearance and geometry.The corresponding CNN architecture is designed to learn geometrically plausible semantic correspondence.
- The proposed method substantially outperforms both recent deep learning architectures and previous methods based on hand-crafted features.The authors state that this result demonstrates the effectiveness of learning geometric matching for semantic correspondence.
- Future work will explore better models and architectures for leveraging geometric information.