Source-linked AI summary
FCSS: Fully Convolutional Self-Similarity for Dense Semantic Correspondence
Seungryong Kim, Dongbo Min, Bumsub Ham, Sangryul Jeon, Stephen Lin, Kwanghoon Sohn
TL;DR
Dense semantic correspondence must match points across different instances of the same class despite appearance variation, with limited training data available. FCSS learns local self-similarity in a multi-scale fully convolutional network using weak supervision, and outperforms handcrafted and CNN-based descriptors on various benchmarks.
Problem
Semantic correspondence must handle substantial appearance differences between same-class instances, while available training data are limited.
Source: §1 · p. 1Method
FCSS learns local self-similarity sampling patterns and measures in a multi-scale fully convolutional network, using correspondence consistency and object-location priors for weak supervision.
Source: §1 · p. 2, §3.4 · p. 5Results
FCSS outperforms conventional handcrafted and CNN-based descriptors on various benchmarks.
Source: §1 · p. 2Takeaways & Limitations
FCSS may benefit instance-level object detection and segmentation through its robustness to intra-class variation and precise localization.
Source: §5 · p. 8Weakly supervised matching assumes true matches lie within the object region identified by object-location priors such as a bounding box.
Source: §3.4 · p. 5
Abstract
from arXiv · showhide
We present a descriptor, called fully convolutional self-similarity (FCSS), for dense semantic correspondence. To robustly match points among different instances within the same object class, we formulate FCSS using local self-similarity (LSS) within a fully convolutional network. In contrast to existing CNN-based descriptors, FCSS is inherently insensitive to intra-class appearance variations because of its LSS-based structure, while maintaining the precise localization ability of deep neural networks. The sampling patterns of local structure and the self-similarity measure are jointly learned within the proposed network in an end-to-end and multi-scale manner. As training data for semantic correspondence is rather limited, we propose to leverage object candidate priors provided in existing image datasets and also correspondence consistency between object pairs to enable weakly-supervised learning. Experiments demonstrate that FCSS outperforms conventional handcrafted descriptors and CNN-based descriptors on various benchmarks.
1. Introduction
Dense semantic correspondence must match points across same-class object instances despite substantial appearance variation, while preserving precise localization. FCSS addresses this by learning a multi-scale local self-similarity descriptor within a fully convolutional network.
1. Introduction
- Semantic correspondence supports scene recognition, registration, segmentation, and image editing, but differs from depth or optical-flow matching across visually similar images of the same scene.Its challenge is matching across intra-class variations among object instances.Source: §1 · p. 1
- Within-class differences in colors and gradients, along with occlusion and background clutter, can distract handcrafted-feature matching; optimization enforcing correspondence-map smoothness helps, but remains limited without a proper matching descriptor.Source: §1 · p. 1
- Local self-similarity is preserved across same-class instances despite appearance differences, providing robustness to intra-class variation.The approach is motivated by similar local structural layouts even when colors, gradients, or feature positions differ.Source: Figure 1 · p. 1, §1 · pp. 1–2
- CNN features localize matches precisely, but semantic correspondence needs greater appearance invariance; deeper networks risk losing localization precision, and training data are limited.The passage notes that deeper convolutional networks could increase invariance but reduce precision in matching details.Source: §1 · p. 1
- FCSS learns local self-similarity sampling patterns and measures jointly in a fully convolutional network, end-to-end and at multiple scales.Its weakly supervised learning leverages correspondence consistency between object locations in existing datasets.Source: §1 · p. 2
2. Related Work
Prior approaches to dense semantic correspondence use handcrafted descriptors with optimization or CNN features, while LSS-based methods offer another route but had mainly been handcrafted. The related work also describes limitations in dense CNN descriptors and in a CAD-supervised learning approach.
Feature Descriptors
- CNN-based descriptors have limitations with large intra-class variation and dense matching, while semantic correspondence lacks a dense-ground-truth benchmark for supervised CNN training.Combining existing convolutional networks also leaves a tradeoff between appearance invariance and localization precision.Source: §2 · p. 2
- Earlier LSS descriptors were handcrafted; DASC performs satisfactorily but cannot handle non-rigid deformations because of fixed patch pooling, whereas DSC reformulates LSS in a deep non-CNN architecture.Source: §2 · p. 2
Dense Semantic Correspondence
- Dense semantic correspondence methods use handcrafted features such as SIFT or HOG, often alongside optimization approaches including SIFT Flow.Other examples include exemplar-LDA, joint image set alignment, and cosegmentation.Source: §2 · p. 2
Weakly-Supervised Feature Learning
- Zhou et al. use cycle-consistency with a 3D CAD model to supervise semantic correspondence, but require a suitable model for each object class.Source: §2 · p. 2
3. The FCSS Descriptor
FCSS builds dense descriptors from local self-similarity in a fully convolutional network, jointly learning sampling patterns and similarity measures. It combines multiple scales with weak supervision from image-pair correspondence consistency and object-location priors.
3.1. Problem Formulation and Overview
- Each descriptor component records local patch-pair self-similarity, with exponential gating and a spatial-window maximum for robustness to outliers and sampling-position shifts.The descriptor is defined around each pixel over a local support window.Source: §3.1 · pp. 2–3
3.2. CSS: Convolutional Self-Similarity Layer
- CSS uses CNN activations to measure self-similarity between sampled patches within one image, rather than matching patches across two images.The approach is designed to provide robustness to intra-class variation while preserving localization around fine-grained object boundaries.Source: §3.2 · p. 3
- Differentiable shifting transformers make patch-sampling offsets learnable, allowing sampling patterns and similarity measures to be trained end-to-end.The shifts are network parameters applied to activations to form the two sampled streams.Source: §3.2 · pp. 3–4, §3.1 · p. 3
3.3. Network Configuration for Dense Descriptor
- Hierarchical CSS layers concatenate responses from intermediate CNN activations, encoding self-similarity across receptive-field scales with optimized sampling patterns at each scale.Bilinear upsampling is applied after CSS layers because intermediate activations have lower spatial resolution than the input image.Source: §3.3 · pp. 4–5
3.4. Weakly-Supervised Dense Feature Learning
- Image-pair training uses correspondence consistency to select positive and negative pixel pairs, avoiding the need for ground-truth correspondence maps.Image-wise learning also reduces redundant computation of convolutional activations for adjacent pixels.Source: §3.4 · p. 5
- Object-location priors restrict candidate matches to the object region, under the assumption that true matches lie there, mitigating background-clutter effects during training.The paper also says this restriction expedites feature learning.Source: §3.4 · p. 5
4. Experimental Results and Discussion
Experiments compare FCSS with descriptor and correspondence baselines across four benchmarks; the reported results favor FCSS, with Proposal Flow performance depending on the optimizer.
4.2. Results
- On Taniai, FCSS outperforms handcrafted and CNN descriptors under fixed SF, and leads correspondence techniques quantitatively and qualitatively.Table 1 evaluates descriptors with SF optimization fixed; Table 2 compares correspondence techniques.Source: Table 2 · p. 6, Table 1 · p. 6, §4.2 · p. 7
4.1. Experimental Settings
- The experiments compare descriptors and optimizers across Taniai, Proposal Flow, PASCAL-VOC, and Caltech-101.The implementation uses ImageNet-pretrained VGG features with three CSS layers and 64 sampling patterns per layer; training uses Caltech-101 pairs excluding test pairs. Descriptor comparisons fix SF, while correspondence methods are also tested with PF and DSP.Source: §4.1 · pp. 5–6
4.2. Results
- With SF optimization on Proposal Flow, FCSS is competitive; with PF optimization, it significantly outperforms state-of-the-art descriptors and correspondence techniques.The evaluation uses PCK on keypoints warped by the estimated flow.Source: Table 4 · p. 7, §4.2 · p. 8
- On PASCAL-VOC parts, FCSS yields the highest matching accuracy among the compared methods.Evaluation includes weighted IoU for transferred part segments and PCK for keypoint alignment.Source: Figure 8 · p. 8, Table 5 · p. 8, §4.2 · p. 8
- On Caltech-101, FCSS clearly outperforms the comparison techniques across the reported matching evaluation.The evaluation uses LT-ACC, IoU, and localization error, with mask-transfer visualizations.Source: Figure 9 · p. 8, Table 6 · p. 8, §4.2 · p. 8
5. Conclusion
The conclusion presents FCSS as a fully convolutional formulation of local self-similarity, jointly learned across scales and trained weakly supervised. It identifies potential benefits for instance-level detection and segmentation.
5. Conclusion
- FCSS formulates local self-similarity within a fully convolutional network.The sampling patterns and self-similarity measure are jointly learned end-to-end in a multi-scale manner.Source: §5 · p. 8
- Training uses correspondence consistency between object bounding boxes in image pairs in a weakly supervised manner.Source: §5 · p. 8
- The authors suggest FCSS could potentially benefit instance-level object detection and segmentation through robustness to intra-class variations and precise localization.Source: §5 · p. 8