Source-linked AI summary
Do Convnets Learn Correspondence?
Jonathan Long, Ning Zhang, Trevor Darrell
TL;DR
The paper asks whether convnet features trained with whole-image labels preserve fine-scale correspondence despite large receptive fields. It examines their locality through visualization, intraclass alignment, and keypoint localization, finding that they match or exceed conventional features across these tasks.
Problem
The paper addresses whether large receptive fields and whole-image training eliminate the precise correspondence needed for localizing object parts.
Method
The authors evaluate activations from an ImageNet-trained convnet using feature visualization, MRF-based intraclass alignment, and keypoint localization experiments.
Results
Convnet features are at least as useful as conventional features for local correspondence and outperform SIFT by a large margin in PASCAL VOC 2011 keypoint prediction.
Takeaways & Limitations
Classification-trained convnet features retain local visual information useful for precise correspondence despite their large receptive fields and weak label supervision.
Abstract
from arXiv · showhide
Convolutional neural nets (convnets) trained from massive labeled datasets have substantially improved the state-of-the-art in image classification and object detection. However, visual understanding requires establishing correspondence on a finer level than object category. Given their large pooling regions and training from whole-image labels, it is not clear that convnets derive their success from an accurate correspondence model which could be used for precise localization. In this paper, we study the effectiveness of convnet activation features for tasks requiring correspondence. We present evidence that convnet features localize at a much finer scale than their receptive field sizes, that they can be used to perform intraclass alignment as well as conventional hand-engineered features, and that they outperform conventional features in keypoint prediction on objects from PASCAL VOC 2011.
1 Introduction
The paper asks whether classification-trained convnet features preserve precise part correspondence despite large receptive fields and whole-image supervision. It investigates this question through alignment and localization tasks, positioning convnet features against conventional hand-engineered features.
- Convnet features had already achieved strong image-classification and coarse object-detection results, but their fine-scale spatial specificity remained uncertain.
- The central question is whether large receptive fields pool away precise object-part correspondence, making hand-engineered features preferable.
- The paper provides evidence that convnet features perform at least as well as conventional features for point-to-point correspondence.
- The study connects correspondence to image alignment, a step used in face verification, motion analysis, stereo matching, and object recognition.
- The authors use a publicly available ImageNet-trained network, examining activations from convolutional, pooling, and fully connected layers as features.
2 Feature visualization
This section visually investigates the effective pooling regions of convnet features by reconstructing patches from nearest neighbors and averaging receptive fields. The visualizations indicate that features retain localized information despite large receptive fields, with deeper layers producing more semantic replacements.
- The experiments provide a visual investigation of convnet features’ effective pooling regions.
- Feature reconstruction replaces image patches with averages of their top-k nearest neighbors in convnet feature space.
- Nearest-neighbor patches are matched using cosine similarity from databases containing at least one million patches per layer.
- Despite large receptive fields, reconstructed images preserve localized features such as bicycle tires and cat facial features at corresponding locations.
- As layer depth increases, replacements become more semantic and less visually specific, with greater diversity in substituted eyes, noses, and fur.
- Similar feature vectors tend to correspond to similar colors near receptive-field centers, although stride 16 limits observable alignment quality.
3 Intraclass alignment
The paper tests whether convnet features can align different instances of the same class for point-to-point correspondence. Convnet flow matches SIFT flow on average and performs somewhat better under stricter keypoint-transfer tolerances.
- 3 Intraclass alignment: The alignment pipeline retrieves near neighbors with fc7 features, then computes dense correspondence using convnet or conventional features at shared locations.The same nearest neighbors and feature locations are used for both methods to isolate alignment quality.
- 3 Intraclass alignment: An MRF assigns each source feature location a displacement to the target while combining feature matching with spatial regularization.Optimization uses belief propagation on a 4-neighborhood graph, with a rotationally invariant squared-Euclidean regularizer.
- 3 Intraclass alignment: Keypoint-transfer accuracy is measured with mean PCK after warping 25 neighbors and taking coordinate-wise medians from the five lowest-deformation alignments.Correctness uses a tolerance proportional to the maximum bounding-box dimension, and invisible target keypoints are not penalized.
- 3 Intraclass alignment: Convnet flow performs at least as well as SIFT flow for alignment and is slightly better at stricter tolerances.Table 2 reports means at α = 0.1, 0.05, and 0.025; the figure examples show category-dependent wins for both methods.
4 Keypoint classification
The paper evaluates whether classification-trained convnet features encode semantic information at object-part scale. They perform at least as well as, and often better than, SIFT for keypoint classification, while responses can localize keypoints more finely than the features’ strides.
- 4 Keypoint classification: Keypoint classification asks whether a classifier can label a keypoint given an image and its coordinates.The study uses keypoint data from the twenty PASCAL VOC 2011 categories.
- 4 Keypoint classification: The evaluation trains one-vs-all linear SVMs using SIFT descriptors and activations from five convolutional layers.Convnet features are sampled from the layer column whose receptive-field center is closest to each keypoint; SIFT is tested at five radii.
- 4 Keypoint classification: Convnet layers consistently perform at least as well as and often better than SIFT, with the highest performance from conv4 and conv5.These features were trained only for image classification, not specifically for keypoint prediction.
- 4 Keypoint classification: Convnet classifier responses can localize keypoints more finely than their strides and receptive-field sizes, whereas SIFT responses are often less sensitive to precise position.The analysis histograms maximum responses measured with a single-pixel stride inside a 21 by 21 pixel window around ground-truth keypoints.
5 Keypoint prediction
The paper predicts object keypoints with sliding-window detectors over convnet features and compares them with SIFT using PASCAL VOC 2011 bounding boxes and PCK evaluation.
- Keypoint prediction uses independently trained sliding-window part detectors to predict each keypoint location.
- Each bounding box is rescaled to 500 × 500, and local 3 × 3 conv5 regions provide 2304-dimensional descriptors for detection.
- The final keypoint score combines a local SVM detector with a nearest-neighbor spherical Gaussian prior using tradeoff parameter η = 0.1.
- PCK with α = 0.1 counts a prediction as correct when its distance from ground truth is below 0.1 times the bounding-box maximum dimension.
- Conv5 local detectors outperform SIFT by a large margin, and the prior improves both feature types on this keypoint task.The authors report these as the first keypoint prediction results on this dataset; conv5 often produces satisfactory outputs despite the task’s difficulty.
6 Conclusion
The paper studies whether intermediate features from a state-of-the-art convnet classifier capture specific local correspondence. Across visualization, alignment, and keypoint prediction, convnet features are at least as useful as conventional features and sometimes considerably more useful.
- The study evaluates local correspondence through visualization, alignment, and keypoint prediction.
- Despite large receptive fields and weak-label training, convnet features are at least as useful as conventional features for extracting local visual information.The conclusion states that convnet features are sometimes considerably more useful.