Source-linked AI summary
Learning to Compare Image Patches via Convolutional Neural Networks
Sergey Zagoruyko, Nikos Komodakis
TL;DR
Comparing image patches is a fundamental computer-vision subroutine, motivating the question of whether datasets of patch correspondences can support automatically learned similarity functions. The paper learns such a function directly from raw image patches with CNNs, explores task-specific architectures, and reports significant state-of-the-art improvements across several problems and benchmark datasets.
Problem
Comparing image patches supports many computer-vision tasks, but the paper asks whether patch-correspondence datasets can be used to automatically learn a similarity function.
Method
The paper learns a general patch-similarity function directly from annotated raw image-patch pairs using CNN models, exploring architectures with different trade-offs.
Results
The approach significantly outperforms the state-of-the-art on several problems and benchmark datasets, while 2-channel architectures are strongest overall.
Takeaways & Limitations
The learned CNN-based similarity function can account for varied image transformations and produce efficient dense descriptors with better performance than manually designed and other learned descriptors.
Takeaways & Limitations
The experiments used a training set considered rather small by today’s standards, so a larger training set could further improve performance.
Abstract
from arXiv · showhide
In this paper we show how to learn directly from image data (i.e., without resorting to manually-designed features) a general similarity function for comparing image patches, which is a task of fundamental importance for many computer vision problems. To encode such a function, we opt for a CNN-based model that is trained to account for a wide variety of changes in image appearance. To that end, we explore and study multiple neural network architectures, which are specifically adapted to this task. We show that such an approach can significantly outperform the state-of-the-art on several problems and benchmark datasets.
1. Introduction
The paper addresses the challenge of comparing image patches under diverse appearance changes by learning a general similarity function directly from annotated raw patch pairs. It explores CNN architectures and reports improved performance over handcrafted and learned alternatives across multiple benchmarks.
- Patch comparison supports vision tasks ranging from structure from motion and panorama construction to recognition, retrieval, and classification.
- Viewpoint, illumination, occlusion, shading, and camera changes make deciding whether two patches correspond challenging.
- The paper learns a patch similarity function from matching and non-matching raw image-patch pairs without manually designed features.
- The study explores CNN architectures with different trade-offs and trains them using large databases of matching and non-matching patch pairs.Adding more automatically generated samples can further enrich the training database.
- Across several problems and benchmark datasets, the approach significantly outperforms state-of-the-art, handcrafted, and other learned descriptors.The resulting convolutional descriptors are also efficient to compute densely.
2. Related work
Prior patch matching typically relies on handcrafted descriptors or separately learned descriptors, while CNN-based approaches had narrower demonstrated scope. This paper positions CNN learning as a route to broader appearance-change handling.
- Conventional patch comparison commonly uses descriptors with squared Euclidean distance, including handcrafted SIFT and DAISY features.
- Earlier learning methods included learned descriptors and a convex procedure trained for both patch-comparison tasks.
- ImageNet-trained convolutional descriptors were reported to outperform SIFT in most tested cases except blur.
- A prior CNN patch-comparison method targeted very small patches in narrow-baseline stereo, whereas this work seeks broader appearance-change coverage and wider applications.
3. Architectures
The paper studies several ways to process patch pairs, balancing flexibility, accuracy, and test-time efficiency. Additional designs address multi-resolution context and arbitrary patch sizes.
- Basic models: The basic models are 2-channel, Siamese, and pseudo-Siamese architectures with different speed–accuracy trade-offs.
- Siamese: Siamese branches share architecture and weights, compute patch representations independently, and feed concatenated outputs to a top similarity network.
- Pseudo-Siamese: Pseudo-Siamese networks uncouple branch weights, increasing flexibility over Siamese models while retaining their test-time efficiency.
- 2-channel: The 2-channel model feeds both patches jointly as a two-channel image into convolutional layers and a one-output fully connected decision layer.
- Additional models: Deep networks replace larger convolutional kernels with 3x3 kernels separated by ReLUs to increase nonlinearities and potentially improve discrimination.
- Additional models: Central-surround two-stream networks process central high-resolution and surround low-resolution inputs, emphasizing multi-resolution information and patch centers.
- Additional models: SPP layers aggregate convolutional features so fully connected inputs retain fixed dimensionality for arbitrary patch sizes.
4. Learning
All models are trained with supervised matching labels using a hinge-based objective with squared l2 regularization. Data augmentation and GPU implementation support training and efficient descriptor computation.
- Training uses matching and non-matching patch labels in a strongly supervised manner.
- The learning objective combines a hinge-based loss term with squared l2-norm regularization on network weights.
- Labels are y_i ∈ {−1, 1}, with −1 denoting a non-matching pair and 1 denoting a matching pair.
- Models are trained from scratch with ASGD, momentum 0.9, learning rate 1.0, weight decay λ = 0.0005, and mini-batches of 128.
- Augmentation flips patch pairs horizontally and vertically and rotates them by 90, 180, and 270 degrees to combat overfitting.
- GPU storage and on-the-fly augmentation enable efficient pair retrieval, while Siamese descriptors run two times faster than ImageNet descriptors on GPU.
5. Experiments
The experiments evaluate multiple CNN architectures for patch comparison across local matching, wide-baseline stereo, and descriptor settings. Results favor architectures that jointly process patch pairs and incorporate multi-resolution information, outperforming hand-designed and prior learned features.
- 5.1. Local image patches benchmark: 2ch-based architectures achieved the best performance on the local image patches benchmark, indicating the value of jointly processing both patches from the first layer.The authors report this conclusion from Table 1 and explicitly connect it to early information sharing between patches.
- 5.1. Local image patches benchmark: 2ch-2stream was the top-performing network, achieving a 2.45-times better score than and a 6.65-times better score than SIFT on mean(1,4).2ch-deep followed closely, supporting the reported importance of multi-resolution information and increased depth.
- 5.1. Local image patches benchmark: Siamese models also outperformed existing state-of-the-art systems; siam-2stream performed best among them, while pseudo-siam outperformed siam.These models used standard max-pooling rather than learning pooling-region shape, size, or placement.
- 5.1. Local image patches benchmark: ImageNet-trained conv4 features reached an FPR95 of 17.98, outperforming SIFT but remaining much worse than the proposed models.The comparison uses l2-normalized ImageNet-trained CNN features.
- 5.1. Local image patches benchmark: The learned 2-channel filters often formed opposite left and right parts, suggesting that the network learned to compute feature differences between patches.The authors note that this pattern does not occur in every first-layer filter.
- 5.2. Wide baseline stereo evaluation: Neural-network photometric costs produced more robust depth maps than hand-crafted features, while sparse siamese errors could be easily eliminated during global optimization.The reported qualitative results also show more fine details without global optimization for the neural models.
- 5.2. Wide baseline stereo evaluation: Across all error thresholds and baseline distances, siamese models performed much better than DAISY in the wide-baseline stereo evaluation.The plots include six stereo pairs, increasing baselines, and separate distributions for unoccluded pixels.
- 5.3. Local descriptors performance evaluation: On the descriptor evaluation, siam-2stream-l2 significantly outperformed SIFT and matched ImageNet-trained features using a 512-dimensional representation.This model was not trained with l2 distances.
6. Conclusions
The paper concludes that CNNs can learn general patch similarity directly from raw pixels and substantially outperform existing methods across multiple tasks and benchmark datasets. Among the tested designs, 2-channel networks were strongest overall, while multi-resolution and SPP enhancements improved siamese models.
- CNNs learned a general patch-similarity function directly from raw pixels and significantly outperformed the state-of-the-art across several problems and benchmark datasets.The approach avoids manually designed features and studies architectures adapted specifically to patch comparison.
- 2-channel architectures were clearly superior to the other tested architectures in overall results.
- Multi-resolution 2-stream siamese models consistently delivered a significant performance boost, underscoring the value of multi-resolution information for patch comparison.
- SPP-based siamese networks consistently improved result quality, with further gains potentially available because multiple-aspect-ratio patches were absent during training.
- The overall approach could benefit further from a larger training set, since the experiments used a dataset considered small by current standards.