Source-linked AI summary
Convolutional Kernel Networks
Julien Mairal, Piotr Koniusz, Zaid Harchaoui, Cordelia Schmid
TL;DR
Visual recognition needs representations invariant to particular transformations, but CNN invariance is not precisely understood. The paper introduces CKNs, unsupervised convolutional networks trained to approximate reproducing-kernel maps, and reports competitive performance across MNIST, CIFAR-10, and STL-10 with simple architectures and no data augmentation.
Problem
Visual recognition requires representations invariant to particular transformations, while the exact nature of CNN invariance is not precisely understood.
Method
CKNs use an approximation scheme to make reproducing-kernel methods computationally feasible and train an unsupervised convolutional network to approximate the kernel map.
Results
CKNs achieve competitive or near-state-of-the-art performance on MNIST, CIFAR-10, and STL-10 with simple architectures and no data augmentation.
Takeaways & Limitations
The methodology connects kernel methods and convolutional neural networks while providing simple architectures with few parameters to learn.
Takeaways & Limitations
The paper assumes finite-dimensional input maps whose initial feature maps extract patches from the input data.
Abstract
from arXiv · showhide
An important goal in visual recognition is to devise image representations that are invariant to particular transformations. In this paper, we address this goal with a new type of convolutional neural network (CNN) whose invariance is encoded by a reproducing kernel. Unlike traditional approaches where neural networks are learned either to represent data or for solving a classification task, our network learns to approximate the kernel feature map on training data. Such an approach enjoys several benefits over classical ones. First, by teaching CNNs to be invariant, we obtain simple network architectures that achieve a similar accuracy to more complex ones, while being easy to train and robust to overfitting. Second, we bridge a gap between the neural network literature and kernels, which are natural tools to model invariance. We evaluate our methodology on visual recognition tasks where CNNs have proven to perform well, e.g., digit recognition with the MNIST dataset, and the more challenging CIFAR-10 and STL-10 datasets, where our accuracy is competitive with the state of the art.
1 Introduction
The paper proposes CKNs, which use reproducing kernels to encode invariance and approximate kernel maps with unsupervised convolutional networks. This connects kernel methods and neural networks while achieving competitive visual-recognition results with simple architectures.
- CNN invariance is empirically observed but its exact nature is not precisely understood.
- CKNs use reproducing kernels to produce multilayer image representations with explicitly modeled invariance.
- The CKN approximation scheme makes kernel methods computationally feasible and trains an unsupervised CNN to approximate the kernel map.
- CKNs bridge kernel methods and neural networks while using nonlinearities that naturally emerge from approximating the Gaussian kernel map.
- CKNs achieve competitive results on MNIST, CIFAR-10, and STL-10 with simple architectures, few learned parameters, and no data augmentation.
2 Convolutional Multilayer Kernels
The multilayer convolutional kernel builds hierarchical image feature maps from local patches, with successive layers adding larger-pattern information and local shift invariance. Its kernel compares normalized features across spatial locations, while concrete inputs include gradients and image patches.
- A feature map assigns each image coordinate a representation in a Hilbert space, and multilayer kernels transform these maps into increasingly invariant representations.
- The kernel sums pairwise comparisons across spatial locations, weighting comparisons by coordinate proximity and normalized feature similarity.
- The parameter β controls positional invariance: larger β makes the kernel more invariant to the positions of compared features.
- Gradient feature maps encode orientation after normalization, yielding the kernel descriptor introduced for natural image patches.
- Patch feature maps encode contrast-normalized m × m image patches, or m × m × 3 patches for three-channel images.
- Each layer represents a local patch from the preceding feature map, combining information from several locations into potentially larger patterns.
3 Training Invariant Convolutional Kernel Networks
CKNs make multilayer kernel representations computationally feasible by learning finite-dimensional approximations with convolutional filters, nonlinearities, and pooling. Their training approximates the kernel map on data, while retaining CNN-like architectures and feedforward parameter learning.
- Fast Approximation of the Gaussian Kernel: Kernel approximation is critical because computing full multilayer kernel matrices is infeasible even for moderately sized image databases.The paper motivates approximation methods for databases of approximately 10,000 images and moderate numbers of layers.
- Approximating the Multilayer Convolutional Kernel: CKNs approximate multilayer kernel maps with spatial convolutions, learned filters, pointwise nonlinearities, and pooling operations on regular grids.The resulting finite-dimensional maps are constructed recursively across layers.
- Approximating the Multilayer Convolutional Kernel: Each layer combines information from multiple lower-layer points, can represent larger patterns, and adds spatial locality and invariance.Patches are mapped into finite-dimensional feature vectors, with pooling producing the subsequent feature map.
- Approximating the Multilayer Convolutional Kernel: CKNs use CNN-like hyperparameters and differ mainly in the objective optimized for filters and the choice of nonlinearities.The method learns layer parameters in a feedforward manner, including filters, weights, and smoothing parameters.
- Fast Approximation of the Gaussian Kernel: The Gaussian kernel is approximated by learned linear filters followed by pointwise nonlinearities, whose shape resembles a rectified linear unit.For higher-dimensional data, the approximation is learned on training pairs using importance weights and sampling points.
- Approximating the Multilayer Convolutional Kernel: Training uses an unsupervised kernel-approximation objective, with L-BFGS-B applied to 300 000 randomly selected training pairs and K-means initialization.The authors ran L-BFGS-B for 4 000 iterations and leave speed improvements for future work.
4 Experiments
Experiments show that CKNs learn interpretable filters and achieve strong recognition performance with compact architectures and no data augmentation. Evaluations cover natural image patches, MNIST, CIFAR-10, and STL-10.
- 4.1 Discovering the Structure of Natural Image Patches: 197 of 256 learned filters exhibit interpretable Gabor-like structures on whitened natural image patches.The filters were learned from 300 000 patches of size 12 × 12.
- 4.2 Digit Classification on MNIST: CKN-GM1 achieves its MNIST result with only 50 filters in the second layer, while CKN-GM2 uses 400 filters and performs best overall.The paper reports state-of-the-art accuracy for this task, with lower error rates reported only using data augmentation.
- 4.3 Visual Recognition on CIFAR-10 and STL-10: CIFAR-10 and STL-10 experiments select architectures using validation data or cross-validation, with the best models consistently using two layers and 800 top-layer filters.CKN-GM and CKN-PM use different information, while CKN-CO concatenates their normalized representations.
- 4.3 Visual Recognition on CIFAR-10 and STL-10: CKN performance is competitive with the state of the art on CIFAR-10 and STL-10, especially STL-10, despite using two layers, few learned parameters, and no data augmentation.The paper notes that stronger reported results use data augmentation or external data.
5 Conclusion
The paper combines kernels and CNNs through a methodology that achieves near-state-of-the-art performance on several datasets with simple architectures and no data augmentation. It identifies supervised kernel approximation and theoretical analysis of learned feature spaces as open challenges.
- The proposed methodology combines kernels and convolutional neural networks for visual recognition.
- Near-state-of-the-art performance is achieved on MNIST, CIFAR-10, and STL-10 with simple architectures and no data augmentation.
- Open challenges include using supervision to better approximate the prediction-task kernel and analyzing the theoretical properties of produced feature spaces.
A Positive Definiteness of K
The kernel K is shown to be positive definite by decomposing it into combinations and products of known positive-definite kernels. Normalization preserves positive definiteness in the Gaussian-kernel composition.
- A nonnegative linear combination of positive-definite kernels is positive definite, reducing the proof to a component kernel.
- The component kernel is positive definite because it is the pointwise product of two positive-definite kernels.
- The second factor remains positive definite through composition of the Gaussian kernel with normalized linear-kernel feature maps.The convention φ/∥φ∥H = 0 is used when φ = 0.
B List of Architectures Reported in the Experiments
The paper reports its network architectures in a dedicated table, specifying layer counts, filter counts, patch sizes, subsampling, output-map size, and parameter counts.
- Table 3 lists N as the number of layers and p1 and p2 as the numbers of filters at the respective layers.
- The architecture descriptions specify m1 and m2 as patch sizes P′1 and P′2 on feature maps ζ1 and ζ2.
- The table also records γ1 as the subsampling factor, S as output feature-map size, and the number of learned parameters.