PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-End Kernel Learning with Supervised Convolutional Kernel Networks

Julien Mairal

arXiv:1605.06265v2stat.MLcs.CVcs.LG

TL;DR

Traditional kernel methods decouple data representation from the prediction task, limiting their fit with end-to-end task-adapted representations. The paper learns discriminative RKHS subspaces in multilayer convolutional kernel networks, and reports competitive classification and super-resolution outcomes.

  • Problem

    Traditional kernel methods decouple data representation from learning, seeming incompatible with end-to-end representations adapted to the task in image modeling.

    Source: §1 · pp. 1–2

  • Method

    The method jointly optimizes discriminative linear subspaces in RKHSs across convolutional kernel network layers using labeled data.

    Source: §1 · p. 2

  • Results

    The method achieves about 2% test error on SVHN and about 10% on CIFAR-10, and outperforms recent CNN approaches for image up-scaling.

    Source: §1 · p. 2, §4.1 · p. 7, §4.2 · p. 8

  • Takeaways & Limitations

    The reported classification and super-resolution results demonstrate the approach across multiple image-related tasks.

    Source: §1 · p. 2

    Without GPUs, the authors could not exhaustively explore hyperparameters or evaluate large networks, and they did not investigate dropout, batch normalization, or methods for training very deep networks.

    Source: §1 · p. 2

Abstract

from arXiv · show

In this paper, we introduce a new image representation based on a multilayer kernel machine. Unlike traditional kernel methods where data representation is decoupled from the prediction task, we learn how to shape the kernel with supervision. We proceed by first proposing improvements of the recently-introduced convolutional kernel networks (CKNs) in the context of unsupervised learning; then, we derive backpropagation rules to take advantage of labeled training data. The resulting model is a new type of convolutional neural network, where optimizing the filters at each layer is equivalent to learning a linear subspace in a reproducing kernel Hilbert space (RKHS). We show that our method achieves reasonably competitive performance for image classification on some standard "deep learning" datasets such as CIFAR-10 and SVHN, and also for image super-resolution, demonstrating the applicability of our approach to a large variety of image-related tasks.

1 Introduction

The paper addresses the apparent incompatibility between kernel methods and end-to-end, task-adapted image representations. It develops supervised convolutional kernel networks and reports competitive image-classification and super-resolution results, while noting limits on the experiments conducted.

1 Introduction

  • Traditional kernel methods decouple data representation from learning, seeming incompatible with end-to-end representations adapted to the task.The paper targets this issue in image modeling.Source: §1 · pp. 1–2
  • The approach jointly learns discriminative subspaces in RKHSs across layers, yielding a convolutional neural network with a non-standard parametrization.The subspaces can also be learned without supervision using classical kernel approximation techniques.Source: §1 · p. 2
  • The model obtains about 2% test error on SVHN and about 10% on CIFAR-10, and outperforms recent CNN approaches for image up-scaling.The classification results are reported without model averaging or data augmentation.Source: §1 · p. 2
  • The experiments omit GPU training, limiting exhaustive hyperparameter exploration and evaluation of large networks.The authors also leave dropout, batch normalization, and techniques for training very deep networks unexplored [11] [10] [23].Source: §1 · p. 2
  • Earlier image CKNs and hierarchical kernel descriptors produced unsupervised representations [18], while multiple kernel learning selects combinations from a predefined collection.The paper also distinguishes its approach from supervised Fisher-kernel parameter learning and notes concurrent deep Gaussian-process work [6].Source: §1 · p. 2

2 Learning Hierarchies of Subspaces with Convolutional Kernel Networks

Convolutional kernel networks represent image patches in RKHSs and approximate those representations by projecting onto finite-dimensional subspaces. Pooling and kernel composition build multilevel image maps, while supervision can optimize the subspaces by back-propagation.

2 Learning Hierarchies of Subspaces with Convolutional Kernel Networks

  • A kernel maps each local patch to an RKHS, where kernel values equal inner products and provide nonlinear patch representations.The kernel must be positive definite; examples include arc-cosine and Gaussian kernels.Source: §2 · p. 3
  • CKNs approximate RKHS inner products with finite-dimensional feature maps, and a Nyström-style projection keeps approximated points within the RKHS.Spherical K-means selects unit-norm centroids whose embedded points span the projection subspace.Source: §2 · p. 3
  • The resulting feature map can be implemented as convolution, contrast normalization, a pointwise nonlinearity, a linear transform, and norm rescaling.This produces a convolutional neural network with a non-standard parametrization.Source: §2 · p. 3
  • The illustrated layer projects patch embeddings onto a finite-dimensional subspace, optimized by projection residuals without supervision or by back-propagation with supervision.Higher layers follow by stacking this model and shifting indices.Source: Figure 1 · p. 4
  • Pooling linearly combines nearby feature vectors with Gaussian weights to downsample the map and gain invariance to small shifts.The Gaussian weights act as an anti-aliasing filter, and the subsampling factor need not be an integer.Source: §2 · p. 4
  • Stacking these operations produces maps whose vectors represent increasingly large image neighborhoods with greater pooling-based invariance.The RKHS embedding allows subsequent layers to take previous-layer RKHS points as inputs and apply kernels to larger neighborhoods.Source: §2 · p. 4

3 End-to-End Kernel Learning with Supervised CKNs

The section formulates supervised CKN training as joint learning of RKHS subspaces and prediction weights, derives filter-gradient backpropagation, and presents optimization heuristics.

3 End-to-End Kernel Learning with Supervised CKNs

  • Supervised CKNs jointly optimize the prediction weights and layer filters, whose columns are constrained to the Euclidean sphere.The filter optimization is non-convex, while the objective is differentiable.Source: §3.1 · p. 5

3.1 Backpropagation Rules for Convolutional Kernel Networks

  • Filter gradients use chain-rule backpropagation through the feature maps, with a forward pass followed by a backward pass.The backward pass composes layer functions; the forward step is dominated by convolutions, and backward-pass cost is the same up to a constant factor.Source: §3.1 · pp. 5–6
  • For RBF kernels, the authors optimize kernel hyperparameters by gradient steps when training data are plentiful and overfitting is not an issue; otherwise, they leave other strategies for future work.The passage says more involved techniques may be needed in the latter setting.Source: §3.1 · p. 6

3.2 Optimization and Practical Heuristics

  • Alternating optimization fits the convex prediction-weight problem with accelerated MISO for fixed filters, then updates filters with projected stochastic gradients.The network filters are initialized using the unsupervised learning method described earlier.Source: §3.2 · p. 6
  • Sphere preconditioning changes variables using each layer’s inverse patch covariance to reduce gradient-entry correlation during constrained filter updates.The authors describe this as an improvement to SGD, not a critical component.Source: §3.2 · p. 7
  • The learning-rate heuristic halves the rate and restores the previous network parameters when training loss rises; the active-set heuristic drops zero-gradient samples and reactivates 10% each epoch.For squared hinge loss, the gradient vanishes when the margin exceeds one.Source: §3.2 · p. 7

4 Experiments

The experiments evaluate supervised CKNs on image classification and super-resolution. They report competitive classification errors and super-resolution results that outperform cited CNN baselines, including for a non-trained x3 procedure.

4.1 Image Classification on “Deep Learning” Benchmarks

  • The classification evaluation uses CIFAR-10 and SVHN, both 32 × 32 images from 10 classes, with 50,000/10,000 and 604,388/26,032 train/test examples, respectively, on a nine-layer network learning 512 filters per layer.CIFAR-10 parameters αj are kept fixed during training, whereas SVHN αj are updated like the filters.Source: §4.1 · p. 7

4 Experiments

  • About 2% test error on SVHN and about 10% on CIFAR-10; for x3 upscaling, the method outperforms end-to-end-trained baselines despite using a model not trained for x3.The x3 procedure reuses the x2 model twice, then downsamples by 3/4.Source: §4.1 · p. 7, Table 1 · p. 8, Table 2 · p. 8

4.2 Image Super-Resolution from a Single Image

  • The super-resolution task predicts high-resolution luminance from bicubic-upscaled 16 × 16 patches derived from 32 × 32 BSD500 patches.The training set contains 200,000 patches; image 302003.jpg is removed because it overlaps with a test image.Source: §4.2 · p. 8
  • For super-resolution, the model is a nine-layer network with 128 filters per layer; a linear model predicts pixel values from the last-layer features.The pipeline removes a local mean using a 5 × 5 averaging filter and adds it back after upscaling.Source: §4.2 · p. 8

A Orthogonal Projection on the Finite-Dimensional Subspace F1

The section derives a finite-dimensional RKHS projection for image patches, using normalized patches and kernel evaluations. A small regularization stabilizes the projection when its kernel matrix is singular or poorly conditioned.

A Orthogonal Projection on the Finite-Dimensional Subspace F1

  • Unit-norm patches are projected onto F1 before inverse rescaling, using the homogeneity of K1.This normalization is stated to hold without loss of generality.Source: A Orthogonal Projection on the… · p. 10
  • The section defines the orthogonal projection of a unit-norm patch onto the finite-dimensional subspace F1.Source: A Orthogonal Projection on the… · p. 10
  • The projection coefficients α⋆ are computed from kernel evaluations using κ1(Z⊤Z)−1κ1(Z⊤x), assuming κ1(Z⊤Z) is invertible.The passage applies this parameterization to normalized patches.Source: A Orthogonal Projection on the… · p. 10
  • When κ1(Z⊤Z) is noninvertible or poorly conditioned, ε-regularization improves its condition number.The modification is interpreted as a slightly regularized projection onto F1.Source: A Orthogonal Projection on the… · p. 10

B Computation of the Gradient with Respect to the Filters

The gradient derivation linearizes the network's quantities under filter perturbations and propagates those changes layer by layer. It obtains the required gradient relations by induction and elementary matrix-calculus rules.

B Computation of the Gradient with Respect to the Filters

  • The derivation analyzes perturbations of the quantities in equation (11) to obtain the gradient relations in equations (8) and (9).It defines Z+E and uses o(∥E∥) for terms negligible relative to ∥E∥.Source: B Computation of the Gradient with… · p. 10
  • The perturbation analysis proceeds recursively from the unaffected initial feature map through the later layers.The induction step tracks perturbed layer quantities and establishes changes of order O(∥E∥).Source: B Computation of the Gradient with… · pp. 10–11
  • Elementary matrix-calculus rules yield the linear functions gj and hj, and the derivation concludes by induction.The j-th feature map is identified as the map before the j-th linear pooling step.Source: B Computation of the Gradient with… · pp. 11–12

C Preconditioning Heuristic on the Sphere

The section motivates the sphere update by viewing preconditioning as a change of variables, then adapting gradient descent to the resulting manifold geometry. The paper uses the resulting update as a heuristic in a stochastic setting.

C Preconditioning Heuristic on the Sphere

  • On the sphere, manifold gradient descent is equivalent to projected gradient descent in Rp with a particular step size.The descent direction is the gradient projected onto the tangent space.Source: C Preconditioning Heuristic on the… · p. 12
  • In unconstrained optimization, preconditioning corresponds to a change of variables; a better-conditioned transformed Hessian is hoped to yield faster convergence.The transformed objective uses z = Q1/2w.Source: C Preconditioning Heuristic on the… · p. 12
  • With the sphere constraint, the change of variables maps optimization to an ellipsoid, whose tangent-space projection informs the preconditioned update.The paper adopts this update as a heuristic in a stochastic setting.Source: C Preconditioning Heuristic on the… · p. 13

D Additional Results for Image Super-Resolution

Visual super-resolution comparisons show that the supervised kernel approach matches the CNN baseline in textured regions while recovering sharper fine details and avoiding boundary ghosting. The comparison setup applies upscaling to luminance, and Table 3 reports SSIM for reconstruction accuracy.

D Additional Results for Image Super-Resolution

  • Table 3 compares reconstruction accuracy using SSIM, for which higher values indicate better results.The supplied passages do not state the table's cell values.Source: Table 3 · p. 13
  • The approach matches the classical CNN in textured areas, recovers thin high-frequency details, and avoids the boundary ghosting observed with that method.Figure 2 also reports sharper, artefact-free edges for SCKN, particularly in the butterfly image.Source: D Additional Results for Image… · p. 13, Figure 2 · p. 14
  • RGB images are converted to YCbCr, with upscaling applied only to luminance and bicubic interpolation used for color channels.Source: Figure 2 · p. 14
Loading 1605.06265v2…