Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks

Alexey Dosovitskiy; Philipp Fischer; Jost Tobias Springenberg; Martin Riedmiller; Thomas Brox

Source-linked AI summary

Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks

Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springenberg, Martin Riedmiller, Thomas Brox

arXiv:1406.6909v2Jun 26, 2014cs.LGcs.CVcs.NE

TL;DR

The paper tackles generic feature learning without large labeled datasets, particularly for tasks that differ from object classification. It trains a CNN to discriminate automatically generated surrogate classes from transformed unlabeled patches, achieving strong unsupervised classification and geometric-matching results.

Problem
Generic feature learning is needed because supervised CNNs require large labeled datasets, while class-specific annotations may not benefit class-independent tasks such as descriptor matching.
Method
The method trains a CNN on unlabeled images by forming surrogate classes from randomly transformed versions of seed patches, with transformations specifying desired invariances.
Results
The learned representation outperforms previous unsupervised methods on four classification benchmarks and outperforms AlexNet features and SIFT for descriptor matching.
Takeaways & Limitations
The approach provides generic discriminative features for object classification and geometric matching, and its transformation set can be adapted to the target application.
Takeaways & Limitations
Performance can decrease when too many similar samples are used, making the number and selection of surrogate classes relevant parameters.

Abstract

from arXiv · show

Deep convolutional networks have proven to be very successful in learning task specific features that allow for unprecedented performance on various computer vision tasks. Training of such networks follows mostly the supervised learning paradigm, where sufficiently many input-output pairs are required for training. Acquisition of large training sets is one of the key challenges, when approaching a new task. In this paper, we aim for generic feature learning and present an approach for training a convolutional network using only unlabeled data. To this end, we train the network to discriminate between a set of surrogate classes. Each surrogate class is formed by applying a variety of transformations to a randomly sampled 'seed' image patch. In contrast to supervised network training, the resulting feature representation is not class specific. It rather provides robustness to the transformations that have been applied during training. This generic feature representation allows for classification results that outperform the state of the art for unsupervised learning on several popular datasets (STL-10, CIFAR-10, Caltech-101, Caltech-256). While such generic features cannot compete with class specific features from supervised training on a classification task, we show that they are advantageous on geometric matching problems, where they also outperform the SIFT descriptor.

1 INTRODUCTION

The paper addresses the need for generic visual features learned without large labeled datasets, especially for tasks unlike object classification. It proposes discriminative training on automatically generated surrogate classes and evaluates the resulting representation across vision tasks.

Supervised CNN training requires huge labeled datasets, which are difficult to collect and offer diminishing returns as they grow.
Object-class annotations may be less beneficial for class-independent tasks such as descriptor matching.
The method trains a CNN without labeled data using a surrogate task automatically generated from unlabeled images.
Each surrogate class contains one seed image and versions transformed to simulate variation, producing generic features robust to those transformations.
The resulting representation is discriminative and transformation-invariant, and supports both object classification and descriptor matching.
Classification accuracy exceeds previous unsupervised methods on STL-10, CIFAR-10, Caltech-101, and Caltech-256, while descriptor matching outperforms AlexNet features and SIFT.

2 CREATING SURROGATE TRAINING DATA

The training data are built from unlabeled image patches and their randomly transformed versions. These transformations define the surrogate classes and encode the invariances the CNN is trained to learn.

The procedure samples N 32 × 32 patches from unlabeled images, favoring regions with considerable gradients.
Each sampled patch serves as a seed whose transformed versions form surrogate training data for the CNN.
Transformations are composed from parameterized elementary operations, with each transformation Tα determined by a parameter vector α.
For object classification, the transformation set includes translation, scaling, rotation, contrast changes, and color changes.
The transformation list can be extended for other applications; descriptor matching benefits from adding blur.
After generating transformed patches, the method subtracts the mean of each pixel over the resulting dataset and applies no other preprocessing.

3 LEARNING ALGORITHM

The learning algorithm trains a CNN to discriminate among surrogate classes formed from transformed versions of image patches, producing representations that distinguish samples while encouraging transformation invariance. Its objective can be decomposed into classification of average transformed representations and a regularizer that penalizes variation across transformations, subject to transformations preserving image identity.

Surrogate-class training: Each transformed-patch set is assigned a surrogate label, and a CNN with softmax output is trained using multinomial negative log likelihood.The loss evaluates transformed samples against their surrogate class labels.
Objective decomposition: The objective separates into multinomial logistic regression on average transformed representations and an additional term enforcing consistency across transformations.The first term classifies average feature representations, while the second term penalizes variation in the network outputs.
Objective decomposition: Perfect transformation invariance makes the regularizer attain its global minimum because transformed representations produce identical last-layer activations.Under invariance, h(Tαxi) equals the weighted average representation used in the bound.
Representation properties: The representation is designed to satisfy both recognition requirements: similarity across same-category images and separability across different-category images.The discriminative surrogate task addresses separability without directly modeling or reconstructing the input distribution.
Representation properties: The approach does not require input reconstruction, leaving more freedom to model variability relevant to recognition or matching tasks.Invariance is encouraged by requiring transformed patches to remain predictive of their surrogate labels.
Assumption and limitation: The method assumes transformations preserve image identity; color transformations can impair features on tasks that rely on color distinctions.The paper gives distinguishing black panthers from pumas as an example of this boundary.

4 EXPERIMENTS: CLASSIFICATION

Experiments evaluate Exemplar-CNN classification across datasets and training configurations, showing strong unsupervised transfer performance and sensitivity to surrogate-data design. Classification benefits from sufficient surrogate classes and samples, while transformation effects depend on the dataset and task.

Classification results: 74.2% accuracy on STL-10 substantially improves over previously reported unsupervised results.The larger Exemplar-CNN networks outperform the best prior result on all four evaluated datasets.
Classification results: The larger networks outperform the best prior unsupervised result on STL-10, CIFAR-10, Caltech-101, and Caltech-256.The networks were trained on unlabeled STL-10 data, so results on CIFAR-10 and Caltech datasets use transfer learning.
Surrogate classes: Classification accuracy peaks at about 8000 surrogate classes; adding more classes leaves performance unchanged or can reduce it.Validation error rises as the surrogate task becomes harder, with similar or identical samples creating difficult surrogate labels.
Clustering: Clustering surrogate seeds increases classification accuracy on all datasets, by up to 2.4% on STL-10 depending on the network.Clustering allows the approach to use larger amounts of data.
Surrogate samples: Performance improves with more samples per surrogate class and saturates at around 100 samples.Too few samples do not provide enough data to learn the desired invariance properties.
Transformations: Translations, color variation, and contrast variation matter more than rotation and scaling, with spatial and color-contrast invariance similarly important on STL-10 and CIFAR-10.On Caltech-101, color and contrast transformations matter more than spatial transformations because its images are often well aligned.

5 EXPERIMENTS: DESCRIPTOR MATCHING

The descriptor-matching experiments compare supervised and unsupervised CNN features with SIFT across datasets and transformations. Exemplar-CNN features generally outperform SIFT on Flickr, while transformation-specific training improves robustness, especially to blur.

5.1 Compared Features: The matching evaluation uses MSER regions, normalized patches, and method-specific optimized patch sizes; neural networks are slower than SIFT but not prohibitively so with parallel hardware.The Mikolajczyk benchmark has only 40 image pairs, limiting the reliability of conclusions about design choices.
5.4 Patch size and network layer: Network performance depends on patch size and feature-extraction layer: lower layers degrade after an optimal patch size, while higher layers can benefit from larger patches.The experiments use patch size 157 for SIFT, 113 for the neural methods, layer 4 for AlexNet and Exemplar-CNN-blur, and layer 3 for Exemplar-CNN-orig.
5.5 Results: Descriptor matching does not favor class-specific features: a surrogate-class loss outperforms object-class supervision when learning matching descriptors.The comparison indicates that matching requires invariances different from those emphasized by object classification.
5.5 Results: On Flickr, both AlexNet and Exemplar-CNN outperform SIFT, although SIFT is better for some image pairs and on Mikolajczyk can outperform AlexNet.The Mikolajczyk dataset contains real viewpoint and camera-setting changes, unlike the newly constructed dataset.
5.5 Results: Blur is the main weakness of standard neural-network descriptors: increasing blur reduces matching performance, while Exemplar-CNN-blur is substantially less affected.Training without blur leaves Exemplar-CNN with problems matching blurred images to originals; adding blur to training transformations addresses this issue.

6 CONCLUSIONS

The paper proposes discriminative unsupervised CNN training using surrogate labels generated from transformed image patches. The learned features improve unsupervised classification and, with transformation-specific training such as blur augmentation, support geometric matching against SIFT.

6 CONCLUSIONS: The method trains a CNN without object labels by generating surrogate labels through transformations whose variation defines the learned invariances.This objective is discriminative and uses data augmentation to construct surrogate classes.
6 CONCLUSIONS: For geometric matching, unsupervised features can outperform SIFT on most image pairs after blur is included among the training transformations.Without blur training, neural-network descriptors have a matching weakness on blurred images; blur augmentation reduces that problem.
6 CONCLUSIONS: Matching performance varies by transformation: networks are stronger for viewpoint changes, whereas SIFT remains more robust to strong blur and lighting changes.The reported transformation analyses cover Flickr and Mikolajczyk matching datasets.

APPENDIX A FORMAL ANALYSIS

The appendix proves convexity of the log-sum-exp function and characterizes its sole Hessian null direction. It then derives the equality condition for the associated inequality using Jensen’s inequality.

Convexity: The log-sum-exp function Z(x) is convex, as established from its Hessian and the Cauchy–Schwarz inequality.The Hessian quadratic form is shown to be nonnegative for all x and z.
Hessian kernel: The kernel of ∇2Z(x) is span(1), so the only Hessian-null direction is the all-ones direction.The equality analysis gives z = c1, with c independent of the coordinate index.
Proposition 1: Equality in inequality (7) occurs only when x(α1) − x(α2) lies in span(1) for every α1, α2 ∈ A.This condition means pairwise differences of the continuous function’s values must follow the Hessian’s null direction.
Proposition 1: Inequality (7) follows directly from convexity of log ∥exp(·)∥1 and Jensen’s inequality.The proof invokes Jensen’s inequality after establishing convexity.

APPENDIX B METHOD DETAILS

This appendix describes the evaluated network architectures and training procedure, including a clustering process used to improve Exemplar-CNN.

Method details: The appendix details the network architectures evaluated with the Exemplar-CNN training procedure.It also introduces the clustering process used to improve Exemplar-CNN.
Method details: The training procedure is evaluated across multiple network architectures.
Method details: Clustering is included as a process intended to improve Exemplar-CNN.

B.1 Network Architecture

The experiments use small and large convolutional architectures, with rectified linear units throughout hidden layers and dropout on the fully connected layer.

Network Architecture: 64c5-64c5-128f is the small network used to evaluate augmentation-procedure components.It has two convolutional layers with 64 filters of size 5 × 5, followed by a fully connected layer with 128 units.
Network Architecture: 64c5-128c5-256c5-512f is the large network used to achieve better classification performance.
Network Architecture: All networks use rectified linear units in every layer except the softmax layer.
Network Architecture: Dropout is applied to the fully connected layer.

B.2 Training the Networks

The networks are trained with stochastic gradient descent using momentum and a staged learning-rate schedule until validation error stops improving and training converges.

Training the Networks: Stochastic gradient descent is used with fixed momentum of 0.9 and an initial learning rate of 0.01.
Training the Networks: The learning rate is reduced by a factor of 3 whenever validation error stops improving.This reduction-and-training cycle is repeated until convergence.
Training the Networks: Training the 64c5-64c5-128f network takes roughly 1.5 days on a Titan GPU.The passage also reports roughly 4 days for the larger network, though the specification is truncated.

B.3 Clustering

Cluster similarity is estimated from overlaps among image groups formed by selecting top-firing images from linear SVMs. A greedy procedure uses fixed merge and discard thresholds to process these overlaps.

Linear SVMs select Npercluster = 10 top-firing images per SVM from the unlabeled STL-10 dataset to form initial clusters.
Pairwise cluster similarity is measured by counting the number of common images between clusters.
The clustering procedure uses thresholds Tmerge = 3 and Tdiscard = 1 in a greedy process starting with the most overlapping pair.

APPENDIX C DETAILS OF COMPUTING THE MEASURE OF INVARI-

The invariance measure compares normalized feature vectors for original image patches and transformed versions, then summarizes distance changes across transformation magnitudes. Additional normalization removes scale differences and prevents constant features from appearing perfectly invariant.

Feature vectors are computed for image patches and their transformed versions, normalized to unit Euclidean norm, and compared using Euclidean distance.
Distances between each original patch and its transformed versions are averaged across patches for every transformation and magnitude.
The resulting distance curves are divided by their maximal values to compensate for different feature variation across transformations.
Unit-length normalization places feature values in the same range, while curve normalization prevents constant features from being classified as perfectly invariant.
A steep curve indicates rapid loss of correlation under small transformations, whereas gradual growth indicates greater invariance.