PaperlayerView exact revision on arXiv

Source-linked AI summary

Building high-level features using large scale unsupervised learning

Quoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeff Dean, Andrew Y. Ng

arXiv:1112.6209v5cs.LG

TL;DR

The paper asks whether high-level, class-specific feature detectors can be learned from only unlabeled images, despite prior unsupervised methods mainly yielding low-level features. It trains a large sparse deep autoencoder with local connectivity, pooling, and local contrast normalization on extensive unlabeled imagery. The resulting representations produce face-selective, transformation-robust detectors and strong ImageNet recognition performance.

  • Problem

    The paper asks whether high-level, class-specific feature detectors can be learned from only unlabeled data, when prior unsupervised methods mainly produced low-level features.

  • Method

    The authors scale a sparse deep autoencoder with local receptive fields, pooling, and local contrast normalization to large unlabeled images.

  • Results

    15.8% accuracy for ImageNet object recognition across 20,000 categories is a 70% relative improvement over the state-of-the-art, alongside face-selective and other high-level detectors.

  • Takeaways & Limitations

    The experiments show that unlabeled data can yield high-level class-specific features, including a face detector robust to translation, scaling, and out-of-plane rotation.

Abstract

from arXiv · show

We consider the problem of building high-level, class-specific feature detectors from only unlabeled data. For example, is it possible to learn a face detector using only unlabeled images? To answer this, we train a 9-layered locally connected sparse autoencoder with pooling and local contrast normalization on a large dataset of images (the model has 1 billion connections, the dataset has 10 million 200x200 pixel images downloaded from the Internet). We train this network using model parallelism and asynchronous SGD on a cluster with 1,000 machines (16,000 cores) for three days. Contrary to what appears to be a widely-held intuition, our experimental results reveal that it is possible to train a face detector without having to label images as containing a face or not. Control experiments show that this feature detector is robust not only to translation but also to scaling and out-of-plane rotation. We also find that the same network is sensitive to other high-level concepts such as cat faces and human bodies. Starting with these learned features, we trained our network to obtain 15.8% accuracy in recognizing 20,000 object categories from ImageNet, a leap of 70% relative improvement over the previous state-of-the-art.

1. Introduction

This work asks whether high-level, class-specific feature detectors can be learned from unlabeled images, addressing the limitations of labeled-data-dependent methods and prior low-level unsupervised features. By scaling data, model, and computation, the authors report face-selective and other high-level detectors, robust transformations, and improved ImageNet recognition.

  • 1. Introduction: Unlabeled data have rarely produced high-level features beyond simple edge or blob detectors, motivating this study of complex invariances.Prior methods had generally succeeded only on low-level features, while labeled datasets were costly or scarce.
  • 1. Introduction: The authors scale the dataset, model, and computational resources to train a deep autoencoder on large 200x200 images.The model uses pooling and local contrast normalization, with computation distributed across a large cluster.
  • 1. Introduction: A hold-out face-versus-distractor test reveals a feature highly selective for faces without labels indicating face presence.Classification and visualization were used to confirm the learned high-level feature.
  • 1. Introduction: The learned face detector is invariant to translation, out-of-plane rotation, and scaling.These properties were established through control experiments.
  • 1. Introduction: The same network learns concepts including cat faces and human bodies.These findings extend the observed high-level selectivity beyond faces.

2. Training set construction

The training set contains one 200x200 color image sampled from each of 10 million YouTube videos. A face-detector check found that fewer than 3% of sampled patches were detected as faces.

  • 2. Training set construction: 10 million YouTube videos contribute one sampled frame each, avoiding duplicates in the training dataset.Each example is a color image with 200x200 pixels.
  • 2. Training set construction: 200x200 color images form the training examples.The dataset construction uses one image per video.
  • 2. Training set construction: Fewer than 3% of 100,000 randomly sampled 60x60 patches were detected as faces by OpenCV.This experiment estimated the proportion of faces present in the dataset.

3. Algorithm

The algorithm builds a scalable sparse deep autoencoder from local receptive fields, pooling, and local contrast normalization. It jointly trains this architecture with a reconstruction-and-invariance objective using distributed, asynchronous optimization.

  • Architecture: The model is a sparse deep autoencoder built from local receptive fields, pooling, and local contrast normalization.Each feature connects only to a small lower-layer region; pooling and normalization are used to obtain invariance to local deformations.
  • Architecture: Three repetitions of local filtering, pooling, and normalization form a nine-layered network.The output of each stage feeds the next stage.
  • Architecture: 18x18 receptive fields and 5x5 overlapping pooling neighborhoods implement the local connectivity pattern.The first sublayer produces linear filter responses, while L2 pooling outputs the square root of summed squared inputs.
  • Architecture: Unshared local weights allow the network to learn invariances beyond translational invariance.Unlike convolutional architectures, parameters are not shared across image locations.
  • Learning and Optimization: The objective combines reconstruction with a sparsity-and-invariance tradeoff and is optimized jointly across all three layers.The tradeoff parameter is λ = 0.1; the second objective term encourages pooling features to group similar features.
  • Learning and Optimization: Model parallelism and asynchronous SGD distribute weights, replicas, and gradient updates across machines.Asynchronous SGD allows other replicas to continue when one machine is slow or fails.

4. Experiments on Faces

The learned network produces a face-sensitive neuron from unlabeled training data, with strong recognition performance and robustness to several transformations. Visualization and control experiments further test what the neuron detects and which training conditions matter.

  • 4.3. Recognition: The test set contained 37,000 images, including 13,026 non-aligned faces and distractors sampled from ImageNet.Neuron accuracy was selected as the best result among 20 activation thresholds.
  • 4.3. Recognition: 81.7% accuracy was achieved by the best neuron in detecting faces, versus 64.8% for always guessing negative.The one-layered network and best randomly sampled linear filter achieved 71% and 74%, respectively.
  • 4.3. Recognition: Face images generally produced neuron activations above 0, whereas random images generally produced activations below 0.Activation histograms compare faces with no-face distractors using a test set subsampled to equal class ratios.
  • 4.4. Visualization: Numerical optimization and highly responsive test stimuli both confirm that the tested neuron learns the concept of faces.The two visualization methods have complementary weaknesses: stimulus visualization can fit noise, while optimization can encounter local minima.
  • 4.5. Invariance properties: The face detector is robust to translation, scaling, and out-of-plane rotation.The transformation experiments evaluate averaged neuron responses on distorted or 3D-rotated face images.
  • 4.6. Control experiments: 72.5% accuracy remained after training images detected as containing faces were removed, compared with 81.7% using the original training set.The reduced result was as low as the simple linear-filter results reported in the section.

5. Cat and human body detectors

The network is evaluated for additional high-level detectors using datasets for cat faces and human bodies. It learns both concepts and outperforms the reported linear-filter baselines on recognition rates.

  • 5. Cat and human body detectors: The experiments construct separate datasets for classifying human bodies and cat faces against random distractors.The positive-to-negative ratio is kept identical to the face dataset for easier interpretation.
  • 5. Cat and human body detectors: The same experimental protocols confirm that the network learns concepts of cat faces and human bodies in addition to faces.The results are presented through visualizations of the corresponding neurons.
  • 5. Cat and human body detectors: 74.8% and 76.7% recognition rates were achieved for cat and human body detectors, respectively.The corresponding best linear filters achieved 67.2% and 68.1%.
  • 5. Cat and human body detectors: The comparison includes deep autoencoders and K-means as additional baseline algorithms.These baseline results are reported in the bottom of Table 1.

6. Object recognition with ImageNet

The learned features were evaluated for ImageNet object recognition and compared with established baselines under the paper’s validation protocol. Starting from raw pixels, the method outperformed many hand-engineered features and achieved its largest reported relative improvement on the 22K-category task.

  • The ImageNet evaluation added one-versus-all logistic classifiers on top of the network’s highest-layer features.
  • Table 1 compares the algorithm with random guesses, random initialization, linear filters, trained neurons, autoencoders, and K-means.The table separates methods that do not require training from trained methods and other feature-learning baselines.
  • The datasets were randomly split into training and validation halves, with validation performance compared against state-of-the-art baselines.The authors note that splits differed from previous work and could slightly affect validation performance.
  • 70% relative improvement over the highest other reported result was achieved on ImageNet with 22K categories.Random guessing achieves less than 0.005% accuracy for this dataset.
  • 15% relative improvement over the previous best published result was achieved on ImageNet with 10K categories.

7. Conclusion

The paper concludes that unlabeled data can produce high-level, class-specific representations that support both selective detection and discriminative recognition. These representations also exhibit robustness to scale and out-of-plane variation.

  • The study simulated high-level class-specific neurons using unlabeled data.
  • The learned detector showed invariances to out-of-plane rotation and scale.
  • 15.8% accuracy on ImageNet with 20,000 categories represented a 70% relative improvement over the state-of-the-art.

A. Training and test images

The training and test images contain varied visual appearances, including diverse face positions, scales, and orientations. The paper illustrates both the training set and the held-out test examples used to identify the face neuron.

  • Training images contained faces with diverse positions, scales, and orientations.
  • Figure 7 shows thirty randomly selected training images before whitening.
  • A subset of test images was used for identifying the face neuron.
  • Figure 8 shows example test-set images before whitening.

B. Models

The model uses locally connected architecture and parallelized computation, with connectivity, parameter storage, and processing distributed across machines and cores. Control experiments examine sensitivity to receptive-field size and map count.

  • B. Models: Locally connected networks restrict each neuron to a local region of the preceding layer.
  • B. Models: The architecture uses distinct connectivity patterns for pooling, simple, and local contrast normalization neurons.Pooling neurons connect to one map, while simple and LCN neurons connect to all maps; the figure also distinguishes map-specific and all-map weights.
  • B. Models: Model parallelism distributes parameter storage and gradient computations across machines according to image locality.Weights connected to the left, center, and right image regions are stored in separate partitions.
  • B. Models: The implementation divides cores among data reading, data transfer, and arithmetic computation, allowing these tasks to run in parallel.
  • B. Models: Increasing receptive-field size or map count requires additional memory, computation, machines, or network bandwidth.
  • B. Models: Control experiments found that face-recognition accuracy was only slightly sensitive to receptive-field size and map count.The effects were measured by varying each hyper-parameter and observing test-set accuracy.

F. Example out-of-plane rotated face sequence

The experiments examine whether learned features respond to out-of-plane face rotations and whether neuron activations distinguish target concepts from distractors. Distinct activation histograms suggest that the network learned concept detectors.

  • F. Example out-of-plane rotated face sequence: Figure 12 presents a sequence of 3D out-of-plane rotated faces from one individual, with 10 sequences in the dataset.The faces are black and white but treated as color pictures in testing.
  • F. Example out-of-plane rotated face sequence: The control experiments compare the learned features with “best linear filters.”The baseline samples 100,000 random image-sized patches, computes cosine distances to test images, and selects thresholds for face-versus-distractor classification.
  • F. Example out-of-plane rotated face sequence: 100,000 randomly selected training patches determine the reported baseline accuracy by taking the best-performing patch.Each patch produces one test-set accuracy after threshold selection.
  • F. Example out-of-plane rotated face sequence: Distinctive activation histograms for positive and negative images suggest that the network learned concept detectors.The paper shows detailed histograms for neurons across the entire test sets, including face, human-body, and cat neurons.
  • F. Example out-of-plane rotated face sequence: Figure 16 displays the most responsive cat and human-body stimuli on the test sets.The human-body neuron’s top stimuli are black and white because the corresponding test images are black and white.

J. Implementation details for autoencoders and K-means

The implementation uses locally connected sigmoidal autoencoders and computationally reduced K-means models, with parameters selected under resource constraints. The experiments also visualize cat and human-body neuron responses.

  • J. Implementation details for autoencoders and K-means: The deep autoencoders are locally connected and use sigmoidal activation functions.The implementation also uses parallelism comparable to that described in the paper.
  • J. Implementation details for autoencoders and K-means: K-means downsamples images to 40x40 and uses 30,000 centroids to reduce computational costs.Autoencoder, K-means, and related parameters were varied to maximize performance given resource constraints.
  • J. Implementation details for autoencoders and K-means: The cat-neuron figure shows the most responsive cat stimuli on the test set.The figure pairs these with the most responsive human-body stimuli for the human-body neuron.
Loading 1112.6209v5…