PaperlayerView exact revision on arXiv

Source-linked AI summary

Representation Learning: A Review and New Perspectives

Yoshua Bengio, Aaron Courville, Pascal Vincent

arXiv:1206.5538v3cs.LG

TL;DR

Machine-learning performance depends strongly on how data representations organize explanatory factors, yet suitable objectives for learning such representations remain unclear. The paper reviews unsupervised feature learning and deep-learning approaches, including probabilistic models, auto-encoders, manifold learning, and deep networks. It concludes that progress depends on training criteria and priors that better support disentangling factors, while noting important optimization, modeling, and sampling limitations.

  • Problem

    Representation learning lacks a clear training objective for producing representations that disentangle underlying factors of variation rather than merely serving downstream prediction.

  • Method

    The paper surveys representation-learning research spanning probabilistic models, auto-encoders, manifold learning, and deep networks.

  • Results

    The review identifies priors and training criteria that may help representations discover and disentangle explanatory factors, while synthesizing progress in deep representation learning.

  • Takeaways & Limitations

    Future successful representation-learning systems should incorporate multiple priors, including sparsity, structured dependencies, and temporal or spatial coherence.

  • Takeaways & Limitations

    Stochastic maximum-likelihood training can become ineffective when large weights make sampling mix slowly, producing poor gradient estimates and stalled training.

Abstract

from arXiv · show

The success of machine learning algorithms generally depends on data representation, and we hypothesize that this is because different representations can entangle and hide more or less the different explanatory factors of variation behind the data. Although specific domain knowledge can be used to help design representations, learning with generic priors can also be used, and the quest for AI is motivating the design of more powerful representation-learning algorithms implementing such priors. This paper reviews recent work in the area of unsupervised feature learning and deep learning, covering advances in probabilistic models, auto-encoders, manifold learning, and deep networks. This motivates longer-term unanswered questions about the appropriate objectives for learning good representations, for computing representations (i.e., inference), and the geometrical connections between representation learning, density estimation and manifold learning.

1 INTRODUCTION

Machine-learning performance depends heavily on data representation, making feature engineering both important and labor-intensive. The paper surveys representation learning, especially deep methods, as a way to learn more useful and abstract representations with less reliance on hand-designed features.

  • Machine-learning performance is heavily dependent on the choice of data representation.
  • Feature engineering uses human ingenuity and prior knowledge but remains labor-intensive and limits the ease of deploying novel applications.
  • Representation learning aims to learn data representations that make useful information easier to extract for classifiers and other predictors.
  • The review focuses on deep learning methods that compose multiple non-linear transformations to produce increasingly abstract and useful representations.

2 WHY SHOULD WE CARE ABOUT LEARNING REPRESENTATIONS?

Representation learning has produced notable results across speech, music, vision, language, and transfer learning. These applications support the view that learned representations can capture shared factors and transfer statistical strength across tasks.

  • Deep learning reduced word error rates by about 30% on four major speech benchmarks, including RT03S from 27.4% to 18.5%.
  • Deep networks reduced ImageNet state-of-the-art error from 26.1% to 15.3% for object recognition.
  • The paper hypothesizes that learned representations support transfer because they capture underlying factors, some of which are relevant to particular tasks.
  • Representation-learning algorithms won two 2011 transfer-learning challenges, while related work found them advantageous for domain adaptation and multi-task learning.

3 WHAT MAKES A REPRESENTATION GOOD?

Good representations should encode useful structure in expressive, distributed, and potentially disentangled forms while supporting abstraction, feature reuse, and generalization. Deep architectures are attractive because they can reuse features across levels, although defining suitable training objectives remains an open challenge.

  • Useful representations can express general-purpose priors that help learners discover and disentangle underlying factors of variation.
  • Simple parametric models cannot capture the complexity of many AI tasks without an appropriate feature space, while smoothness alone is insufficient in raw high-dimensional input space.
  • Deep architectures promote feature reuse and can produce progressively more abstract features at higher representation layers.
  • 3.4 Depth and abstraction: Deep representations can be exponentially more efficient than insufficiently deep ones for some function families because their number of reusable paths grows with depth.
  • 3.4 Depth and abstraction: Representations with fewer parameters may require fewer examples and improve computational and statistical efficiency through parameter reuse.
  • A central open problem is how to define training criteria that make representations disentangle explanatory factors rather than merely maximize likelihood under a model.

4 BUILDING DEEP REPRESENTATIONS

The paper examines how single-layer feature-learning modules are stacked into deep architectures, including unsupervised pre-training, supervised use, and joint training. Several deep generative approaches are presented, but combining unsupervised layers remains unresolved.

  • Layerwise pre-training: Greedy layerwise unsupervised pre-training learns a hierarchy of features one level at a time.The resulting features can initialize deeper supervised models or serve as inputs to standard predictors.
  • Layerwise pre-training: Stacked feature extraction was empirically associated with better classification error, generated-sample quality, and feature invariance.These observations motivated the use of layerwise stacking in deep architectures.
  • Open problem: No clear winner emerges for combining unsupervised pre-trained layers into a better unsupervised model.The paper calls for further validation of existing proposals and improvement of these methods.
  • Deep generative models: Deep Belief Networks stack pre-trained RBMs, interpreting the top layer as an RBM and lower layers as a directed sigmoid belief network.The paper notes uncertainty about approximating maximum-likelihood training for further generative optimization.
  • Deep generative models: Deep Boltzmann Machines combine RBM parameters and apply approximate maximum-likelihood training jointly.The paper reports substantial improvements in likelihood and classification performance, while noting that more work is needed.
  • Alternative deep architectures: An alternative constructs a deep free-energy function by composing lower-layer transformations followed by top-level hidden units.This approach raises the question of how to train an arbitrary parametrized free-energy function.

5 SINGLE-LAYER LEARNING MODULES

The paper frames single-layer representation learning through probabilistic, neural, and manifold-learning perspectives. It reviews how these paradigms overlap in shallow modules but diverge in deeper models because inference can become intractable.

  • Parallel paradigms: Representation-learning research spans probabilistic graphical models and neural networks, differing in whether layers denote latent variables or computation graphs.The distinction concerns the interpretation of the same layered architecture.
  • Parallel paradigms: The two paradigms share similarities in recent single-layer greedy modules, especially restricted Boltzmann machines and auto-encoders.Their common focus on shallow modules has kept the deeper distinction less prominent.
  • Deep-model divergence: The connection weakens in deeper models because exact inference in probabilistic models typically becomes intractable.For deep models, the computational graph can diverge from the model structure; deep Boltzmann machine inference unrolls into a recurrent graph.
  • Review organization: The review organizes single-layer modules by whether they directly parameterize generative decoding pathways or encoding pathways.It presents probabilistic models separately from typically neural-network-based models.
  • Unified views: The paper introduces a single-layer algorithm spanning probabilistic, auto-encoder, and manifold-learning views, then uses PCA as an illustration.PCA learns a linear transformation whose columns span principal directions of greatest training-data variance.
  • Scope: ICA is not covered in detail because of space constraints, although its variants can produce nonlinear features and build deep networks.The paper directs readers to dedicated ICA references for further treatment.

6 PROBABILISTIC MODELS

From a probabilistic perspective, representation learning seeks latent random variables that parsimoniously describe the distribution of observed data. Feature values arise through inference over the latent variables given the observations.

  • Latent-variable view: Probabilistic feature learning models a joint distribution over latent variables h and observed variables x.The representation is obtained by inferring the latent-variable distribution conditioned on observed input.

6.1 Directed Graphical Models

Directed graphical models represent data through latent causes, separating a prior over latent variables from the conditional likelihood of observations. Sparse coding exploits explaining away to produce selective representations, with strong performance in low-label regimes but unresolved probabilistic interpretations.

  • Directed latent-factor models: Directed latent-factor models factor the joint distribution as p(x,h) = p(x|h)p(h), separately parameterizing likelihood and prior.Examples include PCA, sparse coding, sigmoid belief networks, and spike-and-slab sparse coding.
  • Explaining away: In directed models, observing an event can make otherwise independent causes dependent, a phenomenon called explaining away.Latent activations are interpreted as causes of observed data, so posterior inference over h becomes central to representation.
  • Sparse coding: Sparse coding uses a sparsity-inducing Laplace prior rather than PCA’s Gaussian prior, corresponding to an L1 penalty.Its representation contains many features set exactly to zero.
  • Sparse coding: Sparse coding recovers a MAP code h* and learns the dictionary by maximizing likelihood conditioned on those inferred codes.The dictionary is typically constrained because scaling can be exchanged between dictionary columns and feature activations.
  • Caveats: The sparse-coding MAP interpretation is not theoretically settled because the training criterion admits alternative Bayesian interpretations.The paper also notes that MAP-based parameter learning is not standard probabilistic graphical-model practice.
  • Explaining away: Sparse coding can select appropriate bases and zero correlated alternatives even with an overcomplete dictionary.The paper attributes this selectivity to explaining away and contrasts it with RBMs and auto-encoders.
  • Empirical results: Sparse coding significantly outperformed other competitive encoding schemes on CIFAR-10 when fewer than 1000 labeled examples per class were available.The comparison used a patch-based feature-extraction pipeline.
  • Spike-and-slab sparse coding: Spike-and-slab sparse coding uses binary spike variables to determine sparsity patterns and real-valued slab variables for activations.On CIFAR-10 and CIFAR-100 in relatively low-label regimes, it followed sparse coding’s superior pattern and sometimes outperformed sparse coding.

6.2 Undirected Graphical Models

Undirected graphical models represent visible and hidden variables through energy-based joint distributions. Restricting Boltzmann-machine interactions to an RBM makes conditional inference tractable, although partition-function estimation remains difficult.

  • Undirected graphical models parameterize joint distributions with products of non-negative clique potentials, normalized by a partition function.
  • Boltzmann machines use visible units for data and hidden units to mediate dependencies through an energy-defined interaction pattern.
  • Inference in a general Boltzmann machine is intractable because computing P(h_i | x) requires summing over 2^(d_h−1) hidden configurations.
  • RBMs remove within-layer interactions, forming a bipartite graph whose hidden and visible conditional distributions factorize.
  • RBM feature representations use posterior marginals P(h_i | x), which are immediately available because of conditional independence.
  • RBM conditional tractability does not extend to the partition function, which still requires summing exponentially many terms and therefore needs approximation.
  • RBMs have been applied across fMRI classification, motion transformations, collaborative filtering, and natural image modeling.

6.3 Generalizations of the RBM to Real-valued data

RBM generalizations extend the framework to real-valued data by modeling conditional covariance as well as mean structure. The mcRBM, mPoT, and ssRBM differ in how hidden units encode or constrain that covariance, with sparse-representation consequences in overcomplete settings.

  • Gaussian RBMs model real-valued observations by adding a quadratic visible-bias term, but they have proved somewhat unsatisfactory for natural images.
  • The mcRBM uses separate hidden-unit sets to parameterize the conditional mean and covariance of Gaussian-distributed visible data.
  • The mPoT model adds nonzero Gaussian means to PoT-style covariance modeling and has synthesized large-scale natural images with shadowing structure.
  • The ssRBM pairs each hidden unit’s binary spike with a real-valued slab to encode both mean and covariance information.
  • mcRBM, mPoT, and ssRBM all target real-valued data whose hidden units encode conditional mean and covariance, but use different covariance mechanisms.
  • The convolutionally trained μ-ssRBM samples are not reported to overfit particular CIFAR-10 training examples.
  • When hidden dimensionality differs substantially from input dimensionality, covariance parameterizations diverge in their behavior.
  • In overcomplete settings, sparse ssRBM activation permits variance only along selected directions, whereas mPoT and mcRBM constraints appear poorly suited to sparse representations.

6.4 RBM parameter estimation

RBM parameter estimation commonly approximates maximum-likelihood gradients because the positive phase is tractable but the negative phase is not. CD, SML, and FPCD use different Gibbs-chain strategies, each with distinct approximation or mixing concerns.

  • RBM training methods commonly include contrastive divergence and stochastic maximum likelihood, alongside earlier projection-pursuit learning for harmoniums.
  • Maximum-likelihood gradient updates contrast expectations under the data-conditioned positive phase with expectations under the model’s joint negative phase.
  • RBM conditional independence makes the positive-phase expectation tractable, while the partition-function-derived negative phase requires approximation.
  • Monte Carlo approximations estimate the negative phase using block Gibbs samples that alternate draws from P(x | h) and P(h | x).
  • Contrastive divergence initializes a short Gibbs chain at training data, reducing gradient-estimator variance while approximating the negative phase.
  • Stochastic maximum likelihood persists the Gibbs chain across updates instead of restarting it from each positive-phase sample.
  • SML depends on adequate Gibbs mixing; as RBM weights grow, ergodicity can break down and learning can fail unless the learning rate is reduced.
  • FPCD perturbs model parameters to encourage transitions between modes, but may generate spurious samples and does not converge to the model distribution as learning rates vanish.

7 DIRECTLY LEARNING A PARAMETRIC MAP FROM INPUT TO REPRESENTATION

Auto-encoders learn a parametric encoder from inputs to representations and a decoder back to reconstructed inputs, typically by minimizing reconstruction error. Regularized variants address representational limitations, while denoising and contractive objectives connect learned representations to robustness and estimated data structure.

  • Auto-Encoder Framework: An auto-encoder computes a representation h=fθ(x) with an encoder and reconstructs the input through a decoder gθ(h).The encoder and decoder are commonly affine mappings followed by nonlinearities or identity functions.
  • Auto-Encoder Framework: Training minimizes reconstruction error, usually with stochastic gradient descent, with the decoder and loss chosen according to the input domain.Squared error is natural for unbounded inputs, while sigmoid decoders and binary cross-entropy can suit bounded binary inputs.
  • Auto-Encoder Results: Linear auto-encoders with squared reconstruction error learn the same subspace as PCA, although tied weights prevent the parameter scaling that yields a linear encoder.The learned weight vectors need not form an orthonormal basis or have a meaningful ordering.
  • Regularized Representations: Overcomplete representations can memorize inputs through duplication, motivating regularization and alternatives such as sparse coding and predictive sparse decomposition.Predictive sparse decomposition uses an L1 penalty on hidden codes and constrains them to be approximated by a parametrized encoder.
  • Denoising Auto-Encoders: Denoising auto-encoders reconstruct clean inputs from corrupted ones, learning distributional structure and a reconstruction vector aligned with the estimated score.In a linear reconstruction and squared-error setting, affine-sigmoid-affine DAE training corresponds to an energy-based model close to a GRBM and uses denoising score matching.
  • Contractive Auto-Encoders: Contractive auto-encoders penalize encoder sensitivity analytically, while CAE+H additionally encourages Jacobians at nearby inputs to be similar to address infinitesimal-robustness limits.The CAE penalty trades reconstruction against robustness through λ; CAE+H penalizes higher-order derivatives stochastically.

8 REPRESENTATION LEARNING AS MANIFOLD LEARNING

Representation learning can be viewed as modeling a low-dimensional manifold supporting high-dimensional data, with representations providing local coordinates and tangent directions. Parametric and auto-encoding methods avoid some costs of neighborhood-based approaches while capturing nonlinear local structure.

  • 8 REPRESENTATION LEARNING AS MANIFOLD LEARNING: The manifold hypothesis posits that high-dimensional real-world data concentrate near a lower-dimensional manifold whose tangent directions capture meaningful variations.Data may lie near multiple disconnected manifolds, and density can fall sharply away from them.
  • 8.1 Learning a parametric mapping based on a neighborhood graph: Non-parametric manifold methods can be extended with parametrized mappings that compute embeddings for new points, while semi-supervised embedding jointly learns an embedding and classifier.The neighborhood-based criterion encourages training-set neighbors to have similar representations.
  • 8.1 Learning a parametric mapping based on a neighborhood graph: Parametric methods generalize manifold shape non-locally, but high-dimensional neighborhood graphs can be statistically unreliable and pairwise computation scales quadratically with training-set size.Nearest neighbors may share little semantic content, and many pairs may be needed even when sampled stochastically.
  • 8.2 Learning to represent non-linear manifolds: Unlike PCA’s fixed tangent space, nonlinear representations have input-dependent tangent spaces specified locally by encoder derivatives.If the encoder captures a concentrated manifold, its derivatives are non-zero mainly along tangent directions.
  • 8.3 Connections with other representation-learning methods: Sparse coding and feedforward encoders can be interpreted geometrically: active features or sensitive input directions span the local tangent space around an example.LCC further shows that Lipschitz-smooth functions on a smooth nonlinear manifold can be approximated by globally linear functions in the resulting code.
  • 8.2 Learning to represent non-linear manifolds: A Contractive Auto-Encoder’s sharply decreasing Jacobian singular spectrum provides empirical evidence for a low-dimensional manifold, whose leading singular vectors form its tangent plane.The tangent vectors correspond to plausible local deformations and differ across inputs because the estimated manifold is highly nonlinear.

9 CONNECTIONS BETWEEN PROBABILISTIC AND DIRECT ENCODING MODELS

The paper connects probabilistic latent-variable models and direct encoders through approximate inference, while interpreting regularized auto-encoders geometrically and probabilistically. These connections support sampling procedures but expose mixing and evaluation challenges as learned distributions become sharper or model capacity grows.

  • 9.1 PSD: a probabilistic interpretation: PSD combines a sparse-coding generative model with a jointly trained encoder that approximates MAP inference within a variational interpretation.Its training balances reconstruction-based likelihood improvement with reducing the discrepancy between encoder output and latent values.
  • 9.2 Regularized Auto-Encoders Capture Local Structure of the Density: Regularized auto-encoders can correspond to score matching, with contractive regularization making reconstruction functions and derivatives estimate derivatives of the data-generating density.This interpretation connects auto-encoder regularization to both density estimation and manifold geometry.
  • 9.2 Regularized Auto-Encoders Capture Local Structure of the Density: Auto-encoder sampling alternates between moving toward high-density manifold regions through reconstruction and adding noise along estimated tangent directions.The noise is added mainly along leading singular vectors of the reconstruction or encoder Jacobian.
  • 9.3 Generalizing the PSD Perspective: Iterative inference can be treated as a computation graph mapping inputs to representations and, for deterministic inference, as a function that can be optimized directly.This generalizes the direct-encoding perspective beyond PSD to latent-variable models.
  • 9.4 Sampling Challenges: MCMC becomes inefficient or unreliable during training when learned modes sharpen and become separated by low-density regions, slowing mixing between modes.Early high-entropy distributions mix more easily, whereas later distributions can stall learning that relies on good mixing.
  • 9.5 Evaluating and Monitoring Performance: Test reconstruction error may be misleading for auto-encoders and sparse coding because increasing capacity or training time systematically lowers it even on held-out data.Denoising reconstruction error avoids this issue for DAEs, but not for the hyperparameter controlling noise or contraction.

10 GLOBAL TRAINING OF DEEP MODELS

Global training of deep models is difficult because jointly optimizing many nonlinear layers creates challenging inference and optimization problems. The review discusses layerwise pre-training, supervised alternatives with abundant labels, and approximate training for deep unsupervised models.

  • 10 GLOBAL TRAINING OF DEEP MODELS: Jointly training all levels of a deep architecture is difficult, motivating methods that combine single-layer models or use layerwise training.The paper identifies joint training as a central challenge and notes that layerwise unsupervised or supervised training was easier to exploit.
  • 10 GLOBAL TRAINING OF DEEP MODELS: Unsupervised pre-training may guide intermediate representations toward useful local minima and provide a prior when labeled data are limited.The proposed explanation links input-distribution factors to target-relevant variation, while separating regularization and optimization effects remains difficult.
  • 10 GLOBAL TRAINING OF DEEP MODELS: Deep optimization is sensitive to initialization and nonlinearities, and small Jacobian singular values can cause gradients to vanish across layers.These difficulties motivate second-order methods, including Hessian-free approaches, for deep and recurrent networks.
  • 10 GLOBAL TRAINING OF DEEP MODELS: With large quantities of labeled data, very deep purely supervised networks can succeed without layerwise pre-training, which may then add little or no improvement.The review presents this as evidence that pre-training acts as a prior whose value decreases when labeled data are abundant.
  • 10.2.2 Training Deep Boltzmann Machines: For Deep Boltzmann Machines, hidden-unit interactions make posterior inference intractable, so training uses a mean-field approximation and maximizes a likelihood lower bound.The procedure iterates mean-field updates, estimates the negative phase through SML or Contrastive Divergence, and performs approximate stochastic gradient ascent.
  • 10.2.2 Training Deep Boltzmann Machines: DBM training can fall into poor local minima that leave many hidden units effectively inactive.This limitation is reported for the SML-based procedure used after approximate inference.

11 BUILDING-IN INVARIANCE

Building-in invariance uses generic and domain-informed priors, including transformations, topology, convolution, pooling, scattering, and temporal coherence. The central tension is making representations invariant to irrelevant variation without discarding informative factors or recoverable detail.

  • 11 BUILDING-IN INVARIANCE: Fair comparisons require tracking the prior knowledge used by each learning algorithm, while generic inductive biases aim to remain broadly applicable.The review contrasts hand-designed features with learned representations and emphasizes awareness of prior assumptions.
  • 11 BUILDING-IN INVARIANCE: Data augmentation improves generalization by generating examples through transformations known not to change the target variables.Examples include translations, rotations, scaling, and shearing of images.
  • 11 BUILDING-IN INVARIANCE: Convolution exploits local topology by sweeping shared feature extractors across inputs, producing similarly shaped feature maps and enabling hierarchical stacking.Patch-based unsupervised training learns local extractors that can be applied across positions and reused at successive layers.
  • 11.3 Temporal coherence and slow features: Temporal coherence has been used to model video and combined with unsupervised feature learning, where a coherence penalty improved classification performance.The simplest prior penalizes squared changes between feature values at consecutive times.
  • 11.2 Building-in Invariance: Invariant features remove sensitivity to task-irrelevant variation, but disentangling informative factors such as identity, action, and pose requires preserving their distinctions.Pooling and subsampling create higher-level invariance while exposing the representation to information loss.
  • 11.2 Building-in Invariance: Overlapping pools could provide redundant invariant features with less information loss, but no learning principle is identified that guarantees both invariance and maximal information retention.This remains an open design problem within feature-pooling approaches.
  • 11 BUILDING-IN INVARIANCE: Generative interpretations connect high-level representations to reconstruction and expose limitations of linear feature composition for disentangling factors.Pooling features alone may not accurately recover the data because credit cannot be apportioned to lower-level features.

12 CONCLUSION

The conclusion unifies probabilistic, reconstruction-based, and manifold-learning approaches while identifying representation priors, inference, and optimization as continuing challenges. Future progress depends on training criteria that better capture structured explanatory factors without ignoring practical limitations of deep models.

  • 12 CONCLUSION: The review connects probabilistic models, autoencoder-related reconstruction methods, and geometrically motivated manifold-learning approaches.These paradigms are presented as apparently disconnected, with their connections remaining an active research direction.
  • 12 CONCLUSION: Generic priors include multiple factors of variation, hierarchical abstraction, sparsity, simple dependencies, and temporal or spatial coherence.The paper argues that successful representation learning should incorporate several such priors rather than focus on only one.
  • 12 CONCLUSION: Pooling can cause information loss because high-level feature values do not specify how to apportion credit among lower-level features during reconstruction.This is the generative counterpart of the information loss introduced by pooling.
  • 12 CONCLUSION: Better training criteria that incorporate these priors are expected to move toward disentangling underlying explanatory factors.The conclusion frames this as a long-term objective rather than an established capability.
  • 12 CONCLUSION: Learned inference may incorporate iterative computation from probabilistic latent-variable models, but multimodal posteriors and approximate inference remain fundamental challenges.The review separates modeling, inference, and learning while noting that unaccounted approximation can harm optimization.
  • 12 CONCLUSION: The successes and failures of deep-architecture training, especially in unsupervised settings, remain insufficiently understood.The paper states that these basic questions remain unanswered and require further study.
Loading 1206.5538v3…