Source-linked AI summary
Deep Learning of Representations: Looking Forward
Yoshua Bengio
TL;DR
Deep learning research asks how to scale models and datasets, ease optimization, improve inference and sampling, and learn disentangled representations. The paper organizes these challenges and proposes research directions, while noting that current training becomes less efficient as models grow and that disentangling remains insufficient in some tasks. Its conclusion frames these four areas as the major challenges for advancing deep learning toward human-level performance.
Problem
Deep learning faces four major challenges: scaling computation, optimizing parameters, handling expensive inference and sampling, and learning representations that disentangle underlying factors of variation.
Method
The paper examines these challenges and proposes forward-looking research directions, including conditional computation, direct prediction that avoids latent-variable inference, and approaches to improve training efficiency and disentanglement.
Results
The paper identifies scaling computations, optimization difficulty, inference and sampling, and disentangled representation learning as the central challenges requiring further research.
Takeaways & Limitations
Future progress depends on exploring multiple directions across computation, optimization, inference, sampling, and representation disentanglement.
Takeaways & Limitations
Current training may scale worse than linearly with model size, and unsupervised pre-training did not produce enough disentangling for at least one difficult task.
Abstract
from arXiv · showhide
Deep learning research aims at discovering learning algorithms that discover multiple levels of distributed representations, with higher levels representing more abstract concepts. Although the study of deep learning has already led to impressive theoretical results, learning algorithms and breakthrough experiments, several challenges lie ahead. This paper proposes to examine some of these challenges, centering on the questions of scaling deep learning algorithms to much larger models and datasets, reducing optimization difficulties due to ill-conditioning or local minima, designing more efficient and powerful inference and sampling procedures, and learning to disentangle the factors of variation underlying the observed data. It also proposes a few forward-looking research directions aimed at overcoming these challenges.
1 Background on Deep Learning
Deep learning seeks multiple levels of increasingly abstract representations to reduce reliance on labor-intensive, human-designed feature engineering. Early work established representation learning's transfer benefits and broad empirical impact.
- Deep learning achieved important empirical successes in traditional AI applications, including computer vision and natural language processing.
- Feature engineering improves learning by using human ingenuity and prior knowledge, but remains labor-intensive and limits application scope.
- Representation learning aims to discover explanatory factors or features automatically rather than depending on manually engineered inputs.
- Deep learning discovers multiple representation levels, with higher-level features encoding more abstract aspects of data.
- Unsupervised representation learning won both 2011 Transfer Learning Challenges, demonstrating transfer across tasks.
2 Quick Overview of Deep Learning Algorithms
Deep learning combines layered representations with supervised, unsupervised, and architectural methods for learning useful features. The overview also highlights trade-offs involving optimization sensitivity, inference tractability, and the interpretation of learned representations.
- Deep architectures construct abstract concepts from less abstract ones, motivating automated discovery of semantic representations.
- Deep supervised networks became trainable through improved initialization, rectifying nonlinearities, convolutional architectures, and dropout.
- Convolutional networks use local receptive windows and shared parameters across positions, while dropout randomly removes units during training to limit co-adaptation.
- Layer-wise pre-training reduces sensitivity to initialization by training successive layers with local supervised or unsupervised objectives.
- Directed graphical models face expensive or intractable inference because latent factors compete through explaining away, whereas regularized auto-encoders directly map inputs to representations without intermediate latent variables.
- Sparse coding offers convex MAP inference with fast approximations, but reconstruction error can be small at density troughs as well as modes, limiting its use as an energy function.
3 Scaling Computations
Scaling deep learning requires larger models and datasets, but sequential training, communication costs, and parallelization limits make this difficult. The paper proposes sparse updates and learned conditional computation to reduce computation and communication.
- Scaling Computations: 0.27% error was reported for unconstrained MNIST, while ImageNet top-five error fell from 26.1% to 15.3%.These results illustrate progress from small digit images to large-scale natural-image recognition.
- Scaling Computations: Deep learning must scale beyond digit recognition toward larger models, huge datasets, and harder tasks such as object, scene, speech, and language understanding.The paper contrasts near-human handwritten-digit performance with substantial remaining gaps on broader AI-scale tasks.
- Scaling Computations: Current SGD updates are inherently sequential, and very large datasets may not fit on a disk or be scanned by a single CPU core.Parallelizing across many nodes introduces communication costs, while asynchronous implementations can use parallel resources inefficiently.
- Scaling Computations: Sparse updates would reduce the fraction of parameters exchanged between nodes for each minibatch.The paper also proposes learned conditional computation, using sparse activations and multiplicative gating to avoid unnecessary computation.
- Scaling Computations: Training signals for middle-layer gating are difficult because enumerating gating configurations is exponential in distributed settings.Stochastic gating is proposed to sample configurations and provide noisy training signals for the gating units.
4 Optimization
Deep networks remain difficult to optimize because increasing size can reduce training efficiency, while gradients and parameter interactions create conditioning challenges. The paper surveys optimizer, architecture, initialization, and training-signal strategies for addressing these difficulties.
- Optimization: Deep-network training involves difficult optimization, with unresolved contributions from local minima and ill-conditioning.The paper motivates studying these difficulties to improve generalization and potentially scaling.
- Optimization: The return on investment from adding hidden units decreases with network size and can fall below 1 after a fixed number of training iterations.The paper links this decline to stronger second-order parameter interactions and a larger Hessian condition number.
- Optimization: Standard black-box algorithms, including supervised and pretrained deep networks, fail on apparently simple tasks requiring compositions of multiple nonlinear abstractions.A network becomes easily solvable when an intermediate layer is pretrained for the first-level subtask.
- Optimization: Candidate remedies include adaptive learning rates, natural-gradient methods, large-minibatch second-order methods, and architecture or objective changes that improve Hessian conditioning.The paper also discusses input normalization, skip-connections, and rectifier nonlinearities as related approaches.
- Optimization: Focused gradient flow helps assign credit and blame, while long nonlinear compositions can produce vanishing or exploding gradients.Initialization procedures and sparse gradients are discussed as ways to address these training difficulties; practical second-order methods have limitations.
5 Inference and Sampling
Inference and sampling are central computational challenges because they can slow training and struggle with highly multimodal posteriors. The paper proposes bypassing explicit latent-variable inference by learning task-oriented computational graphs directly.
- Core challenges: Inference estimates latent variables given visible input, while sampling generates from conditional or joint model distributions.Approaches include MAP, MCMC, and variational inference; sampling is often needed inside learning algorithms.
- Core challenges: Iterative inference and sampling procedures can considerably slow training because they often occur inside the learning loop.
- Core challenges: Highly multimodal posteriors make MAP, factorized variational inference, and MCMC potentially poor, slow, or computationally impractical.The posterior may contain a huge number of significant modes that are difficult to represent, enumerate, or visit efficiently.
- Potential directions: Deeper representations have been observed to make mixing between modes easier in models such as deep belief networks and stacked regularized auto-encoders.
- Potential directions: The paper proposes avoiding explicit inference by skipping the latent variables and directly learning the task-relevant prediction or decision.This aims to leave function approximation as the remaining approximation error rather than compounding it with approximate inference.
- Potential directions: The proposed computational graphs need not approximate latent-variable distributions, allowing architectures inspired by inference or sampling while training shared parameters for the desired tasks.
6 Disentangling
Disentangling seeks representations that separate controllable explanatory factors while preserving information needed across tasks. The paper identifies encouraging observations but emphasizes that current methods remain insufficient and poorly understood.
- The challenge: Underlying factors can be viewed as separately controllable sources whose interactions generate complex observations.For images, examples include light sources, object shapes, and material properties.
- The challenge: Disentangling differs from invariance because it aims to separate factors while discarding as little information as practical.Invariant features reduce sensitivity to task-irrelevant variation, whereas disentangled representations preserve factor information for multiple tasks.
- Potential impact: A disentangled representation would make new supervised tasks easier when relevant factors can be predicted from a simple transformation using few representation elements.
- Evidence: Current unsupervised representation-learning methods show signs of reducing entanglement, including hidden units that are selectively sensitive to known variation factors.
- Evidence: Deeper representations have also been empirically associated with easier mixing between modes, with hypotheses involving more uniform-looking transformed distributions.
- Open limitations: Existing observations are insufficient: the mechanisms remain unclear, and unsupervised pre-training failed to disentangle enough for one artificial task.The paper suggests that algorithmic variants or fundamentally different unsupervised methods may be needed.
- Potential directions: Proposed priors for discovering factors include shared factors across tasks, manifolds, and natural clustering.These priors are intended to guide representation learning toward underlying factors of variation.
7 Conclusion
The paper organizes deep learning’s remaining challenges into four areas: scaling computations, optimization, inference and sampling, and learning disentangled representations.
- The four challenge areas are scaling computations, reducing optimization difficulties, designing or avoiding expensive inference and sampling, and learning disentangled representations.