Deep Learning in Neural Networks: An Overview

Juergen Schmidhuber

Source-linked AI summary

Deep Learning in Neural Networks: An Overview

Juergen Schmidhuber

arXiv:1404.7828v4Apr 30, 2014cs.NEcs.LG

TL;DR

Deep neural networks face a fundamental credit-assignment problem, especially along deep or long temporal paths. This survey synthesizes supervised, unsupervised, reinforcement, evolutionary, and indirect-search approaches, concluding that methods alleviating deep credit-assignment difficulties support learning across these settings.

Problem
Deep neural networks are difficult to train because backpropagated error signals can vanish or explode with layer count or credit-assignment-path depth.
Method
The survey synthesizes historical approaches to deep supervised, unsupervised, reinforcement, evolutionary, and indirect-search learning in neural networks.
Results
The survey concludes that alleviating deep credit-assignment-path problems supports supervised, unsupervised, and reinforcement learning with deep neural networks.
Takeaways & Limitations
Searching for low-complexity, perturbation-resistant neural networks can reduce overfitting and improve deep supervised, unsupervised, and reinforcement learning.
Takeaways & Limitations
Finding a neural network that precisely models a given training set is NP-complete, including for deep neural networks.

Abstract

from arXiv · show

In recent years, deep artificial neural networks (including recurrent ones) have won numerous contests in pattern recognition and machine learning. This historical survey compactly summarises relevant work, much of it from the previous millennium. Shallow and deep learners are distinguished by the depth of their credit assignment paths, which are chains of possibly learnable, causal links between actions and effects. I review deep supervised learning (also recapitulating the history of backpropagation), unsupervised learning, reinforcement learning & evolutionary computation, and indirect search for short programs encoding deep and large networks.

Abbreviations in Alphabetical Order

This section defines abbreviations used throughout the paper, covering neural-network methods, learning algorithms, architectures, and related technical concepts. The entries are arranged alphabetically.

AE denotes Autoencoder; AI, Artificial Intelligence; ANN, Artificial Neural Network; BFGS, Broyden-Fletcher-Goldfarb-Shanno; and BNN, Biological Neural Network.
BM denotes Boltzmann Machine; BP, Backpropagation; BRNN, Bi-directional Recurrent Neural Network; CAP, Credit Assignment Path; and CEC, Constant Error Carousel.
CFL denotes Context Free Language; CNN, Convolutional Neural Network; CoSyNE, Co-Synaptic Neuro-Evolution; CSL, Context Senistive Language; and CTC, Connectionist Temporal Classificati.
HTM denotes Hierarchical Temporal Memory; HMAX, Hierarchical Model “and X”; LSTM, Long Short-Term Memory (RNN); MDL, Minimum Description Length; and MDP, Markov Decision Process.
MP denotes Max-Pooling; MPCNN, Max-Pooling CNN; NE, NeuroEvolution; NEAT, NE of Augmenting Topologies; NES, Natural Evolution Strategies; NFQ, Neural Fitted Q-Learning; NN, Neural Network; OCR, Optical Character Recognition; PCC, Potential Causal Connection; and PDCC, Potential Direct Ca.

1 Introduction to Deep Learning (DL) in Neural Networks (NNs)

This section frames deep learning as a commercially important approach to the fundamental credit assignment problem in artificial neural networks. It distinguishes shallow and deep models by successive nonlinear processing stages and highlights backpropagation, feedforward networks, and recurrent networks in the field’s development.

Motivation: Deep learning addresses the fundamental credit assignment problem: identifying which modifiable components cause success or failure and how to change them to improve performance.The survey narrows this general problem to the commercially important subfield of deep learning in artificial neural networks.
Neural-network foundations: Neural networks comprise connected neurons whose weighted activations can produce desired behavior through learned credit assignment.Inputs activate neurons through sensors, intermediate neurons through weighted connections, and some neurons can trigger actions in the environment.
Historical development: Deep models use several successive nonlinear neuron layers, whereas shallow models contain few such stages; backpropagation enables gradient-based supervised learning at arbitrary depth.The survey traces multilayer models to at least the 1960s and 1970s, with backpropagation developed during those decades and applied to neural networks in 1981.
Architectures: Both feedforward and recurrent neural networks have won contests, while recurrent networks can in principle process memories of arbitrary input sequences.The survey characterizes recurrent networks as more powerful general computers than feedforward networks because of their cyclic structure.

2 Event-Oriented Notation for Activation Spreading in NNs

The section introduces event-oriented notation for describing changing NN topologies and partially causal activation-spreading episodes. It also distinguishes weight sharing and the supervised-learning and reinforcement-learning objectives expressed through these events.

Topology: An NN at a given moment is represented by finite units N and directed edges H, with feedforward networks acyclic and recurrent networks cyclic.The topology may change over time; feedforward layers are defined by edge-path lengths from input units.
Event-oriented notation: An episode is a partially causal sequence of real-valued events, comprising environment inputs or unit activations dependent on earlier events through topology-dependent incoming links.A topology-encoding function maps event index pairs to weight indices, and activation functions may be nonlinear.
Weight sharing: The same weight may be reused across space or time in recurrent and convolutional networks, representing weight sharing across topology-dependent activations.Episodes can contain different time-varying activations of the same unit, especially in sequence-processing recurrent networks.
Learning objectives: Supervised learning seeks weights producing episodes with small total error, whereas reinforcement learning seeks weights producing high cumulative environmental reward through suitable actions.Supervised learning associates output events with teacher-given targets; its independence assumption does not hold in sequential decision making and reinforcement learning.

3 Depth of Credit Assignment Paths (CAPs) and of Problems

The section defines Credit Assignment Paths (CAPs) as causal chains and uses their depth to characterize how far credit assignment can move backward to modifiable weights. Problem depth is independent of temporal distance and does not necessarily determine difficulty; for this overview, depth > 10 requires Very Deep Learning.

Credit Assignment Paths are chains of possibly causal links connecting events, such as inputs, hidden layers, and outputs in FNNs or transformations over time in RNNs.CAPs provide a basis for distinguishing deep from shallow credit assignment.
A CAP’s depth is the suffix length beginning at the first successive link with a modifiable weight, determining how far backward credit assignment can search.Depth is 0 when no links are modifiable.
Problem and solution depth depend on causal-chain depth rather than temporal distance, so shallow FNNs can handle long input sequences with long relevant-event lags.Large input time windows can support shallow problems despite long temporal gaps.
The boundary between Shallow Learning and Deep Learning remains inconclusive, but this overview defines problems of depth > 10 as requiring Very Deep Learning.The threshold is an authorial convention rather than a conclusive general answer.
Problem difficulty may have little to do with depth because random guessing, direct or indirect search, transfer from shallow problems, and operation collapsing can solve some deep problems quickly.These mechanisms can reduce the practical difficulty of finding solutions even when causal chains are deep.

4 Recurring Themes of Deep Learning

Deep learning repeatedly draws on dynamic programming, unsupervised representations, hierarchical abstraction, simplicity principles, and advances in computing hardware. These themes support credit assignment, reduce learning search spaces, structure representations, evaluate solution complexity, and enable efficient computation.

Dynamic Programming: Dynamic programming supports credit assignment under certain assumptions and can reduce problem depth in supervised and reinforcement learning.Backpropagation is described as a dynamic-programming-derived method, while traditional Markovian reinforcement learning uses DP-derived methods to greatly reduce problem depth.
Unsupervised Learning: Unsupervised learning encodes raw data into less redundant or more compact representations for subsequent supervised or reinforcement learning.These codes can make downstream search spaces smaller and credit assignment paths shallower.
Hierarchical Representations: AI and machine-learning methods commonly learn hierarchies of increasingly abstract data representations.Examples include syntactic pattern-recognition methods that discover hierarchies of formal rules modeling observations.
Simplicity and Description Length: Occam’s razor favors simple solutions, while Minimum Description Length measures candidate complexity by the shortest program that computes it.The measure depends on the chosen programming language.
Hardware: Cheap, multiprocessor graphics cards or GPUs produced a deep-learning hardware breakthrough in the new millennium.Earlier work pursued neural-network-specific hardware and exploitation of standard hardware, whereas GPUs were widely used for video games.

5 Supervised NNs, Some Helped by Unsupervised NNs … 5.5 1960-1981 and Beyond: Development of Backpropagation (BP) for NNs

The section traces neural-network development from early supervised and unsupervised models through deep architectures and the historical emergence of efficient backpropagation. It shows how biological inspiration, GMDH, the Neocognitron, and BP shaped later deep learning.

5 Supervised NNs, Some Helped by Unsupervised NNs: Supervised learning dominated recent pattern-recognition contests, while unsupervised methods often supplemented it and blurred the boundary between learning paradigms.Gradient-based methods such as BP optimized objectives in both supervised and unsupervised learning, especially for time series and sequence classification.
5.1 Early NNs Since the 1940s (and the 1800s): Early neural networks progressed from nonlearning architectures to simple supervised and unsupervised learners, alongside associative memories, from the 1940s through the early 1980s.Examples include McCulloch–Pitts networks, Hebbian learning, perceptrons, Widrow–Hoff learning, Grossberg models, Kohonen maps, and Hopfield memories.
5.2 Around 1960: Visual Cortex Provides Inspiration for DL (Sec. 5.4, 5.11): Visual-cortex studies identified simple and complex cells responding to visual properties such as edge orientation, with complex cells providing greater spatial invariance.These findings inspired later deep architectures and modern award-winning deep learners.
5.3 1965: Deep Networks Based on the Group Method of Data Handling: GMDH networks, developed from 1965 onward, were perhaps the first feedforward multilayer-perceptron deep-learning systems.Their units could use polynomial activation functions implementing Kolmogorov–Gabor polynomials, more general than widely used neural-network activations.
5.4 1979: Convolution + Weight Replication + Subsampling (Neocognitron): The Neocognitron was perhaps the first deep artificial neural network integrating neurophysiological insights, introducing convolution, weight replication, and subsampling.Its subsampling units responded to neighboring convolutional activity and were insensitive to certain small image shifts.
5.5 1960-1981 and Beyond: Development of Backpropagation (BP) for NNs: Gradient descent for complex multilayer systems was discussed by the early 1960s, and backpropagation could be derived by iterating the chain rule through network layers.Early formulations used Euler–Lagrange methods, dynamic-programming ideas, or simplified chain-rule derivations.
5.5 1960-1981 and Beyond: Development of Backpropagation (BP) for NNs: 1970: Linnainmaa apparently first described explicit, efficient error backpropagation for arbitrary, discrete, possibly sparsely connected networks, later applied to neural-network cost minimization.BP is also known as reverse-mode automatic differentiation, with backward derivative costs essentially matching forward activation costs.
5.5.1 BP for Weight-Sharing Feedforward NNs (FNNs) and Recurrent NNs (RNNs): As of 2014, BP remained the central learning algorithm for feedforward and recurrent neural networks, and most contest-winning networks used supervised BP without unsupervised augmentation.The backward pass costs essentially the same as the forward pass, and the passes are repeated until sufficient performance is reached.

5.6 Late 1980s-2000 and Beyond: Numerous Improvements of NNs

From the late 1980s onward, researchers developed methods addressing backpropagation’s apparent limitations on deep and long-lag problems. Improvements included sequence-processing methods, optimization and regularization techniques, program search, and unsupervised learning objectives.

Limits of backpropagation: By the late 1980s, backpropagation appeared effective mainly for shallow problems, while recurrent networks often restricted credit assignment or pretrained on shallow tasks.Although backpropagation allowed deep problems in principle, additional hidden layers often showed no empirical benefit, and many recurrent algorithms assigned credit only one step backward.
Temporal sequence learning: Sequence-processing methods addressed long time lags using activation decay, time delays, adaptive extensions, NARX networks, hierarchical RNNs, and related mechanisms.These approaches were designed for deep learning of temporal sequences.
Optimization improvements: Numerous BP improvements accelerated training through momentum, adaptive step sizes, sign-based derivatives, gradient normalization, Hessian approximations, and conjugate-gradient or quasi-Newton methods.Least-squares and full quasi-Newton methods were considered too expensive for large neural networks, motivating more efficient variants.
Complexity and generalization: Researchers sought low-complexity networks with strong generalization using weight decay, validation-based penalties, flat-minimum search, and searches biased toward short, fast programs.Weight decay penalizes large weights, while flat-minimum search favors broad low-error regions in weight space.
Unsupervised learning: Unsupervised-learning methods frequently optimized entropy- or information-theoretic objectives to uncover and disentangle hidden sources underlying observed signals.The surveyed methods operated without teacher-provided labels.

5.7 1987: UL Through Autoencoder (AE) Hierarchies (Compare Sec. 5.15)

Ballard (1987) proposed stacking unsupervised autoencoders so higher-level networks learned from lower-level hidden codes, anticipating later unsupervised pre-training methods. The experiment found faster learning than backpropagation through an equivalent deeper autoencoder, although the task did not clearly require depth.

1987: UL Through Autoencoder (AE) Hierarchies: Ballard (1987) proposed hierarchical unsupervised autoencoders in which each higher-level autoencoder learned from the previous level’s hidden-layer codes.The lowest-level autoencoder reconstructed inputs, and its codes were fed into a higher-level autoencoder of the same type.
1987: UL Through Autoencoder (AE) Hierarchies: The AE-stack experiment learned a mapping faster than backpropagation through an equivalent deeper autoencoder without pre-training.It used an AE-specific learning algorithm rather than traditional backpropagation.
1987: UL Through Autoencoder (AE) Hierarchies: However, the experiment’s task did not really require a deep autoencoder, so the benefits of unsupervised pre-training were not obvious.The proposed motivation was that hidden-layer codes would facilitate subsequent learning.
1987: UL Through Autoencoder (AE) Hierarchies: The same stacking principle could in principle be applied to other unsupervised methods, including history-compressing RNNs, RBMs, and hierarchical Kohonen networks, before subsequent supervised learning.The passage also relates this idea to competitive unsupervised pre-training followed by backpropagation-based fine-tuning.

5.8 1989: BP for Convolutional NNs (CNNs, Sec. 5.4) · 5.9 1991: Fundamental Deep Learning Problem of Gradient Descent · 5.10 1991: UL-Based History Compression Through a Deep Stack of RNNs

The 1989 combination of backpropagation, convolutional weight sharing, adaptive connections, and max-pooling became foundational for modern visual deep learners. In 1991, vanishing or exploding gradients were identified as a fundamental training problem, while unsupervised RNN pre-training enabled credit assignment across hundreds of nonlinear operators through history compression.

5.8 1989: BP for Convolutional NNs (CNNs, Sec. 5.4): In 1989, backpropagation was applied to Neocognitron-like convolutional layers with weight sharing and adaptive connections, augmented by max-pooling.This combination became an essential ingredient of many modern, competition-winning feedforward visual deep learners and introduced MNIST.
5.9 1991: Fundamental Deep Learning Problem of Gradient Descent: Hochreiter’s 1991 thesis formally identified vanishing or exploding gradients as a major reason traditional deep feedforward and recurrent networks are hard to train by backpropagation.With standard activation functions, cumulative backpropagated error signals can become problematic in typical deep networks.
5.9 1991: Fundamental Deep Learning Problem of Gradient Descent: Unsupervised pre-training through a hierarchy of RNNs alleviates the gradient problem and greatly facilitates subsequent supervised credit assignment through backpropagation.Conceptually related effects can be achieved for feedforward networks through autoencoder stacks and Deep Belief Networks.
5.9 1991: Fundamental Deep Learning Problem of Gradient Descent: LSTM-like networks alleviate the gradient problem through a special architecture unaffected by it.This approach differs from methods that primarily improve optimization or computational reach.
5.9 1991: Fundamental Deep Learning Problem of Gradient Descent: Today’s GPU-based computers have a million times the computational power of early-1990s desktop machines, allowing traditional networks to propagate errors a few layers farther within reasonable time.The passage says this underlies many current image-recognition competition wins, although it does not fundamentally overcome the gradient problem.
5.9 1991: Fundamental Deep Learning Problem of Gradient Descent: Hessian-free optimization can alleviate the gradient problem for feedforward and recurrent networks, while gradient-free weight-matrix search avoids the problem altogether.The survey also describes random weight guessing, Universal Search for weight-matrix-computing programs, and linear methods for output-event connections.
5.10 1991: UL-Based History Compression Through a Deep Stack of RNNs: The 1991 Very Deep Learner performed credit assignment across hundreds of nonlinear operators by unsupervised pre-training a hierarchy of RNNs.Each RNN predicts its next input; only unexpected inputs are passed upward, compressing information onto progressively slower self-organizing time scales without information loss.
5.10 1991: UL-Based History Compression Through a Deep Stack of RNNs: A lower-level RNN could imitate a trained higher-level RNN’s hidden units, developing slowly changing memories that bridge very long time lags and partially collapse the stack.The 1991 system was a working modern-style Deep Learner and a first Neural Hierarchical Temporal Memory, using sequence-processing RNNs rather than feedforward networks.

5.11 1992: Max-Pooling (MP): Towards MPCNNs (Compare Sec. 5.16, 5.19) · 5.12 1994: Early Contest-Winning NNs · 5.13 1995: Supervised Recurrent Very Deep Learner (LSTM RNN)

These sections trace max-pooling from the Cresceptron to backpropagation-trained MPCNNs, note early neural-network contest success, and present supervised LSTM RNNs as learners that overcome vanishing or exploding errors on difficult sequential tasks. LSTM subsequently matched or surpassed HMM-based systems, supported diverse applications and metalearning, and achieved competition successes despite not being universally superior.

5.11 1992: Max-Pooling (MP): Towards MPCNNs (Compare Sec. 5.16, 5.19): Max-pooling partitions a two-dimensional activation array into rectangular regions and replaces each region with its maximally active unit’s activation.The Cresceptron used max-pooling instead of alternative local subsampling or winner-take-all methods.
5.11 1992: Max-Pooling (MP): Towards MPCNNs (Compare Sec. 5.16, 5.19): Convolutional networks combined with max-pooling form MPCNNs with alternating convolutional and max-pooling layers, trained by backpropagation unlike the Cresceptron and HMAX.Backpropagation-trained MPCNNs became central to many modern competition-winning feedforward visual deep learners.
5.12 1994: Early Contest-Winning NNs: 1990s neural networks already won controlled pattern-recognition contests with secret test sets, including an internal-delay-line network winning the Santa Fe time-series competition.The competition concerned chaotic intensity pulsations of an NH3 laser, and no very deep credit-assignment paths were needed.
5.13 1995: Supervised Recurrent Very Deep Learner (LSTM RNN): Supervised LSTM RNNs eventually performed feats similar to the 1991 deep RNN hierarchy and overcame the Fundamental Deep Learning Problem without unsupervised pre-training.LSTM also learned tasks without local sequence predictability, which the partially unsupervised 1991 History Compressor could not learn.
5.13 1995: Supervised Recurrent Very Deep Learner (LSTM RNN): LSTM’s Constant Error Carousels use identity activation and fixed self-weight 1.0, preventing backpropagated errors from vanishing or exploding within the carousel.Because the identity function’s derivative is 1.0, errors remain unchanged unless they flow out to other adaptive network parts.
5.13 1995: Supervised Recurrent Very Deep Learner (LSTM RNN): LSTM learned previously unlearnable tasks involving long-range temporal order, high-precision storage, continuous-stream arithmetic, temporal distances, and noisy extended patterns.It also handled context-free languages requiring the functional equivalent of a runtime stack.
5.13 1995: Supervised Recurrent Very Deep Learner (LSTM RNN): In 2003, LSTM trained from scratch on TIDIG-ITS utterances achieved results comparable to HMM-based systems, and in 2007 it outperformed HMMs in keyword spotting.Early-2000s speech recognition was dominated by HMMs combined with feedforward neural networks.
5.13 1995: Supervised Recurrent Very Deep Learner (LSTM RNN): LSTM improved results across robot localization and control, driver-distraction detection, protein analysis, handwriting recognition, and voice activity detection, while also enabling an LSTM metalearner for quadratic functions.LSTM RNNs later won international pattern-recognition competitions and set benchmark records, although other methods sometimes outperformed gradient-based LSTM on particular tasks.

5.14 2003: More Contest-Winning/Record-Setting NNs; Successful Deep NNs · 5.15 2006/7: UL For Deep Belief Networks / AE Stacks Fine-Tuned by BP · 5.16 2006/7: Improved CNNs / GPU-CNNs / BP for MPCNNs / LSTM Stacks

From 2003 to 2007, neural networks achieved contest and benchmark successes, while unsupervised pre-training, improved CNN architectures, GPU implementations, and LSTM stacks advanced deep learning. These results helped establish deep networks across image, speech, document retrieval, and sequence-labelling tasks.

5.14 2003: More Contest-Winning/Record-Setting NNs; Successful Deep NNs: A Bayes neural-network ensemble won the NIPS 2003 Feature Selection Challenge, although its two hidden layers gave it shallow CAPs of depth 3.The result demonstrated contest success despite limited depth.
5.14 2003: More Contest-Winning/Record-Setting NNs; Successful Deep NNs: 0.4% was the MNIST error rate of a BP-trained CNN using training pattern deformations without unsupervised pre-training, versus 0.7% for a standard BP net.The corresponding CAP depth was low.
5.14 2003: More Contest-Winning/Record-Setting NNs; Successful Deep NNs: Deep LSTM RNNs produced early speech-recognition results comparable to HMM-based systems, while recurrent feedback improved image interpretation and FNNs with CAP depth up to 6 classified high-dimensional data.These findings extended successful deep-network applications beyond image benchmarks.
5.15 2006/7: UL For Deep Belief Networks / AE Stacks Fine-Tuned by BP: Around 2006, unsupervised pre-training of deep FNNs accelerated subsequent BP-based supervised learning, and DBNs emerged as stacks of unsupervisedly trained RBMs.Autoencoder stacks became a popular alternative for unsupervised pre-training before BP fine-tuning.
5.15 2006/7: UL For Deep Belief Networks / AE Stacks Fine-Tuned by BP: 1.2% was the MNIST error rate of a DBN fine-tuned by BP without training pattern deformations, helping arouse interest in DBNs.DBNs also achieved a 26.7% error rate on the TIMIT core test set.
5.15 2006/7: UL For Deep Belief Networks / AE Stacks Fine-Tuned by BP: Semantic Hashing mapped semantically similar variable-size documents to nearby representation addresses and outperformed previous similar-document searchers such as Locality Sensitive Hashing.This illustrated DBN-based methods beyond classification benchmarks.
5.16 2006/7: Improved CNNs / GPU-CNNs / BP for MPCNNs / LSTM Stacks: 4 times faster than CPU-CNNs was the reported speed of an early GPU-based CNN implementation, foreshadowing GPUs’ growing importance for deep learning.Earlier GPU implementations of standard FNNs reported a speed-up factor of 20.
5.16 2006/7: Improved CNNs / GPU-CNNs / BP for MPCNNs / LSTM Stacks: In 2007, BP was first applied to Neocognitron-inspired MPCNNs with alternating convolutional and max-pooling layers, while hierarchical LSTM stacks outperformed HMMs on spoken-digit recognition.Hierarchical CTC back-propagated errors through lower LSTM levels for sequence labelling.

5.17 2009: First Official Competitions Won by RNNs, and with MPCNNs · 5.18 2010: Plain Backprop (+ Distortions) on GPU Breaks MNIST Record · 5.19 2011: MPCNNs on GPU Achieve Superhuman Vision Performance

From 2009 to 2011, deep RNNs and GPU-based CNNs won official contests, while plain backpropagation with distortions set a 0.35% MNIST error record. GPU-MPCNN ensembles then reached human-competitive or better performance in restricted visual benchmarks, despite test-set probing concerns.

5.17 2009: First Official Competitions Won by RNNs, and with MPCNNs: Deep LSTM RNNs won three ICDAR 2009 handwriting competitions across French, Arabic, and Farsi without a priori linguistic knowledge.They simultaneously performed segmentation and recognition on secret test sets.
5.17 2009: First Official Competitions Won by RNNs, and with MPCNNs: A 3-dimensional CNN combined with SVMs helped win three 2009 TRECVID surveillance-video competitions, among the first official contests won with CNNs.The larger system used bag-of-features region-of-interest extraction.
5.17 2009: First Official Competitions Won by RNNs, and with MPCNNs: A Convolutional DBN combined CNN and DBN ideas and was successfully applied to audio classification.It used a probabilistic variant of max-pooling.
5.18 2010: Plain Backprop (+ Distortions) on GPU Breaks MNIST Record: 0.35% MNIST error rate was achieved in 2010 by deep standard neural networks using backpropagation, training-pattern deformations, and GPU acceleration.The method used neither unsupervised pre-training nor convolution, while GPU backpropagation was up to 50 times faster.
5.18 2010: Plain Backprop (+ Distortions) on GPU Breaks MNIST Record: The 2010 result suggested that exploiting modern computing hardware mattered more than algorithmic advances, given the age of backpropagation and pattern deformations.Backpropagation was 3–5 decades old, and pattern deformations were 2 decades old.
5.19 2011: MPCNNs on GPU Achieve Superhuman Vision Performance: GPU-MPCNNs alternated convolutional and max-pooling layers, while multi-column systems averaged outputs democratically to classify inputs.Several MPCNNs processed the same input and the highest average class probability determined the classification.
5.19 2011: MPCNNs on GPU Achieve Superhuman Vision Performance: 1.02% qualifying-round error beat 1.03% for second place, while human performance was 1.19%; repeated submissions could incrementally reveal test-set information.The results appeared human-competitive, but probing enabled potential overfitting to the evaluation set.
5.19 2011: MPCNNs on GPU Achieve Superhuman Vision Performance: Around 0.2% MNIST performance was human-competitive, improving on a record that had remained around 0.4% for almost a decade.GPU-MPCNNs were commercially relevant through efficient coding rather than scientific novelty and became common among feedforward competition winners.

5.20 2011: Hessian-Free Optimization for RNNs · 5.21 2012: First Contests Won on ImageNet, Object Detection, Segmentation · 5.22 2013-: More Contests and Benchmark Records

From 2011 onward, deep learning advanced through Hessian-free optimization for RNNs and a succession of winning image-recognition, detection, segmentation, speech, and other benchmark contests. These results included large-scale unsupervised ImageNet learning, biomedical applications, and records across vision and sequence-processing tasks.

5.20 2011: Hessian-Free Optimization for RNNs: Hessian-free optimization alleviated the Fundamental Deep Learning Problem in RNNs and outperformed standard gradient-based LSTM RNNs on several tasks.The result was reported in 2011 and builds on Hessian-free optimization methods.
5.21 2012: First Contests Won on ImageNet, Object Detection, Segmentation: 2012: An ensemble of GPU-MPCNNs achieved the best ImageNet classification results, using 256x256-pixel images rather than 48x48-pixel traffic-sign images.This marked a major computer-vision benchmark result for deep learning.
5.21 2012: First Contests Won on ImageNet, Object Detection, Segmentation: 109 free parameters: An unsupervised neural network trained on unlabeled data achieved the best results so far on 20,000 ImageNet classes.Training used 1,000 standard machines with 16,000 cores, after which top-layer codes trained a simple supervised classifier.
5.21 2012: First Contests Won on ImageNet, Object Detection, Segmentation: 2012: A GPU-MPCNN ensemble became the first deep-learning system to win a visual object-detection contest on large biomedical images.It won the ICPR 2012 Contest on Mitosis Detection in Breast Cancer Histological Images.
5.21 2012: First Contests Won on ImageNet, Object Detection, Segmentation: 2012: A GPU-MPCNN ensemble won the first pure image-segmentation contest, while fast MPCNN scans accelerated naive implementations by up to three orders of magnitude.The segmentation challenge concerned reconstructing neuronal structures from electron-microscopy image stacks.
5.22 2013-: More Contests and Benchmark Records: 17.7% test set error rate: A stack of bi-directional LSTM RNNs trained by CTC broke a famous TIMIT phoneme-recognition record.This followed thousands of man-years of HMM-based speech-recognition research.
5.22 2013-: More Contests and Benchmark Records: LSTM systems won or set benchmark records in handwriting, language identification, speech recognition, prosody prediction, audio onset detection, translation, and other sequence tasks.CTC-LSTM scored first at NIST’s OpenHaRT2013 evaluation, while LSTM-based systems beat previous state of the art in large-vocabulary speech recognition and English-to-French translation.
5.22 2013-: More Contests and Benchmark Records: Deep GPU-MPCNNs achieved new best or state-of-the-art results across ImageNet, PASCAL object detection, localization, Chinese handwriting, mitosis detection, and tibial-cartilage segmentation.Three 2D-CNNs using orthogonal projections outperformed traditional full 3D methods on low-field knee MRI scans, and a GPU-MPCNN ensemble won the MICCAI 2013 mitosis challenge.

5.23 Currently Successful Techniques: LSTM RNNs and GPU-MPCNNs … 5.26 DL with Spiking Neurons?

The surveyed successes center on supervised LSTM RNNs and GPU-MPCNNs, while newer training techniques improve deep networks. The discussion also connects artificial and biological neural networks and contrasts GPU efficiency with spike-based neural communication.

5.23 Currently Successful Techniques: LSTM RNNs and GPU-MPCNNs: Most competition-winning or benchmark-record-setting deep learners use either LSTM RNNs trained by CTC or feedforward GPU-MPCNNs trained through BP.GPU-MPCNNs combine CNNs with max pooling, while LSTM RNNs represent the recurrent alternative.
5.23 Currently Successful Techniques: LSTM RNNs and GPU-MPCNNs: Deep GPU-MPCNNs support purely supervised transfer learning, with pre-training on one training set greatly improving performance on quite different datasets.The passage contrasts this capability with exceptions based on transfer learning between datasets.
5.23 Currently Successful Techniques: LSTM RNNs and GPU-MPCNNs: Training trends moved from partially unsupervised RNN and FNN stacks toward purely supervised LSTM RNNs and MPCNNs, although unsupervised pre-training can remain advantageous.The passage describes this as a shift across the 1990s and 2000s and recommends combining supervised learning with unsupervised pre-training in some applications.
5.24 Recent Tricks for Improving SL Deep NNs (Compare Sec. 5.6.2, 5.6.3): Recent improvements include gradient enhancements, automatic learning-rate adjustment, RBM regularization, contractive and denoising autoencoders, dropout, ReLUs, Maxout, and RmsProp.These methods respectively target optimization, robustness, generalization, activation quality, competitive interactions, and gradient-descent speed.
5.25 Consequences for Neuroscience: Artificial neural networks can help explain biological neural networks, whose early visual feature detectors resemble those learned by single-layer visual ANNs.The passage further proposes that deep visual ANN features may predict discoveries in deeper biological visual layers.
5.25 Consequences for Neuroscience: Deep visual ANN feature detectors may predict those found in deep biological visual layers, despite potentially different learning algorithms and similar objective functions.This claim extends the stated similarity between learned ANN features and early biological visual processing.
5.26 DL with Spiking Neurons?: Many recent deep-learning results rely on GPU-based traditional deep networks, but current GPUs consume far more energy than biological brains.Biological neurons communicate efficiently through brief spikes and often remain quiet.
5.26 DL with Spiking Neurons?: Spiking-neuron models have been proposed and analyzed as computational models of the brief, energy-efficient signaling used by biological neurons.The passage frames spike-based computation as a contrast to energy-hungry GPU implementations.

6 DL in FNNs and RNNs for Reinforcement Learning (RL)

The section surveys deep feedforward and recurrent neural networks for reinforcement learning, where agents must learn to act from delayed reward signals in initially unknown environments. It contrasts model-based deep approaches and recurrent solutions for partial observability with Markovian dynamic-programming methods that reduce credit-assignment depth.

RL setting: Reinforcement-learning agents must maximize expected cumulative reward from occasional pain and pleasure signals despite unknown environments and arbitrary delays between actions and consequences.The passage characterizes reinforcement learning as broadly expressive because any computable task can be formulated in this framework.
Model-based RL: In deterministic, predictable environments, a separate feedforward network can learn a world model that predicts controller inputs from previous actions and inputs.The controller and model can thereby form a model-based RL system rather than relying only on direct policy learning.
Model-based RL: Model-based planning creates deep credit-assignment paths, and it enabled learned truck backing, sequential visual saccades, and selective-attention control.The active-vision system learned sequential foveal shifts to detect targets in visual scenes.
Model-based RL: Learned world models can optimize expected reward by planning future action sequences, including the 2004 RoboCup fast-league winning approach for four-motor robots.During play, the models optimized action sequences to achieve desirable subgoals through quickly planning ahead.
Markov RL: Under the Markov assumption, dynamic programming reduces credit-assignment depth because the current input contains all information needed to compute the optimal next decision, so a feedforward policy network suffices.This contrasts with settings requiring memory of prior events.
POMDPs and RNNs: Because partial observability makes the Markov assumption unrealistic, recurrent networks can map event histories to values or combine recurrent controllers with predictive world models, although POMDPs may greatly increase credit-assignment depth.Deep LSTM RNNs were used as value functions for RL robots, while combined controller–model RNNs yield deep credit-assignment paths.

7 Conclusion and Outlook

The conclusion presents deep learning in neural networks as relevant to supervised, unsupervised, and reinforcement learning, with dynamic programming supporting deep supervised learning and traditional reinforcement learning. It forecasts attention-selective, energy-efficient networks and, further ahead, self-improving algorithms that remain impractical today.

Conclusion and Outlook: Deep learning in neural networks is relevant to supervised, unsupervised, and reinforcement learning, while dynamic programming supports deep supervised learning and traditional reinforcement learning with deep networks.Unsupervised learning can also facilitate supervised sequence and stationary-pattern learning and reinforcement learning by alleviating deep credit-assignment problems.
Conclusion and Outlook: Future deep neural networks are expected to learn selective attention through reinforcement learning of motor and internal actions, closing the general sensorimotor loop through external and internal feedback.The passage connects this outlook to prior work since 1990 on saccade control and attention spotlights within recurrent neural networks.
Conclusion and Outlook: Future deep neural networks will account for the energy costs of activating neurons and transmitting signals, using local competition so only a small fraction are active at a given time.Brains appear to minimize computational costs through winner-take-all mechanisms that shut down neighbouring neurons.
Conclusion and Outlook: Energy minimization can favor simple, highly generalizing solutions that require few active neurons and few, mostly short connections.This passage describes such solutions as a by-product of computational-cost minimization.
Conclusion and Outlook: General-purpose learning algorithms that improve themselves in provably optimal ways may belong to the more distant future, but they are not yet practical or commercially relevant.The conclusion explicitly places this possibility in the distant future and qualifies its current status.