On the Origin of Deep Learning

Motivation

• Deep learning has improved state-of-the-art results in object detection, speech recognition, and machine translation while enabling applications across diverse modern domains.Examples include speech denoising, gene-expression clustering, style-based image generation, and multimodal sentiment analysis.Source: §1 · p. 2

Contribution

• The paper traces major deep learning milestones and explains the ideas behind current models and their relationships to ancestral approaches.It presents a historical walkthrough because deep learning evolved over a long period, with milestones summarized in Table 1.Source: §1 · p. 2, Table 1 · p. 3

Historical progression

• The review begins with brain modeling and connectionism, then follows the maturation of shallow neural networks and the motivation, promises, and challenges of deeper architectures.The paper notes that modern success depends more on deep architecture than necessarily on resemblance to the human brain.Source: §1 · p. 2

Major model families

• After establishing deep neural networks, the paper reviews Deep Belief Nets and Restricted Boltzmann Machines, Convolutional Neural Networks, and Recurrent Neural Networks with successors including LSTM and attention models.It also summarizes optimization techniques such as advanced gradient methods, Dropout, and Batch Normalization.Source: §1 · p. 2

Related work

• Unlike Schmidhuber’s work, which credits incremental contributions, this paper emphasizes milestone ideas and associations to provide background on model development, including recent work derived from classical linear models.It is also complementary to the historical interviews in.Source: §1 · pp. 2, 4

2.1 Associationism

• Associationism, originating with Aristotle around 300 B.C. (Burnham, 1888), models mind as conceptual elements connected by associations governed by contiguity, frequency, similarity, and contrast.These principles remain foundational assumptions in machine-learning methods involving proximity, co-occurrence, and similarity or dissimilarity in latent representations.Source: §2.1 · p. 5

2.2 Bain and Neural Groupings

• Bain’s neural groupings represented associative memory through linked cells whose connections strengthened or weakened with temporally contiguous experience.His constructive storage model and cell-grouping structure anticipated modern neural networks, although Bain doubted the theory’s practical value.Source: §2.2 · pp. 6–7

2.3 Hebbian Learning Rule

• Hebb’s rule, introduced in The Organization of Behavior, strengthened connections when neurons repeatedly co-fired, but caused unstable, potentially exponential weight growth.This instability was identified as a central drawback of the foundational learning rule.Source: §2.3 · p. 7

2.4 Oja’s Rule and Principal Component Analyzer

• Oja’s rule added normalization to Hebbian learning and showed that a neuron’s update performs an online form of Principal Component Analysis.The rule links learned weights to eigenvectors of the input covariance matrix, with the first component corresponding to the largest eigenvalue.Source: §2.4 · p. 8

2.5 MCP Neural Model

• The MCP model introduced by McCulloch and Pitts (1943) modeled neurons as thresholded weighted circuits with inhibitory inputs, but its weights were fixed and manually assigned.An inhibitory input could completely prevent excitation, distinguishing the model from modern adjustable-weight perceptrons.Source: §2.5 · p. 9

2.6 Perceptron

• Rosenblatt’s perceptron transformed associative principles into a learnable vision-inspired device that combines inputs, thresholds their weighted sum, and produces responses.Omitting the projection layer yields the structure of modern perceptrons, while later nonlinear activations softened biological thresholding and improved computational feasibility.Source: §2.6 · pp. 9–10

2.7 Perceptron’s Linear Representation Power

• A single perceptron represents only linear decision boundaries, supporting NOT, AND, and OR but not XOR or NXOR.For two inputs, XOR cannot be described by one linear boundary, motivating neural networks with multiple perceptrons while not guaranteeing greater power from arbitrary depth.Source: §2.7 · pp. 10–11

3.1 Universal Approximation Property

• An MLP with one hidden layer can exactly represent any Boolean function and approximate any bounded continuous function arbitrarily accurately, while two hidden layers suffice for arbitrary functions.These are presented as three universal approximation properties, with the arbitrary-function result allowing discontinuous targets but potentially requiring many neurons.Source: §3.1 · p. 12, §3.1.3 · p. 14

3.1.1 Representation of any Boolean Functions

• Boolean functions are represented by rewriting them in conjunctive normal form, using one layer for AND operations and another for their OR combination.The construction follows from perceptrons implementing AND or OR and De Morgan’s laws.Source: §3.1.1 · p. 12

3.1.2 Approximation of any Bounded Continuous Functions

• One hidden layer bounds complex bounded continuous functions through many linear perceptrons, but the result originally assumed squashing activations and required a later correction for squashed outputs.The original formulation produced a linear output rather than the widely accepted thresholded or squashed output; the property nevertheless holds when the final output is squashed.Source: §3.1.2 · p. 13

3.1.3 Approximation of Arbitrary Functions

• Two hidden layers can approximate arbitrary functions by combining many simple surfaces, although exact representation is not guaranteed and the construction may require exponentially many neurons.The first hidden layer creates surfaces such as localized bumps, while the second combines them; shallow universal approximation therefore has substantial computational cost.Source: §3.1.3 · pp. 14–15

3.2 The Necessity of Depth

• Depth can provide exponentially greater representation efficiency: functions requiring polynomially many neurons at depth k may require exponentially many at depth k−1, supporting deeper architectures.Related analyses find polynomial growth of representation power for shallow networks versus exponential growth for deep networks, and Eldan and Shamir (2015) show depth is exponentially more valuable than width under weak assumptions.Source: §3.2 · p. 16

3.3 Backpropagation and Its Properties

• Backpropagation efficiently computes parameter gradients by propagating output error backward and applying gradient descent, while its modern usage commonly names the combined procedure.The derivation organizes chain-rule terms to obtain gradients for parameter updates.Source: §3.3 · p. 17

3.3.1 Backpropagation Finds Global Optimal for Linear Separable Data

• Backpropagation can guarantee a global optimum for linearly separable data under pyramidal architecture, full-row-rank weight matrices, and sufficient input neurons, but those assumptions limit general relevance.The guarantee depends on the data being linearly separable, a condition under which other models may also apply.Source: §3.3.1 · p. 17

3.3.2 Backpropagation Fails for Linear Separable Data

• Backpropagation can fail to find a separating boundary even for linearly separable data, especially when misclassified samples are outliers; later work links such behavior to saddle points.The outlier-related behavior may be desirable for robustness, but the inability to overfit training data motivates improved optimization techniques.Source: §3.3.2 · pp. 17–18

• The section follows the branch from unsupervised data modeling to Restricted Boltzmann Machines and Deep Belief Nets.Source: §4 · p. 19
• Figure 7 organizes six milestones by computation complexity on the horizontal axis and representation power on the vertical axis.Source: §4 · p. 19, Figure 7 · p. 19

4.1 Self Organizing Map

• SOM reduces data usually to one or two dimensions while retaining topological similarity, making it useful for clustering with an imposed topology.It learns a low-dimensional representation whose unit positions encode relationships among data points.Source: §4.1 · pp. 19–20
• SOM unit positions matter because similarity preservation constrains representations, unlike one-hot vectors whose coordinate assignments can be arbitrary.For colors, green cannot split the more similar red and purple when represented on a one-dimensional SOM.Source: §4.1 · p. 20
• A trained SOM supports intuitive visualization by positioning related multidimensional observations nearby, enabling rankings such as countries’ happiness and richness to be read at a glance.In the example, corner units represent combinations of highest or lowest happiness and richness, with other countries positioned accordingly.Source: §4.1 · pp. 20–21

4.1.1 Learning Algorithm

• The classic SOM algorithm samples a data vector, finds its best matching unit, selects neighboring units, and updates their weights using distance-based penalties.The learning rate l(t) and neighborhood radius r(t) typically decrease as iteration t increases; Euclidean distance measures data-vector similarity.Source: §4.1.1 · p. 21
• SOM training requires careful neighbor-radius selection and weight initialization; otherwise, local topological similarity may fail globally by separating similar clusters.As described by Germano (1999), green units may separate red and purple units when the network is inadequately trained.Source: §4.1.1 · p. 21

4.2 Hopfield Network

• Hopfield Network is a recurrent3 neural network with bidirectional weights and content-addressable memory, historically introduced by Hopfield (1982).Here, “recurrent” refers to bidirectional connections, rather than the modern usage associated with recurrent neural networks.Source: §4.2 · p. 21

4.2.1 Spin Glass

• Spin-glass dynamics converge toward stable states by iteratively changing dipole alignments until the system’s potential energy is minimized.This physical intuition motivates the Hopfield Network’s energy-based formulation.Source: §4.2.1 · p. 22

4.2.2 Hopfield Network

• The network uses fully connected binary thresholding units with bidirectional weights, and its energy function closely corresponds to spin-glass potential energy.The paper discusses unit values as 0 and 1, noting that other conventions change the energy formulation without changing the underlying idea.Source: §4.2.2 · pp. 22–23

4.2.3 Learning and Inference

• During inference, asynchronous updates invert one unit when energy decreases, whereas synchronous updates change all selected units simultaneously and can cause local optima, energy increases, or oscillations.Calculating weights is parameter learning or training; applying fixed weights to recover hidden states is inference or testing.Source: §4.2.3 · pp. 23–24

4.2.4 Capacity

• A network of N units stores only up to 0.15N2 bits, making Hopfield Network memory extremely inefficient despite its ability to memorize data.Follow-up work attempted to increase capacity, but Hopfield Network was eventually replaced by models inspired by its ideas.Source: §4.2.4 · p. 24

4.3.1 Boltzmann Distribution

• Boltzmann distributions assign state probabilities through energies, with state-probability ratios determined by energy differences and normalized across states.The energy-based formulation also defines the Boltzmann factor.Source: §4.3.1 · pp. 24–25

4.3 Boltzmann Machine

• Boltzmann Machines are stochastic Hopfield Networks with hidden units that separate visible data-connected units from hidden distribution-assisting units while retaining full connectivity.The hidden units conceptually divide the model into visible and hidden parts.Source: §4.3 · p. 24, §4.3.2 · p. 25

4.3.2 Boltzmann Machine

• Stochastic state transfers help Boltzmann Machines escape Hopfield local optima or oscillations, with higher temperatures increasing transfers and lower-energy moves favored; this section sets T = 1.The temperature mechanism is historically related to simulated annealing.Source: §4.3.2 · pp. 25–26

4.3.3 Energy of Boltzmann Machine

• The visible-hidden energy function defines joint probabilities, whose marginalization yields visible-unit distributions that can be sampled to generate data.The model reaches thermal equilibrium when the joint probability distribution remains constant, although individual units may continue varying.Source: §4.3.3 · p. 26

4.3.4 Parameter Learning

• Boltzmann Machines learn parameters by gradient ascent on observed-data log likelihood, but the model-expectation term requires MCMC sampling until equilibrium and can be computationally complicated.The data-driven expectation is generally straightforward, whereas the model-driven expectation requires averaging over Markov-chain samples.Source: §4.3.4 · pp. 26–27

4.4 Restricted Boltzmann Machine

• RBM, originally called Harmonium by Smolensky (1986), removes connections among visible units and among hidden units, producing a bipartite graph with a simpler energy function.This restriction makes sampling for inference easier than in the general Boltzmann Machine.Source: §4.4 · p. 27, Figure 11 · p. 28

4.4.1 Contrastive Divergence

• RBM retains Boltzmann Machine training, but estimating the model distribution still requires potentially infinitely many sampling steps.The parameter update contrasts data-distribution statistics p0 with model-distribution statistics p∞.Source: §4.4.1 · p. 28
• Contrastive Divergence, introduced by Hinton (2002), uses finite k-step sampling instead of waiting for convergence to the model distribution.Carreira-Perpinan and Hinton (2005) found finite k theoretically biased, but empirically effective because the resulting bias is small.Source: §4.4.1 · pp. 28–29
• Hinton (2002) showed that k = 1 is sufficient for the learning algorithm to work well in practice, helping RBM become a fundamental building block of deep neural networks.RBM combines reasonable modeling power with a fast approximation algorithm and underlies Deep Belief Nets and Deep Boltzmann Machines.Source: §4.4.1 · pp. 28–29

4.5 Deep Belief Nets

• Hinton et al. (2006)8 introduced Deep Belief Networks by showing that RBMs could be stacked and trained greedily, opening the modern deep-learning era through layerwise training.The paper identifies this work as the first introduction of training a deep neural network by layerwise training.Source: §4.5 · p. 29
• Unlike a stack of RBMs, a DBN has bidirectional RBM connections only at the top layer and top-down connections below, forming a multilayer generative model.Generation requires sampling the topmost layer to thermal equilibrium and propagating visible states downward.Source: §4.5 · p. 29

4.5.1 Parameter Learning

• DBN parameter learning has two stages: layerwise pre-training followed by fine-tuning.Pre-training treats successive layer pairs as RBMs and trains them bottom up, providing critical support for later fine-tuning.Source: §4.5.1 · p. 30
• Pre-training initializes parameters for effective optimization and better generalization, inspiring pretraining methods for autoencoders, Deep Boltzmann Machines, and related models.Bengio et al. (2007) linked pre-training to lower empirical-cost minima, while Erhan et al. (2010) emphasized regularization and generalization.Source: §4.5.1 · p. 30
• Fine-tuning uses a contrastive wake-sleep procedure for generative DBNs and standard backpropagation for discriminative DBNs.The generative procedure alternates wake and sleep phases with Contrastive Divergence sampling at the top RBM; pre-training remains necessary despite backpropagation’s performance.Source: §4.5.1 · pp. 30–31

4.6 Deep Boltzmann Machine

• Deep Boltzmann Machines extend Restricted Boltzmann Machines with bidirectional connections between bottom layers, better representing stacked RBMs than Deep Belief Networks.Figure 13 illustrates a three-layer DBM.Source: §4.6 · p. 31
• DBMs extend the RBM energy function to N hidden layers, enabling contrastive-divergence training, although pre-training is typically necessary.Source: §4.6 · p. 31

4.6.1 Deep Boltzmann Machine (DBM) v.s. Deep Belief Networks (DBN)

• DBMs and DBNs both originate from Restricted Boltzmann Machines and rely on layerwise pre-training for successful parameter learning.The name “Deep Belief Network” also seems to indicate partial origins in Bayesian Network.Source: §4.6.1 · pp. 31–32
• Unlike DBNs’ directed bottom-layer connections, DBMs’ bidirectional structure learns more complex patterns and incorporates top-down feedback to propagate uncertainty about ambiguous inputs.Source: §4.6.1 · p. 32

• The Deep Boltzmann Machine concludes the paper’s historical discussion of generative models, while substantial future work remains.The authors explicitly frame DBM as the last milestone discussed, not the endpoint of generative-model research.Source: §4.7 · p. 32
• Lake et al. (2015) developed Bayesian Program Learning that achieved one-shot visual learning and human-indistinguishable handwritten-character generation, outperforming several concurrent deep networks.Although not itself a deep neural model, the framework passed a visual Turing Test; a deep counterpart is anticipated to perform even better.Source: §4.7 · p. 32
• Conditional image generation is commonly addressed with Pixel CNN (van den Oord et al., 2016) and Pixel RNN (Oord et al., 2016), although conditioning simplifies generation.Source: §4.7 · p. 32
• Generative Adversarial Networks represent another generative-model contribution but were considered too young for detailed discussion.Source: §4.7 · p. 32

5.1 Visual Cortex

• CNNs are grounded in the visual cortex’s hierarchy, from V1 edge detection through V2 and V4 feature processing to inferior temporal semantic recognition.The section focuses on the ventral stream because it is associated with object recognition and complex feature representations.Source: §5.1 · pp. 33–34

5.2 Neocogitron and Visual Cortex

• Neocognitron established the computational template of feature-extracting S-layers and shift-invariant C-layers, with lower layers detecting local features and higher layers capturing global ones.Its S-layer learning resembles Hebbian learning, while inhibitory cells normalize connection strengths to prevent saturation.Source: §5.2 · p. 35, §5.2.1 · p. 35

5.3.1 Convolution Operation

• Convolution applies a sliding kernel through local element-wise products and sums, shrinking representations while providing local shift invariance that supports robust visual recognition.Different kernels can implement operations such as edge detection, blur, sharpening, or identity transformations.Source: §5.3.1 · p. 36, §5.3.2 · p. 37

5.3.2 Connection between CNN and Visual Cortex

• Learned convolutional kernels reproduce visual-cortex functions such as edge detection, making parameter learning central to CNN performance and enabling well-trained kernels to transfer across tasks with limited fine-tuning.This connection links convolutional processing to biologically inspired vision rather than treating kernels as arbitrary image transformations.Source: §5.3.2 · pp. 37–38

5.4 The Pioneer of Convolutional Neural Networks: LeNet

• LeNet (Le Cun et al. (1990); LeCun et al. (1998a)) assembled convolutional and subsampling layers with fully connected and RBF classification layers, handling digit variations in position, scale, rotation, squeezing, and stroke width.LeCun et al. (1998b) also introduced MNIST, which became a standard digit-recognition benchmark.Source: §5.4 · p. 38, §5.4.3 · p. 39

5.4.1 Convolutional Layer

• Nonlinear transforms such as ReLU remove negative activations to clarify meaningful features, while pooling reduces representations and provides invariance to small image distortions and translations.Pooling may use maximum, average, or probabilistic sampling, and its local aggregation makes small input changes less likely to alter the outcome.Source: §5.4.1 · p. 38, §5.4.2 · p. 39

5.5 Milestones in ImageNet Challenge

• AlexNet (Krizhevsky et al. (2012)) demonstrated that CNNs could perform strongly on ImageNet, using two-GPU training and data augmentation to make optimization computationally feasible.VGG pursued a simpler 19-layer design with repeated 3 × 3 convolutions and 2 × 2 pooling, while GoogLeNet won that year’s ImageNet competition.Source: §5.5.1 · p. 39, §5.5.2 · p. 40

5.5.3 Residual Net

• ResNet enabled substantially deeper CNNs through residual blocks that add an identity path, f(x) + x, improving information propagation and optimization; a 1000-layer version succeeded on CIFAR data.Residual networks can behave like ensembles of shallower networks, helping explain gradient propagation through ultra-deep architectures, although fully connected layers remain inefficient for small image batches.Source: §5.5.3 · pp. 41–42

5.6.1 Network Property and Vision Blindness Spot

• Imperceptible image perturbations can cause neural networks to misclassify recognizable objects, while meaningless patterns can be classified as objects with confidence above 99%, exposing fundamental differences from human vision (Szegedy et al., 2013).Figure 20 illustrates both adversarial images and generated patterns, including classifications as school bus, guitar, peacock, and Pekinese.Source: §5.6.1 · pp. 42–43, Figure 20 · p. 43
• These neural-network blind spots may limit reliable real-world prediction but also resemble blind spots in human vision, motivating research linking and overcoming defects in both systems.The paper relates such failures to changes within a matrix’s null space, which can alter images without affecting the corresponding matrix’s response.Source: §5.6.1 · pp. 42–43

5.6.2 Human Labeling Preference

• ResNet achieves a 3.6% ImageNet error rate versus a general human prediction error rate of 5%-10%, but many remaining errors arise from unexpected or inconsistent human labeling preferences.Figure 21 includes labels focused on tiny image regions or backgrounds rather than the image’s more important content.Source: §5.6.2 · p. 43, Figure 21 · p. 44
• Improving ImageNet scores may therefore require modeling annotators’ preferences, while broader datasets such as COCO, Flickr, and VisualGenome (Krishna et al., 2016) could create more competitive vision challenges.The section questions whether optimizing for ImageNet testing performance is worthwhile when annotation problems, rather than model incompetence, explain some misclassifications.Source: §5.6.2 · p. 44

6. Time Series Data and Recurrent Networks

• RNNs model time series through directed cycles, with recurrence defined by paths that return from a unit to itself.The section excludes Hopfield Networks from its RNN family discussion because their formalization differs from the modern definition.Source: §6 · p. 45, §6.1 · p. 45

6.1 Recurrent Neural Network: Jordan Network and Elman Network

• Jordan and Elman networks differ in whether recurrent information comes from previous outputs or hidden states, with Elman’s hidden-layer connections increasing later design flexibility.Because y_t = W_y h_t, the distinction can be expressed through the choice of recurrent weights W_r.Source: §6.1 · p. 45

6.1.1 Backpropagation through Time

• Backpropagation through Time unfolds an RNN across time into a feedforward network, enabling gradient computation but leaving optimization difficult because of complex cost surfaces and vanishing gradients.BPTT was independently invented by Robinson and Fallside (1987), Werbos (1988), and Mozer (1989).Source: §6.1.1 · p. 46

6.2 Bidirectional Recurrent Neural Network

• Bidirectional RNNs combine forward and reverse recurrent processing so later time steps can provide information to earlier ones.Training treats the model as standard and reverse RNNs, applying BPTT to each and updating weights after both gradients are computed.Source: §6.2 · p. 47

6.3 Long Short-Term Memory

• LSTM addresses RNN failures on long-term dependencies and vanishing gradients with a memory cell and gates controlling input, forgetting, and output information flow.Greff et al. (2015) found that tested LSTM variants did not significantly improve upon the standard LSTM architecture, motivating attention as another direction.Source: §6.3 · p. 48

6.4 Attention Models

• Attention models selectively localize informative encoded representations for each sequential prediction, typically within an encoder-decoder framework.Soft attention computes a weighted sum, whereas hard attention selects one region but is nondifferentiable for standard gradient-based training; variational or policy-gradient methods can address this limitation.Source: §6.4 · pp. 50–52

6.5.1 Deep Recurrent Neural Network

• Deep RNNs extend recurrence by adding layers to the input, recurrent, or output components, while broader designs combine RNNs with CRFs, CNNs, or convolutional LSTM operations.Pascanu et al. (2013a) formalized these three construction directions, and subsequent work assembled recurrent models with additional components.Source: §6.5.1 · p. 52, §6.5.2 · p. 53

6.5.2 The Future of RNNs

• Vanishing and exploding gradients remain a fundamental unresolved RNN problem because chain-rule backpropagation exponentially attenuates error signals across time, despite practical workarounds such as LSTM and ReLU.These methods bypass rather than fundamentally solve the general problem, leaving further work necessary.Source: §6.5.1 · p. 52, §6.5.2 · p. 53

7.1 Gradient Methods

• Neural-network optimization still relies heavily on backpropagation-based gradient descent, while modern variants generally promise faster convergence but often provide only limited practical gains.The review begins after classical, stochastic, batch, and momentum methods.Source: §7.1 · p. 54, §7.1.4 · p. 55

7.1.1 Rprop

• Rprop updates parameters using only gradient signs, enabling it to overcome some difficult cost curvatures that dominant methods may not solve easily.Its sign-based formulation is presented as a reason the older method may warrant renewed study.Source: §7.1.1 · p. 54

7.1.2 AdaGrad; 7.1.3 AdaDelta; 7.1.4 Adam

• AdaGrad adapts learning rates using historical gradient norms, whereas AdaDelta (Zeiler, 2012) reweights accumulated gradients to reduce AdaGrad’s decay; Adam combines momentum and AdaGrad.AdaGrad improves robustness over traditional gradient methods, but its learning-rate fraction decays substantially as accumulated norms grow; AdaDelta is nearly equivalent to RMSprop.Source: §7.1.2 · pp. 54–55, §7.1.3 · p. 55, §7.1.4 · p. 55

7.2 Dropout

• Dropout (Hinton et al., 2012; Srivastava et al., 2014) randomly omits hidden units during training and can approximate model averaging across many networks while reducing overfitting.Its simplicity and empirical performance made it influential and broadly studied.Source: §7.2 · pp. 55–56

7.3 Batch Normalization and Layer Normalization

• Batch Normalization (Ioffe and Szegedy, 2015) addresses shifting layer-input distributions by normalizing each batch and worked well empirically, while Layer Normalization (Ba et al., 2016) suits recurrent networks but was less influential.Layer Normalization computes statistics across a layer’s summed inputs for each training case, but is less simple to implement.Source: §7.3 · p. 56

7.4 Optimization for “Optimal” Model Architecture

• Constructive architecture methods revisit older approaches that grow networks or learn architecture structure automatically, motivated by modern computational resources but presented mainly as inspiration for optimization research.The paper notes that these methods often combine optimization with specially designed architectures and can trace to non-parametric machine-learning counterparts.Source: §7.4 · pp. 56–57

7.4.1 Cascade-Correlation Learning; 7.4.2 Tiling Algorithm

• Cascade-Correlation Learning grows a minimal network by adding hidden units while fixing earlier parameters, whereas Tiling Algorithm learns layers and hidden-unit counts through clustering with theoretical guarantees of greedy improvement.Cascade-Correlation can grow deeper and wider, adding units until existing architecture changes little and stopping when performance is satisfactory; this may cause overfitting.Source: §7.4.1 · p. 57, §7.4.2 · p. 57

7.4.3 Upstart Algorithm; 7.4.4 Evolutionary Algorithm

• Upstart replaces decision-tree nodes with linear perceptrons, while evolutionary algorithms encode architectures as vectors for optimization; these approaches largely faded because greedy search and overfitting hinder architecture discovery.The review nevertheless suggests that improved machine-learning methods and computation may make constructive approaches useful again.Source: §7.4.3 · p. 58, §7.4.4 · p. 58

• The review revisits evolutionary paths for deep generative, convolutional, and recurrent neural network families, together with optimization techniques.Source: §8 · p. 59
• Its broader purpose is to document milestones shaping deep learning and show how remarkable ideas developed among contemporaneous research.Source: §8 · p. 59
• Occam’s razor favors simple, intuitive models, illustrated by Dropout and the successive simplification from Hopfield Networks to RBMs.Source: §8 · p. 59
• Ambitious complexity is justified when it solves important problems, as LSTM addresses vanishing gradients and DBN enables effective training of deep architectures.Source: §8 · p. 59
• Broad reading helps researchers draw on external knowledge, including the visual cortex for CNNs and Residual Networks and game theory for GANs.Source: §8 · p. 59
• These directions are intended to help readers generate greater societal impact, while further principles may emerge from studying historical milestones.Source: §8 · p. 59