Source-linked AI summary
On the Origin of Deep Learning
Haohan Wang, Bhiksha Raj
TL;DR
Deep learning’s rapid progress raises the need to understand how its present models emerged from earlier ideas and trade-offs. This paper reviews evolutionary paths across major model families and optimization, then distills milestone patterns to support a deeper background for future research.
Problem
Deep learning’s current forms require historical understanding of what has been attempted and why models developed as they did.
Method
The paper reviews the evolutionary paths of deep generative, convolutional, and recurrent model families, along with optimization techniques and their predecessors.
Results
The review organizes major milestones and explains the ideas and relationships connecting earlier work to current deep learning models.
Takeaways & Limitations
The paper documents milestones and summarizes recurring directions in how influential deep learning works developed.
Takeaways & Limitations
Traditional recurrent neural networks face vanishing or exploding gradients during backpropagation through long sequences.
Abstract
from arXiv · showhide
This paper is a review of the evolutionary history of deep learning models. It covers from the genesis of neural networks when associationism modeling of the brain is studied, to the models that dominate the last decade of research in deep learning like convolutional neural networks, deep belief networks, and recurrent neural networks. In addition to a review of these models, this paper primarily focuses on the precedents of the models above, examining how the initial ideas are assembled to construct the early models and how these preliminary models are developed into their current forms. Many of these evolutionary paths last more than half a century and have a diversity of directions. For example, CNN is built on prior knowledge of biological vision system; DBN is evolved from a trade-off of modeling power and computation complexity of graphical models and many nowadays models are neural counterparts of ancient linear models. This paper reviews these evolutionary paths and offers a concise thought flow of how these models are developed, and aims to provide a thorough background for deep learning. More importantly, along with the path, this paper summarizes the gist behind these milestones and proposes many directions to guide the future research of deep learning.
1. Introduction
Deep learning has advanced AI performance across diverse tasks, motivating a historical account of how its major models and ideas developed. The paper organizes this account around milestones, model families, optimization, and connections among predecessors and current forms.
- Deep learning has dramatically improved state-of-the-art performance in object detection, speech recognition, and machine translation.
- The paper argues that understanding prior attempts and why current models have their present forms is necessary to push deep learning research forward.
- It traces major milestones and evolutionary relationships among deep learning models to provide readers with a background for understanding their development.
- The review covers brain modeling, shallow-to-deep neural networks, deep belief nets, convolutional neural networks, recurrent neural networks, and optimization techniques.
- The paper complements prior histories by emphasizing milestones and conceptual associations rather than documenting every incremental contribution.
2. From Aristotle to Modern Artificial Neural Networks
The paper traces neural-network ideas from associationism and theories of memory to Hebbian learning, neuron models, and perceptrons. Across this history, associative strengthening supports learning, while single perceptrons remain limited to linear decision boundaries.
- 2.1 Associationism: Associationism treats the mind as conceptual elements organized through associations, including contiguity, frequency, similarity, and contrast.
- 2.1 Associationism: Early associationist accounts linked complex ideas and memories to combinations of simpler experiences, neural activity, and repeated or contiguous impressions.
- 2.1 Associationism: Bain described neural groupings in which linked cells summarize stimulation, store multiple associations, and strengthen or weaken connections through experience.
- 2.3 Hebbian Learning Rule: Hebb’s “cells that fire together, wire together” rule emphasized co-fired-cell activation but can become unstable as frequently co-occurring weights keep increasing.
- 2.4 Oja’s Rule and Principal Component Analyzer: Oja extended Hebbian learning to avoid this instability and showed that its neuron update approximates Principal Component Analysis behavior.
- 2.7 Perceptron’s Linear Representation Power: A perceptron represents linear decision boundaries, handling NOT, AND, and OR but not XOR, which requires a more sophisticated boundary.
3. From Modern Neural Network to the Era of Deep Learning
Neural networks can approximate broad classes of functions, but shallow architectures may require impractically many neurons. This motivates depth, whose representational advantages come with optimization challenges and important qualifications.
- 3.1 Universal Approximation Property: An MLP with one hidden layer can represent any Boolean function exactly and approximate any bounded continuous function arbitrarily accurately.These are presented as distinct universal approximation properties.
- 3.1 Universal Approximation Property: An MLP with two hidden layers can approximate any function with arbitrary accuracy, including functions that are not necessarily continuous.The construction combines outputs representing many circles through an additional OR-performing hidden layer.
- 3.1 Universal Approximation Property: The intuitive Boolean construction rewrites a target formula into conjunctive normal form, computes AND operations in one layer, and combines them with OR.The approach relies on perceptrons implementing AND and OR operations.
- 3.2 The Necessity of Depth: Universal approximation by shallow networks requires exponentially many neurons, making the representation impractical despite its expressive power.The paper frames depth as a way to reduce computation units while retaining representation power.
- 3.2 The Necessity of Depth: Deep architectures can achieve representation growth that is exponential in neuron count, whereas shallow networks exhibit polynomial growth under the discussed analyses.The cited results cover sum-product networks and general networks with activations including tanh and sigmoid.
- 3.2 The Necessity of Depth: Although theory supports depth over width, increasing layers makes learning suitable parameters more difficult and motivates improved optimization methods.The paper connects backpropagation difficulties with later analyses of saddle points in neural-network cost surfaces.
4. The Network as Memory and Deep Belief Nets
This section introduces the deep-learning branches covered after modern neural-network foundations, beginning with unsupervised modeling and emphasizing trade-offs between representation power and computation complexity.
- The section frames model development through a trade-off between representation power and computation complexity.
- The section begins with a model for understanding data unsupervisedly and follows the branch leading to Restricted Boltzmann Machines and Deep Belief Nets.
- Figure 7 organizes the models by computation complexity on the horizontal axis and representation power on the vertical axis.The figure places six section milestones in this representation-complexity space.
4.1 Self Organizing Map
Self-Organizing Maps reduce data to a low-dimensional representation while retaining topological similarity, making them useful for clustering and visualization. Their heuristic learning procedure updates a best-matching unit and its neighbors, but global topology depends on training choices.
- Self-Organizing Maps reduce data usually to one or two dimensions while retaining topological similarity and imposing topology on clustered representations.
- A two-dimensional SOM represents data through fixed-position units whose locations carry meaning beyond their numerical values.The arrangement preserves relative similarity, constraining which units can represent nearby concepts.
- The learned map can visualize multidimensional data by placing similar cases near one another and allowing relative rankings to be read at a glance.The paper illustrates this with countries positioned according to happiness and richness.
- SOM training randomly samples a data vector, selects its best-matching unit, and updates units within a distance-based neighborhood.The learning rate and neighborhood radius typically decrease over iterations.
- SOM training requires careful neighborhood-radius selection and weight initialization; otherwise, local similarity may fail to hold globally.Dissimilar clusters can separate otherwise similar groups in the resulting map.
4.2 Hopfield Network
Hopfield Networks use bidirectional fully connected threshold units and an energy function to store and recover patterns. Their inference searches for lower-energy states, but update procedures can reach local optima and storage capacity is limited.
- A Hopfield Network is a fully connected recurrent network with bidirectional weights, binary threshold units, and content-addressable memory.Here, “recurrent” refers historically to bidirectional connections rather than the modern RNN usage.
- The network’s energy function connects its neural dynamics to the potential energy formulation of spin-glass systems.The physical analogy describes interacting states converging toward stability as energy decreases.
- Hopfield weights are designed or learned so stored patterns minimize energy, allowing a partial or altered input to recover a memorized state.The paper gives face completion as an application example.
- The paper distinguishes parameter learning or training, which calculates weights, from inference or testing, which applies fixed weights to solve a task.
- Inference tests whether flipping individual units decreases energy, using asynchronous or synchronous updates that can converge to local optima or loops.Synchronous updates may even increase energy during the process.
- Hopfield Networks store at most 0.15N^2 bits with N units, limiting memory efficiency despite having N^2 edges.After storing M memories, the connections require N^2log(2M + 1) bits.
4.3 Boltzmann Machine
Boltzmann Machines extend Hopfield Networks with stochastic hidden units, defining a probability distribution through energy. Training maximizes observed-data likelihood, but exact expectation calculations are computationally infeasible, motivating Restricted Boltzmann Machines.
- A Boltzmann Machine is a stochastic Hopfield Network with hidden units, named after the Boltzmann distribution.
- The Boltzmann distribution assigns probabilities to states according to their energies and normalizes those values across possible configurations.The ratio of two state probabilities depends on their energy difference through the Boltzmann factor.
- Visible units connect to data while hidden units help describe its distribution, conceptually separating the model into visible and hidden parts.
- Stochastic state transitions give higher-energy states a chance to transfer while favoring transitions toward lower energy, with temperature controlling transfer likelihood.The paper fixes T = 1 for simplicity in this section.
- The energy function separates visible and hidden units, defines a joint probability over their configurations, and yields visible-unit probabilities by marginalizing hidden units.The visible distribution can be used to sample or generate data.
- Boltzmann Machine training maximizes observed-data likelihood, but its likelihood gradient contains expectations over many states that are generally infeasible to compute exactly.Markov Chain Monte Carlo is used to approximate the required sums; this sampling complexity motivates Restricted Boltzmann Machines.
- The likelihood gradient contrasts an expectation under observed data with an expectation under the model distribution.The data-driven term is generally straightforward, whereas the model term is more complicated.
- The computational difficulty of sampling motivates the next section’s Restricted Boltzmann Machine.
4.4 Restricted Boltzmann Machine
Restricted Boltzmann Machines simplify Boltzmann Machines by removing within-layer connections, yielding a bipartite graph and easier sampling. Contrastive Divergence then replaces effectively infinite sampling with finite steps, with k = 1 working well in practice.
- RBM removes connections between visible units and between hidden units, converting the Boltzmann Machine into a bipartite graph.This restriction also makes the RBM energy function much simpler.
- RBM training retains Boltzmann Machine learning but requires many sampling steps to approximate the model distribution.The simpler energy function makes inference easier, but does not eliminate the sampling burden.
- Contrastive Divergence uses a finite k-step sampling procedure instead of requiring potentially infinitely many steps to approximate the model distribution.Hinton introduced this method to address the sampling difficulty.
- k = 1 is sufficient for the learning algorithm to work well in practice.
- Finite-k Contrastive Divergence is theoretically biased, but empirical results indicate that its bias can be small while the algorithm works well in practice.
- RBM’s modeling power and fast approximation made it a fundamental building block for Deep Belief Nets and Deep Boltzmann Machines.
4.5 Deep Belief Nets
Deep Belief Networks stack RBMs but differ from simple RBM stacking through directed lower-layer connections and layerwise training. Their learning procedure combines unsupervised pre-training with task-dependent fine-tuning, a strategy that influenced later models.
- Deep Belief Networks introduced the possibility of training deep neural networks through greedy, layerwise stacking and training of RBMs.
- DBNs use bidirectional RBM-type connections only in the top layer, while lower layers have top-down connections.This structure makes DBNs multilayer generative models rather than simple stacks of RBMs.
- DBN parameter learning consists of layer-wise pre-training followed by fine-tuning.
- Layerwise pre-training trains adjacent layer pairs bottom up, treating each pair as an RBM before later fine-tuning.This procedure provides critical support for the success of fine-tuning.
- Unsupervised pre-training has been explained as initialization that improves optimization or as regularization that supports better generalization.These are proposed explanations for why pre-training works.
- DBN pre-training inspired related procedures for autoencoders, Deep Boltzmann Machines, and other models.
- Fine-tuning uses a contrastive wake-sleep algorithm for generative models and standard backpropagation for discriminative models.The discriminative strategy applies backpropagation to the pre-trained model, while pre-training remains necessary despite backpropagation’s performance.
4.6 Deep Boltzmann Machine
Deep Boltzmann Machines extend RBM-style energy models across multiple hidden layers with bidirectional connections. Compared with DBNs, this structure supports more complex patterns and top-down feedback during approximate inference.
- Deep Boltzmann Machines are deep generative models introduced as a later milestone in the history of generative modeling.
- DBMs allow bidirectional connections in bottom layers, unlike DBNs, making them closer to literal RBM stacking.The distinction is determined by how connections are formed between lower layers.
- A DBM with N hidden layers defines its energy function as an extension of the RBM energy function.
- The similarity to RBM energy functions permits training DBMs with Contrastive Divergence, although pre-training is typically necessary.
- DBM bidirectional structure can learn more complex data patterns and incorporate top-down feedback after an initial bottom-up pass.This allows approximate inference to better propagate uncertainty about ambiguous inputs.
4.7 Deep Generative Models: Now and the Future
After Deep Boltzmann Machines, generative-model research continued through Bayesian program learning, conditional image-generation models, and Generative Adversarial Networks. These directions extend beyond the classical deep generative models discussed earlier.
- Deep Boltzmann Machine is the final generative-model milestone discussed in the paper, while substantial later and future work remains.
- Bayesian Program Learning models can simulate human learning abilities for large-scale visual concepts and generate handwritten characters indistinguishable from human behavior in a visual Turing Test.
- Conditional image generation is commonly addressed with Pixel CNN and Pixel RNN models when part of the image is given.Providing part of the image simplifies the generation task.
- Generative Adversarial Networks are identified as another contribution to generative modeling but are considered too young for discussion in this paper.
5. Convolutional Neural Networks and Vision Problems
This section traces CNNs from visual-cortex inspiration and convolutional mechanisms to landmark architectures, while examining their capabilities and reliability limits in vision tasks.
- 5.1 Visual Cortex: CNN development is grounded in visual-cortex research, whose basic organization provides the biological background for studying convolutional networks.The section specifically focuses on visual-cortex basics as groundwork for CNNs.
- 5.3.1 Convolution Operation: Convolution applies a kernel locally across an input matrix, producing a smaller output while providing local shift invariance useful for visual recognition.A 5 × 5 input convolved with a 3 × 3 kernel becomes 3 × 3, and element arrangements can yield the same result under shifts.
- 5.3.2 Sampling: Pooling creates smaller embeddings and makes representations more invariant to small transformations, distortions, and translations.Maximum or average values over local neighborhoods can preserve the outcome despite small input changes.
- 5.4 LeNet: LeNet handled digit variations in position, scale, rotation, squeezing, and stroke width, while introducing MNIST as a later standard benchmark.Its robustness covered multiple forms of digit variation.
- 5.5.1 AlexNet: AlexNet provided the first evidence that CNNs could perform well on the historically difficult ImageNet classification task, launching intense CNN competition.Its success shifted CNN development toward increasingly competitive ImageNet architectures.
- 5.5.3 ResNet: Residual blocks enabled substantially deeper networks by providing an express path for inputs, supporting smoother information propagation and simpler optimization.Residual networks were extended successfully to 1000 layers on CIFAR, and their express paths were described as ensembles of shallower networks.
6. Time Series Data and Recurrent Networks
Recurrent neural networks model sequential data through directed cycles, and their history develops from Jordan and Elman feedback structures toward bidirectional, memory-based, deep, and attention-enhanced architectures. Training requires specialized procedures because recurrence creates unbounded temporal dependencies and vanishing or exploding gradients.
- RNNs use directed cyclic connections to process temporal or sequential data.
- 6.1 Recurrent Neural Network: Jordan Network and Elman Network: Jordan and Elman networks differ mainly in whether recurrent information comes from the previous output or hidden layer.The paper notes that the structures are not fundamentally different because y_t = W_y h_t; the distinction lies in the recurrent-weight choice.
- 6.1 Recurrent Neural Network: Jordan Network and Elman Network: Elman’s hidden-layer feedback introduced greater flexibility for later recurrent-structure design.
- Backpropagation through Time unfolds recurrence across selected time steps, converting training into feedforward backpropagation.Traditional backpropagation is infeasible for recurrent structures because they lack an endpoint where propagation can stop.
- BRNNs combine forward and reverse recurrent processing so later time steps can provide information to earlier ones.Training treats the model as two RNNs and applies BPTT to each before simultaneous weight updates.
- LSTM adds a memory cell and gates to address RNNs’ inability to capture long-term dependencies and the vanishing-gradient problem.Large-scale comparisons found no significant improvement from LSTM variants over the standard LSTM architecture.
- Deep RNNs can increase depth in the input, recurrent, or output components, while attention models focus processing on selected image regions.
- Despite practical remedies such as LSTM and ReLU, the fundamental vanishing/exploding-gradient problem remains unsolved for general RNNs.Through the BPTT chain rule, error signals can decrease exponentially across time steps, causing long-term dependencies to be lost.
7. Optimization of Neural Networks
The paper reviews optimization methods for deep neural networks, from gradient variants and normalization to dropout and architecture-growth procedures. It emphasizes both their practical benefits and limits, including decaying adaptive rates, limited convergence gains, and overfitting or greedy search.
- Modern neural-network optimization still relies heavily on gradient descent and backpropagation, which frame the subsequent methods reviewed.
- Rprop updates parameters using gradient signs rather than fully using gradient magnitudes, potentially handling difficult cost-surface curvatures.
- AdaGrad assigns larger learning rates to mildly updated parameters and smaller rates to dramatically updated parameters using historical gradient norms.
- AdaGrad improves robustness over traditional gradient methods, but its learning-rate factor can decay substantially as accumulated gradient norms grow.
- AdaDelta reweights accumulated gradients to reduce AdaGrad’s learning-rate decay, producing an update rule nearly identical to RMSprop.
- Empirical benefits from modern gradient variants are often limited despite claims of improved convergence rates.
- Dropout randomly removes hidden units during training and can approximate model averaging while reducing overfitting at lower computational cost.
- Batch Normalization normalizes layer inputs using batch statistics and learned rescaling and shifting, and works well empirically during training.
8. Conclusion
The paper concludes by tracing the evolutionary paths of major deep-learning families and optimization techniques. It presents historical reconstruction as both documentation and a way to identify how influential ideas emerged, emphasizing simplicity, ambition, and broad reading.
- The review covers deep generative, convolutional, and recurrent model families alongside selected optimization techniques.
- The paper documents milestones that influenced current deep learning and examines how notable works developed among contemporaneous publications.
- Its historical lessons emphasize simple designs, as illustrated by Dropout’s influence through implementation simplicity and intuitive reasoning.
- Remarkable complexity should address problems that simpler contemporaneous models cannot solve well, as illustrated by LSTM and DBN.
- Progress often draws on knowledge beyond machine learning, including biological vision for CNNs and game theory for GANs.
- The paper hopes these evolutionary directions help readers influence current society and derive further directions from the reviewed milestones.