Source-linked AI summary
The Unreasonable Effectiveness of Deep Learning in Artificial Intelligence
Terrence J. Sejnowski
TL;DR
The paper examines paradoxes in deep-learning theory, including questions about optimization and the set of good solutions. It discusses high-dimensional parameter spaces and multilayer-network training, while reporting continued performance gains with added layers, strong generalization, and AlphaGo’s victory over the world champion Go player.
Problem
Empirical studies uncovered paradoxes that conflict with conventional statistical-learning intuition, including questions about optimization and the size of the set of good solutions.
Method
The paper considers how high-dimensional parameter spaces support efficient navigation to solutions and discusses a learning algorithm for training multilayer networks.
Results
Performance continued to improve as more layers were added, and deep learning achieved good generalization contrary to conventional predictions; combined with deep learning, AlphaGo beat the world champion Go player.
Takeaways & Limitations
High-dimensional parameter spaces provide a supported explanation for efficient optimization and generalization despite failures predicted by conventional intuition.
Takeaways & Limitations
A perceptron can represent only categories that are linearly separable in weight space, and training more powerful multilayer perceptrons was historically difficult.
Abstract
from arXiv · showhide
Deep learning networks have been trained to recognize speech, caption photographs and translate text between languages at high levels of performance. Although applications of deep learning networks to real world problems have become ubiquitous, our understanding of why they are so effective is lacking. These empirical results should not be possible according to sample complexity in statistics and non-convex optimization theory. However, paradoxes in the training and effectiveness of deep learning networks are being investigated and insights are being found in the geometry of high-dimensional spaces. A mathematical theory of deep learning would illuminate how they function, allow us to assess the strengths and weaknesses of different network architectures and lead to major improvements. Deep learning has provided natural ways for humans to communicate with digital devices and is foundational for building artificial general intelligence. Deep learning was inspired by the architecture of the cerebral cortex and insights into autonomy and general intelligence may be found in other brain regions that are essential for planning and survival, but major breakthroughs will be needed to achieve these goals.
Figure Captions
The figures trace developments in neural computation, from early analog perceptrons and machine-learning communities to brain organization, biological efficiency, and changing views of dimensionality.
- Dimensionality and perspective: The figures use Flatland and Flammarion’s engraving to evoke how unfamiliar dimensions can challenge ordinary intuitions about space.Flatland depicts a two-dimensional society, while the engraving imagines a point where sky and Earth touch.
- Machine-learning community: The 1987 NeurIPS conference brought together researchers from science and engineering and was initially sponsored by the IEEE Information Theory Society.The conference was held at the Denver Tech Center and became an annual meeting.
- Early neural computation: Early perceptrons used large-scale analog hardware, including motor-driven potentiometers controlled by a learning algorithm.The illustrated system received visual input and implemented variable resistance through mechanical components.
- Biological inspiration: Biological and engineered systems can improve energy efficiency by reducing drag, as illustrated by eagle wingtip feathers and aircraft winglets.The figure connects gliding efficiency with fuel savings from vortex reduction.
- Brains and computation: Brain organization combines energy-efficient synaptic signaling, three-dimensional neural interconnects, locally dense and distantly sparse connectivity, and hierarchical sensory and motor maps.The cortex coordinates with subcortical areas to form the central nervous system that generates behavior.