Source-linked AI summary
There Will Be a Scientific Theory of Deep Learning
Jamie Simon, Daniel Kunin, Alexander Atanasov, Enric Boix-Adserà, Blake Bordelon, Jeremy Cohen, Nikhil Ghosh, Florentin Guth, Arthur Jacot, Mason Kamb, Dhruva Karkada, Eric J. Michaud, Berkan Ottlik, Joseph Turnbull
TL;DR
Deep learning lacks a unified scientific framework, despite its importance and decades of theoretical work. This paper synthesizes emerging research into a proposed mechanics of the learning process, including tractable limits, empirical laws, hyperparameter theories, and universal behaviors, while identifying scope boundaries.
Problem
Deep learning lacks a unified framework explaining why or how neural networks learn, and training still relies largely on trial and error.
Method
The paper synthesizes major research strands involving learning dynamics, coarse aggregate statistics, solvable limits, empirical laws, hyperparameters, and universal behaviors.
Results
The paper argues that a scientific theory of deep learning is emerging and is best understood as a mechanics of the learning process.
Takeaways & Limitations
Learning mechanics emphasizes falsifiable quantitative predictions about training processes, representations, weights, and performance.
Takeaways & Limitations
The proposed mechanics is expected to break down in many small-scale, handcrafted, or otherwise special cases, and scaling-law exponents remain unpredictable a priori across realistic settings.
Abstract
from arXiv · showhide
In this paper, we make the case that a scientific theory of deep learning is emerging. By this we mean a theory which characterizes important properties and statistics of the training process, hidden representations, final weights, and performance of neural networks. We pull together major strands of ongoing research in deep learning theory and identify five growing bodies of work that point toward such a theory: (a) solvable idealized settings that provide intuition for learning dynamics in realistic systems; (b) tractable limits that reveal insights into fundamental learning phenomena; (c) simple mathematical laws that capture important macroscopic observables; (d) theories of hyperparameters that disentangle them from the rest of the training process, leaving simpler systems behind; and (e) universal behaviors shared across systems and settings which clarify which phenomena call for explanation. Taken together, these bodies of work share certain broad traits: they are concerned with the dynamics of the training process; they primarily seek to describe coarse aggregate statistics; and they emphasize falsifiable quantitative predictions. We argue that the emerging theory is best thought of as a mechanics of the learning process, and suggest the name learning mechanics. We discuss the relationship between this mechanics perspective and other approaches for building a theory of deep learning, including the statistical and information-theoretic perspectives. In particular, we anticipate a symbiotic relationship between learning mechanics and mechanistic interpretability. We also review and address common arguments that fundamental theory will not be possible or is not important. We conclude with a portrait of important open directions in learning mechanics and advice for beginners. We host further introductory materials, perspectives, and open questions at learningmechanics.pub.
1 Introduction
Deep learning theory is shifting from explaining what models can represent toward scientifically describing and predicting how complex neural networks learn. The paper argues that converging research programs support calling this emerging framework learning mechanics.
- Motivation: Deep learning remains powerful but lacks a unified scientific framework explaining why or how neural networks learn.Training methods are still largely developed through trial and error, while theory plays little role in everyday practice.
- Motivation: The field’s complexity—nonconvexity, overparameterization, and scale—limits classical theories’ ability to explain neural-network optimization and generalization.Deep learning theory therefore became a scientific effort to describe, explain, and predict empirical behavior.
- Evidence for an emerging theory: Five evidence streams indicate an emerging theory: solvable settings, informative limits, empirical laws, understood hyperparameters, and universal phenomena across settings.These include deep linear networks and kernel methods, infinite-width and depth limits, macroscopic scaling laws, simplified effective dynamics, and recurring behaviors across tasks.
- Learning mechanics: Together, these research programs study training dynamics, coarse aggregate statistics, and accurate average-case predictions rather than primarily worst-case bounds.The paper compares this orientation with mechanics in physics and proposes the name learning mechanics.
- Implications: The proposed framework is intended to support practical utility, scientific understanding, and collaboration with mechanistic interpretability and AI safety research.The paper presents it as a potentially transformative foundation while emphasizing that its usefulness depends on clear applicability boundaries.
- Desiderata: Learning mechanics aims for a first-principles, mathematical, predictive, and comprehensive account at the right level of resolution rather than a full-detail description.Its stated goals include describing training, hidden representations, and final weights while sacrificing detail for insight.
2 Evidence of an emerging mechanics of learning
An emerging mechanics of learning is supported by solvable models, tractable limits, empirical laws, hyperparameter theories, and recurring behaviors that expose regularities in complex training systems.
- Foundations: Deep learning is directly measurable: weights, activations, gradients, losses, and derived statistics can be recorded and used to test theoretical predictions.This measurability makes experiments comparatively easy to design, replicate, and interrogate.
- Analytically solvable settings: Solvable settings show that learning can reduce to simple mathematics, including sequential low-rank acquisition and linearized dynamics with predictive inductive biases.Deep linear networks learn larger-singular-value modes first, while linearized nonlinear networks connect architecture to NTK-based inductive bias and generalization.
- Analytically solvable settings: Linearized models provide useful predictions but omit feature learning and intrinsically nonconvex optimization, limiting their realism for generic neural networks.They can therefore yield overly pessimistic sample-complexity predictions and do not capture the full nonlinear learning process.
- Scope and motivation: The scale of modern systems makes microscopic theories that track individual parameters impractical, motivating coarse-grained analyses and effective descriptions.Modern models contain hundreds of interacting components, hundreds of billions of parameters, and trillions of tokens.
- Insightful limits: Limits manage deep-learning complexity by exposing mathematical structure that can remain informative for finite systems.The paper presents recurring success with such limits as evidence for an emerging theory.
- Insightful limits: Infinite-width, finite-width, and infinite-depth limits reveal distinct lazy, rich, and smooth dynamical regimes that simplify otherwise high-dimensional systems.Output scale can promote either feature learning or near-linearized training, while depth scaling can produce smooth evolution through the residual stream.
3 Relation to other perspectives
The paper presents learning mechanics as a quantitative, dynamics-focused perspective that complements statistical, information-theoretic, physics, neuroscience, and mechanistic-interpretability approaches. These perspectives can jointly connect coarse training laws with interpretable mechanisms.
- Complementary perspectives: Learning mechanics focuses on the training process, while related perspectives offer complementary tools for explaining deep learning.The paper describes statistical, information-theoretic, physics, and neuroscience approaches as complementary rather than competing frameworks.
- Complementary perspectives: Statistical and information-theoretic views ask how architecture, training, data, and optimization produce implicit bias, compression, and generalization.The paper argues that making these perspectives concrete requires examining the training process and its interaction with architecture and data.
- Learning mechanics and mechanistic interpretability: Mechanistic interpretability reverse-engineers features, circuits, and learned algorithms, whereas learning mechanics seeks compact quantitative principles governing learning dynamics.The two approaches study the same systems at different levels of abstraction.
- Learning mechanics and mechanistic interpretability: Learning mechanics can formalize interpretability assumptions and explain how mechanisms develop through training.The paper identifies these as two complementary ways learning mechanics can support mechanistic interpretability.
- Learning mechanics and mechanistic interpretability: Mechanistic interpretability supplies concrete, data-structured phenomena that learning mechanics can turn into well-defined targets for mathematical modeling.This helps address the gap between simplified theoretical data models and behaviors observed in practice.
4 Reasons for skepticism and responses
The paper responds to skepticism by arguing that recent empirical successes, broader disciplinary participation, and partial progress on basic building blocks make a theory increasingly plausible. It also argues that theory remains useful for engineering, science, and human oversight.
- Why theory may now be possible: Recent scaling has exposed new phenomena and shifted the search for deep learning theory from mathematics toward empirical science.The paper cites apparent convergence to universal representations as an example revealed by recent model scaling.
- Scope of attainable theory: A complete first-principles theory of large models may take considerable time, so near-term progress can come from understanding basic building blocks.The paper presents isolated pockets of understanding as useful without requiring a constructive theory of the whole model.
- Levels of explanation: Learning mechanics is intended to complement high-level studies of model behavior by analyzing lower-level physical, algorithmic, and learning processes.The paper argues that multiple levels of analysis may be necessary.
- Theory of data and learning: A useful theory requires both an account of structure in data and an account of how parameterized models learn that structure.The paper treats both as parts of the mechanics of learning.
- Why theory matters: The authors argue that theory can have near-term engineering impact and support human-parseable oversight, including for AI safety.They expect humans to remain involved in breakthrough progress and oversight.
5 Open directions in learning mechanics
The paper proposes open directions spanning solvable nonlinear models, natural-data theory, functional complexity, formal feature definitions, finite-limit approximations, hyperparameters, scaling laws, curvature, optimizers, and representation universality. Together, these questions define a research agenda for learning mechanics.
- Solvable models: No unified solvable framework yet captures both deep nonlinear parameter dynamics and nonlinear function learning.Existing work captures slices of fully nonlinear learning dynamics, motivating models that also illuminate feature learning, depth, optimization, and architectural innovations.
- Natural data: A theory of natural data must identify the structure exploited by deep networks and determine the minimal sufficient statistics that guide learning.The paper asks whether these statistics vary across models and training stages.
- Functional complexity: A general account of implicit functional-complexity minimization remains unresolved, including which complexity notion applies and when minimization is exact.Candidate notions include Kolmogorov complexity, circuit complexity, weight norm, and related biases.
- Features and interpretability: Learning mechanics should define neural features mathematically and test interpretability assumptions such as linear representability, locality, sparsity, and compositionality.The paper also asks how these concepts connect to the more precise rich-versus-lazy feature-learning distinction.
- Limits and hyperparameters: Finite networks may be discretized approximations to infinite-width, infinite-depth, or continuous training systems, but this hypothesis remains open.The proposed analogy treats width as neuron-population discretization and residual depth as discretization of a neural ODE or SDE.
- Quantitative laws and universality: Other open questions ask whether hyperparameters can be eliminated, scaling-law exponents predicted a priori, curvature dynamics explained, optimizers understood, and representation universality characterized.The representation questions include how similarity should be measured and across which training regimes convergence persists.
6 How to get involved in the development of learning mechanics
The paper encourages newcomers to enter learning mechanics from diverse backgrounds, experiment frequently, prioritize simplicity and insight, explore multiple problems, and invest in foundational tools. It frames these practices as ways to produce accessible, extensible, and durable contributions.
- Entering the field: No specific academic background is required, and cross-pollination from physics, mathematics, computer science, neuroscience, and statistics is valuable.The paper emphasizes that established ideas from other fields can be applied to deep learning.
- Research practice: Frequent, simple experiments can check assumptions, inform models, reveal limitations, and expose phenomena for further study.The authors recommend including experiments in every paper when feasible.
- Research practice: Simplicity and insight should take priority over technical complexity because understandable findings are easier for others to extend and apply.The paper recommends identifying underlying intuition and checking it with simple experiments.
- Research practice: Trying several problems before committing deeply can help newcomers learn the field and develop high-level ideas.The paper links broad early exploration with the value of knowing multiple areas.
- Foundational tools: Investing in fundamental tools such as statistical physics, random matrix theory, optimization, signal processing, and information theory can support work across learning-mechanics problems.The paper presents these tools as useful for high-dimensional limits and neural-network optimization.