Source-linked AI summary
Automatic differentiation in machine learning: a survey
Atilim Gunes Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul, Jeffrey Mark Siskind
TL;DR
Machine learning needs derivatives, but manual and numerical differentiation are laborious, error-prone, inaccurate, or poorly scalable. This survey clarifies AD’s relationship to other differentiation techniques and concludes that reverse-mode AD supports machine learning’s large parameter gradients while general-purpose AD expands programming frameworks.
Problem
Manual differentiation is time-consuming and error-prone, while numerical differentiation suffers accuracy and scalability problems for high-dimensional machine learning gradients.
Method
The paper surveys AD’s intersection with machine learning, covering applications, implementation techniques, differentiation modes, and relationships among differentiation approaches.
Results
AD matches symbolic differentiation’s accuracy with constant-factor overhead and supports control flow, while reverse mode is the mainstay for scalar objectives with many parameters.
Takeaways & Limitations
Machine learning frameworks are transitioning toward fine-grained, general-purpose AD that integrates differentiation into regular programs.
Takeaways & Limitations
AD can introduce numerical issues absent from the primal calculation because it remains subject to floating-point arithmetic.
Abstract
from arXiv · showhide
Derivatives, mostly in the form of gradients and Hessians, are ubiquitous in machine learning. Automatic differentiation (AD), also called algorithmic differentiation or simply "autodiff", is a family of techniques similar to but more general than backpropagation for efficiently and accurately evaluating derivatives of numeric functions expressed as computer programs. AD is a small but established field with applications in areas including computational fluid dynamics, atmospheric sciences, and engineering design optimization. Until very recently, the fields of machine learning and AD have largely been unaware of each other and, in some cases, have independently discovered each other's results. Despite its relevance, general-purpose AD has been missing from the machine learning toolbox, a situation slowly changing with its ongoing adoption under the names "dynamic computational graphs" and "differentiable programming". We survey the intersection of AD and machine learning, cover applications where AD has direct relevance, and address the main implementation techniques. By precisely defining the main differentiation techniques and their interrelationships, we aim to bring clarity to the usage of the terms "autodiff", "automatic differentiation", and "symbolic differentiation" as these are encountered more and more in machine learning settings.
1. Introduction
The introduction distinguishes automatic differentiation (AD) from manual, numerical, and symbolic differentiation, emphasizing its accurate, efficient treatment of derivatives in general computer programs. It frames AD as a long-underused machine learning capability closely related to backpropagation and surveys its applications, implementation, and conceptual distinctions.
- Differentiation methods: Derivative computation methods comprise manual, numerical finite-difference, symbolic, and automatic differentiation.AD is also called algorithmic differentiation.
- Machine learning context: Machine learning traditionally relies on gradients and Hessians, often requiring researchers to manually derive analytical derivatives for optimization.Backpropagation is a specialized AD counterpart and a mainstay for training neural networks.
- Automatic differentiation: AD reinterprets programs so operators propagate derivative values according to the chain rule.It can evaluate derivatives at machine precision with a small constant-factor overhead and supports regular code with branching, loops, and recursion.
- Paper scope: The paper reviews AD’s origins, machine-learning applications, implementation methods, and differences from numerical and symbolic differentiation.It also aims to dispel misconceptions that have impeded broader recognition of AD in machine learning.
2. What AD Is Not
AD is neither numerical nor symbolic differentiation: it uses symbolic differentiation rules to compute numerical derivative values. Compared with numerical methods’ approximation and scaling problems and symbolic methods’ expression swell, AD combines symbolic accuracy with efficient evaluation and control-flow support.
- AD versus other differentiation techniques: AD uses symbolic differentiation rules while tracking numerical derivative values rather than constructing derivative expressions.This gives AD a partly symbolic and partly numerical character.
- Numerical differentiation: Numerical differentiation approximates derivatives with finite differences, requiring O(n) function evaluations for an n-dimensional gradient and careful step-size selection.The method is uncomplicated to implement but depends on choosing h appropriately.
- Numerical differentiation: Numerical derivative approximations are inherently ill-conditioned and unstable because truncation and round-off errors affect finite-difference calculations.Complex-variable methods are an exception for a limited set of holomorphic functions.
- AD versus other differentiation techniques: AD avoids these limitations by differentiating elementary operations symbolically while storing intermediate numerical values, achieving symbolic-level accuracy with constant-factor overhead and control-flow support.This interleaves differentiation and simplification during function evaluation and forms the basis of forward accumulation.
- Numerical differentiation: Numerical differentiation is poorly suited to machine learning because gradient computation scales as O(n), with n reaching millions or billions in modern deep-learning models.Approximation errors might be tolerated, but the computational complexity remains the main obstacle.
- Symbolic differentiation: Symbolic differentiation produces derivative expressions that can become exponentially larger than the original expressions, causing expression swell and slow evaluation.The problem arises when common computations are duplicated during symbolic expansion.
3. AD and Its Main Modes
Automatic differentiation augments numerical program evaluation with derivative calculations by applying the chain rule to elementary operations. Its forward and reverse modes compute derivatives through evaluation traces, with reverse mode especially suited to scalar objectives with many parameters.
- AD foundations: AD augments a program’s numerical computation with derivatives, combining derivatives of elementary operations through the chain rule.This interpretation applies to numerical computations expressed as compositions of elementary operations.
- AD foundations: Evaluation traces represent intermediate computations and provide the basis for AD, including programs with branches, loops, recursion, and procedure calls.Unlike symbolic differentiation, AD handles these control-flow constructs because derivatives depend on the resulting numeric trace.
- Forward accumulation mode: Forward mode propagates tangent values alongside primal values, computing Jacobian–vector products in one forward pass when initialized with a vector r.For f : R^n → R^m, the full Jacobian requires n forward evaluations, while f : R → R^m needs one pass.
- Forward accumulation mode: Forward mode is efficient for f : R → R^m but requires n evaluations to compute gradients for f : R^n → R.When n ≫ m, the paper indicates that another technique is often preferred.
- Reverse accumulation mode: Reverse mode propagates derivatives backward from an output and complements intermediate variables with adjoints representing output sensitivities.Because machine learning commonly differentiates a scalar objective with respect to many parameters, reverse mode is the mainstay technique in backpropagation.
4. AD and Machine Learning
AD supports machine-learning optimization by efficiently computing gradients, exact Hessians, and Hessian–vector products, while general-purpose differentiation simplifies neural-network and differentiable-program implementations. Its applications extend across optimization, neural networks, and probabilistic models requiring derivatives through sampling.
- Optimization: Reverse-mode AD efficiently computes gradients for large n, with Helmholtz free-energy benchmarks showing different scaling for forward mode, reverse mode, and numerical differentiation.The benchmark evaluates gradient computation using the Helmholtz free-energy function from prior AD literature.
- Optimization: AD automatically computes exact Hessians for Newton’s method, while graph elimination, partial separability, and matrix coloring exploit Hessian sparsity and symmetry.Newton’s method can converge in fewer iterations but requires Hessian computation at each iteration; large-scale problems often use quasi-Newton approximations instead.
- Optimization: Reverse-on-forward AD computes Hessian–vector products with O(n) complexity, enabling stochastic Newton methods that may provide quadratic convergence in stochastic optimization.The method first computes a directional derivative in forward mode and then applies reverse mode to obtain H_fv.
- Neural networks: Backpropagation is a special case of reverse-mode AD, while general-purpose AD lets neural-network architectures be expressed as regular programs using shared differentiation infrastructure.This differentiable-programming perspective reflects longstanding AD research on differentiable functions and language-level differentiation.
- Probabilistic inference: Probabilistic models require derivatives through random-variable sampling; REINFORCE gives an unbiased gradient estimate but can have high variance.For continuous random variables, sampling can instead be represented through a deterministic differentiable transformation.
5. Implementations
The section surveys AD implementation strategies, classifying them into elemental, operator-overloading, compiler-based, and hybrid methods. It emphasizes performance, perturbation-confusion, numerical, and approximation-related risks, while noting applications of AD implementation techniques in machine learning.
- Implementation costs: AD increases arithmetic by no more than a small constant factor, but careless bookkeeping can add substantial memory, allocation, and dispatch overhead.Naively holding dual numbers incurs memory access and allocation per operation, while operator overloading can slow computation by an order of magnitude.
- Implementation risks: Nested or simultaneous differentiations can trigger perturbation confusion when distinct formal epsilons are not kept separate.The problem is especially easy to introduce in performance-oriented implementations and nested AD.
- Numerical considerations: AD remains subject to floating-point and approximation hazards, including unstable near-zero sums and derivatives of programmed approximations rather than ideal functions.Users may need to explicitly approximate a known derivative when the mathematical function is only computable approximately but has a well-defined derivative.
- Machine learning applications: Checkpointed backpropagation through time saved up to 95% memory at the cost of a 33% computation increase in one recurrent-network instance.The section also notes that compute-bound deep-learning workloads with optimized matrix-operation kernels may suit operator-overloading AD on high-level operations.
- Implementation taxonomy: AD implementations are classified as elemental, operator-overloading, compiler-based, or hybrid methods.The taxonomy follows prior classifications of general-purpose AD implementations.
- Implementation strategies: Elemental methods replace mathematical operations with AD-enabled library calls, whereas source transformation preprocessors convert extended-language code into the original language.Elemental methods require manually decomposing functions into elementary operations; source-transformation tools generate augmented code computing specified derivatives.
6. Conclusions
Backpropagation and gradient-based optimization remain central to machine learning, while the intersection with automatic differentiation offers opportunities for transferring techniques and formalizing emerging methods. Nested AD is highlighted as a future direction for exact hypergradients in hyperparameter optimization.
- Conclusions: Backpropagation and gradient-based optimization underpin virtually all recent machine-learning successes and are expected to remain central for the foreseeable future.The passage cites computer vision, speech recognition and synthesis, and machine translation as examples of state-of-the-art applications.
- Conclusions: The AD–machine-learning intersection offers opportunities to apply AD techniques such as tape reduction, fixed-point iterations, and sparsity-based matrix coloring.These techniques are identified as developments from the AD community that could inform machine-learning problems.
- Conclusions: Direct propagation of the inverse Jacobian emerged from machine learning but has not yet been examined and formalized by the AD community.The passage presents this as an example of an AD mode originating in machine learning.
- Conclusions: Nested AD is an important future direction because arbitrary-depth, referentially transparent differentiation can provide exact hypergradients for hyperparameter optimization.Hypergradients are derivatives of a training objective with respect to optimization-routine hyperparameters.