Gradients are Not All You Need

1 Introduction

• Differentiable implementations apply neural-network optimization techniques to systems including rigid-body physics, graphics, molecular dynamics, optimization procedures, weather, and nuclear-fusion simulation.The authors describe this as a way to place a task of interest into a differentiable pipeline and train end to end.Source: §1 · p. 1
• Automatic differentiation can compute formally correct gradients through differentiable systems, but those gradients may not be algorithmically useful when optimizing system dynamics.The discussion focuses on chaos as one potential issue in iterative differentiable systems.Source: §1 · p. 1
• The paper situates difficulties differentiating chaotic dynamics in earlier climate-modeling discussion [Lea et al., 2000, Köhl and Willebrand, 2002].It notes that the issue has since appeared in neural-network design, control, meta-learning, fluid simulation, and protein-structure learning [Yang and Schoenholz, 2017, Hayou et al., 2019, Parmas et al., 2018, Parmas, Metz et al., 2019, Ni and Wang, 2017, Kochkov et al., 2021, Ingraham et al., 2018].Source: §1 · pp. 1–2
• The gradient’s formal correctness is qualified by numerical precision, and cited cases can yield gradients that are close rather than exact [Chow and Palmer, 1992, Kachman et al., 2017].This qualification appears in the footnote attached to the claim about formal correctness.Source: §1 · p. 1

2 Preliminaries: Iterated Dynamical Systems

• Repeated state transitions are represented as s_{k+1} = A_k s_k, with trajectories tending to diverge when the largest eigenvalue is typically above 1 and vanish when it is below 1.The transition matrices may depend on state.Source: §2 · p. 2
• The general setup treats a transition function as depending on state and control variables, and evaluates losses accumulated over a finite trajectory.The formalism is described as encompassing many machine-learning practices, with f as the iterated function, s as input, θ as parameters, and l as the objective.Source: §2 · p. 2
• Gradients of trajectory losses contain repeated products of state Jacobians, so their behavior depends on the Jacobian spectrum and the number of iterations.As the number of steps grows, products can lead to exponentially growing or shrinking gradients.Source: §2 · p. 3
• Dynamical-system loss gradients depend on Jacobian spectra; when every state-Jacobian eigenvalue has magnitude below one, the system is stable and the resulting Jacobian product is well behaved.Source: §2 · p. 3
• Stochasticity can smooth a loss surface, but reparameterized gradients still multiply state Jacobians and can have extremely high norms.Black-box estimates may have lower variance for the same gradient in some cases [Parmas et al., 2018, Metz et al., 2019].Source: §2 · p. 3
• For a bounded underlying loss, Gaussian smoothing bounds the smoothed-loss gradients according to the smoothing amount, while reparameterized estimates can grow exponentially and require exponentially many samples for accuracy.Black-box estimates based on function evaluations can be better behaved, at the cost of poor performance as dimensionality increases.Source: §2 · p. 4
• At low oscillation frequencies, reparameterization has lower gradient-estimate variance; at higher frequencies, the black-box estimate has lower variance while reparameterization variance continues to grow.The comparison uses a Gaussian-smoothed oscillatory loss with σ = 0.3.Source: Figure 1 · p. 3, §2 · p. 4

3 Chaotic loss across a variety of domains

• These systems are not chaotic throughout their state spaces; restrictions such as avoiding contacts or using very small learning rates can restore stability.The paper notes that optimization can move toward unstable regions.Source: §3 · p. 4

3.1 Rigid Body Physics

• With longer Ant simulation unrolls, gradient variance grows exponentially despite averaging that smooths the loss surface.The effect is sensitive to the parameter location at which gradients are computed.Source: §3.1 · p. 4, Figure 2 · p. 5

3.2 Meta-learning: Backpropagation through learned optimization

• For learned optimizers, longer unrolls produce high-curvature regions, and meta-gradient variance explodes at some parameter settings but grows modestly at others.The smoothed meta-loss can appear well behaved even when variance explodes.Source: §3.2 · p. 5, Figure 3 · p. 6

3.3 Molecular Dynamics

• In disk packing, optimization lowers energy while increasing seed-to-seed variation; gradient variance grows quickly with steps, especially for the smallest diameters.At 256 steps, average energy remains well behaved even where seed-to-seed variance is large.Source: §3.3 · p. 6, Figure 4 · p. 7

3.4 Connecting gradient explosion to spectrum of the recurrent Jacobian

• For unstable Ant initialization, recurrent Jacobian eigenvalues above 1 accompany exponential growth in their cumulative maximum and gradient norms; stable initialization has controlled norms.The stable initialization has many eigenvalues close to 1 and little to no cumulative maximum growth.Source: §3.4 · pp. 6–7

4.1 Pick well behaved systems

• Changing a system can restore stability, but may alter the target physics; the paper contrasts this with language modeling, where an easier-to-optimize architecture may suffice.The authors note that such a change can introduce bias when the goal is to simulate rigid-body physics.Source: §4.1 · p. 8

4.1.1 Recurrent neural networks

• For vanilla RNNs, identity-like initialization delays gradient problems but Jacobians can drift during training; LSTMs and related structures improve stability but may not model chaotic relationships.The cited LSTM design specifically avoids exponential sensitivity to hidden state.Source: §4.1.1 · p. 8

4.1.2 Rigid Body Physics

• For contact physics, softening contact forces or splitting trajectory optimization at contacts can help, though such methods add algorithmic complexity; black-box methods also solved these tasks reliably without task-specific tuning.The paper describes the added complexity as a possible necessary cost for difficult problems.Source: §4.1.2 · p. 8

4.1.3 Well-behaved proxy objectives

• Known landscape properties can serve as proxy objectives; implicit differentiation can reduce unrolling-related conditioning and memory costs, but requires deterministic convergence to a single fixed point.Proxy objectives are usable only when known a priori; implicit gradients do not resolve chaos generally.Source: §4.1.3 · pp. 8–9

4.2 Truncated backpropogation

• Truncated backpropagation can permit Ant locomotion learning at a narrow range of truncation lengths, but introduces gradient bias; gradient clipping is also biased and did not yield positive performance in the reported Ant sweep.Long truncations failed to optimize, while truncations around length 10 produced successful locomotive policies.Source: §4.2 · p. 9, Figure 6 · p. 9, §4.3 · p. 10

4.4 Methods for ergodic systems

• For ergodic systems, least-squares shadowing uses a nearby trajectory to estimate gradients, while shadowing-operator methods map perturbed coordinates to true coordinates to compute gradients.The paper notes least-squares shadowing is most often discussed for continuous-time systems with differential-equation and tangent solvers.Source: §4.4 · p. 10, §4.4.1 · p. 10, §4.4.2 · p. 10

4.4.3 Probabilistic approaches

• Other ergodic approaches differentiate state distributions or average finite-horizon evaluations across ensembles; the S3 algorithm improves convergence rates for the latter family.Source: §4.4.3 · pp. 10–11

4.5 Learned Models

• Learned models can replace or supplement complex dynamics with shallower, often faster approximations; black-box gradients trade dimensional efficiency for potentially lower variance and can be combined with other estimates.The paper reports that PPO outperformed all of its Ant-policy gradient experiments.Source: §4.5 · p. 11, §4.6 · p. 11

• Chaos is the discussion’s focus, but numerical precision, memory requirements, and difficult loss landscapes are also concerns for gradient-based learning.Naive backpropagation requires memory proportional to sequence length, while gradient checkpointing can reduce that requirement.Source: §5 · p. 11
• The authors acknowledge that recurrent-process gradients have produced many successful results and hope the paper clarifies when gradients can be used.Source: §5 · p. 11
• Gradients through recurrent systems may be useful when the recurrent Jacobian has small eigenvalues; otherwise, black-box methods estimate the same quantity with less pathological variance, especially with a smoothed loss proxy.Source: §5 · p. 12

• A dynamical progression uses an iterative algebraic process to transform a state at time t into a state at time t + 1.Source: A Progression of dynamical systems · p. 19
• For long-range temporal processes, cascades of confounders may affect convergence properties or generated dynamics.Source: A Progression of dynamical systems · p. 19
• Automatic differentiation makes these questions salient because the number of dynamical steps can span vast orders of magnitude.Source: A Progression of dynamical systems · p. 19

A.1 Deterministic single transformation

• For a fixed transformation A, the state after k steps is expressed as x_k = A^k x_0.Source: §A.1 · p. 19
• Convergence for every initial state requires eigenvalues other than 1 to have magnitude below 1; the text also gives an additional rank condition when 1 is an eigenvalue.Source: §A.1 · p. 19
• The transformation is represented as A = XDX^-1, with D’s diagonal entries holding the eigenvalues.Source: §A.1 · p. 19
• Eigenvalues with magnitude below 1 decay under repeated maps, while eigenvalues above 1 diverge.Source: §A.1 · p. 19
• An eigenvalue of 1 may produce periodic or nearly periodic orbits, or orbits that diverge to infinity.Source: §A.1 · p. 19

A.2 Deterministic multiple transformation

• The section considers transformations that are inhomogeneous over time, so their spectrum changes over time.Source: §A.2 · p. 19
• For slowly varying matrices, successive transformations are approximated by I + ϵ, with Figure 7 used to numerically justify the small-variation assumption.Source: §A.2 · p. 19, Figure 7 · p. 20
• Eigenvalues greater than 1 lead to divergence, while smaller eigenvalues halt the dynamics; convergence requires a cascade of same-scale eigenvalues.Source: §A.2 · p. 20

A.3 Stochastic transformations

• For stochastic transformations, the matrix A is random, and the analysis considers two cases.Source: §A.3 · p. 20

A.3.1 Gaussian randomness

• The analysis represents the product of eigenvalues using a determinant.Source: §A.3.1 · p. 20
• The section separates the symmetric and nonsymmetric parts of the minor and expresses variance using chi-squared distributions.Source: §A.3.1 · p. 21
• The difference of two chi-squared distributions is a generalized Laplace distribution.Source: §A.3.1 · p. 21
• Convergence depends on the variance.Source: §A.3.1 · p. 21
• Bounding non-Gaussian noise is outside this paper’s scope and is identified as future work.Source: §A.3.1 · p. 21

A.3.2 On the special case of discrete laplacian

• The special case makes the dynamical progression depend on the Hessian.Source: §A.3.2 · p. 21
• The setup specifies x in [a, b], initial condition u(x, 0) = f(x), and zero boundary values at a and b.Source: §A.3.2 · p. 21
• When ∆x2 < 0.5, the map matrix is nonnegative, so the stated Perron-Frobenius result gives a dominant positive eigenvalue and a nonnegative eigenvector.Source: §A.3.2 · p. 21
• The section then invokes Gershgorin’s theorem to bound eigenvalues of a real or complex matrix by disks.Source: §A.3.2 · p. 21

A.3.3 On the special case of discrete drift diffusion

• The discussion introduces the drift diffusion equation as a further escalation and considers it under boundary conditions.Source: §A.3.3 · p. 22
• The case specifies x ∈ [a, b], initial data u(x, 0) = f(x), and zero values at both boundaries.Source: §A.3.3 · p. 22
• For this case, convergence of the map is stated to require a condition.Source: §A.3.3 · p. 22