PaperlayerView exact revision on arXiv

Source-linked AI summary

Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes

James Lucas, Juhan Bae, Michael R. Zhang, Stanislav Fort, Richard Zemel, Roger Grosse

arXiv:2104.11044v2cs.LGcs.AIstat.ML

TL;DR

The paper asks why linear interpolation from initialization to a trained neural-network solution usually yields monotonically decreasing training loss and whether this behavior is stable. It combines empirical studies across training settings with differential-geometric analysis, showing sufficient MSE conditions based on function-space Gauss length and systematic MLI violations when weights move far from initialization.

  • Problem

    The paper investigates why the MLI property persists across neural-network training and whether it is a stable property of architectures and training.

  • Method

    The authors vary architectures, optimizers, datasets, initializations, and training mechanisms, and analyze interpolated function-space paths using Gauss length.

  • Results

    MLI persists in most tested settings but can be systematically broken by mechanisms that encourage weights to move far from initialization; small function-space Gauss length provides a sufficient MSE condition for MLI.

  • Takeaways & Limitations

    MLI is not a stable architectural property and provides a lens for studying neural-network loss-landscape geometry and training mechanisms.

  • Takeaways & Limitations

    For average dataset loss, individual logit trajectories may be non-monotonic, and the study finds no clear relationship between MLI success and generalization ability.

Abstract

from arXiv · show

Linear interpolation between initial neural network parameters and converged parameters after training with stochastic gradient descent (SGD) typically leads to a monotonic decrease in the training objective. This Monotonic Linear Interpolation (MLI) property, first observed by Goodfellow et al. (2014) persists in spite of the non-convex objectives and highly non-linear training dynamics of neural networks. Extending this work, we evaluate several hypotheses for this property that, to our knowledge, have not yet been explored. Using tools from differential geometry, we draw connections between the interpolated paths in function space and the monotonicity of the network - providing sufficient conditions for the MLI property under mean squared error. While the MLI property holds under various settings (e.g. network architectures and learning problems), we show in practice that networks violating the MLI property can be produced systematically, by encouraging the weights to move far from initialization. The MLI property raises important questions about the loss landscape geometry of neural networks and highlights the need to further study their global properties.

1. Introduction

The paper revisits Monotonic Linear Interpolation (MLI), in which training loss typically decreases along the line from initialization to a trained solution, and studies its persistence, causes, and implications for loss-landscape geometry. It combines empirical variation across training settings with theoretical conditions and finds that MLI can be systematically broken.

  • The MLI property is the typical monotonic decrease of training loss along interpolation from initialization to converged parameters, even from an unrelated initialization to the same solution.
  • The study systematically evaluates how persistent MLI is, why it holds, and what it reveals about neural-network loss landscapes.These questions motivate the paper’s expanded empirical and theoretical investigation.
  • Across architectures, optimizers, datasets, initializations, and training mechanisms, MLI persists in most settings but can be consistently broken when weights move far from initialization.The authors describe this as evidence that MLI is not a stable property of network architecture.
  • The paper provides a sufficient MSE condition: small Gauss length of the interpolation trajectory in function space guarantees MLI, while Gauss length correlates with monotonicity in practice.The authors connect larger weight-space movement to non-monotonic interpolation through a power-law relationship with average Gauss length.
  • The work identifies training mechanisms that control MLI and connects those mechanisms to theoretical explanations of interpolation behavior.

2. Related Work

The related work situates MLI within studies of linear connectivity, loss-landscape geometry, function-space linearity, and optimization. It distinguishes initialization-to-solution interpolation from mode connectivity and motivates examining how training dynamics and parameter movement affect MLI.

  • Linear connectivity: Prior work found that linear paths between modes can cross high-loss regions, while SGD solutions may still connect through piecewise linear paths.
  • Linear connectivity: MLI differs from mode connectivity because it connects an initialization to a final solution, whereas mode connectivity connects different final solutions.The paper notes that prior work had not explored connections between these phenomena.
  • Loss landscape geometry: The paper relates MLI to low-dimensional convergence near initialization, treating successful one-dimensional interpolation as a special case of these loss-landscape results.
  • Optimization and geometry: Large learning rates can help reach flatter minima by overcoming initially high-curvature regions, while this paper links larger learning rates to MLI failure and high-loss barriers.
  • Optimization algorithms: The study examines optimization algorithms because adaptive methods can move farther in parameter space than gradient descent, a difference that co-occurs with changes in MLI behavior.

3. The Monotonic Linear Interpolation Property

The MLI property concerns whether loss decreases along the linear path from initialization to convergence, and can be quantified through bounded interpolation bumps. The paper connects MLI to parameter displacement and the geometry of logit trajectories, deriving sufficient conditions under MSE loss.

  • Definition: MLI requires training loss to decrease along the linear interpolation from initialization θ0 to converged parameters θT.The property is evaluated for all interpolation positions α between 0 and 1.
  • Definition: The smallest Δ bounding every positive loss increase along the path provides a quantitative measure of non-monotonicity.It can be approximated by sampling the path and finding the largest positive loss gap.
  • Parameter-space perspective: Networks that move farther in weight space are significantly more likely to produce non-monotonic initialization-to-optimum interpolations.Conversely, sufficiently small displacement supports a quadratic approximation that yields monotonic interpolation, while sufficiently wide lazy-training networks satisfy MLI with high probability.
  • Function-space perspective: MLI can coincide with nearly linear logit interpolations, motivating Gauss length as a unit-free differential-geometric measure of trajectory bending.The Gauss map tracks normalized tangent directions, and its path length measures how much the function-space curve bends.
  • Function-space perspective: If the Gauss length of a smooth function-space trajectory is less than π/2, squared-distance loss to its target is monotonically decreasing.The theorem applies to arbitrary smooth curves and therefore supplies sufficient conditions for MLI under mean squared error, including applications to two-layer linear models.
  • Function-space perspective: Average Gauss length predicts network-level monotonicity even when individual logit trajectories are non-monotonic.The experiments use average loss over the dataset, whereas the theory is stated for the trajectory-level setting.

4. Exploring & Explaining the MLI Property

The study finds that MLI persists across most standard training settings but can be systematically disrupted by large learning rates, adaptive optimizers, and batch normalization. It links monotonicity to weight-space distance and Gauss length, and finds that MLI behavior reflects broader loss-landscape structure.

  • How persistent is the MLI property?: MLI persisted across most standard choices of width, depth, activation, initialization, and regularization, but failures were regularly induced by large learning rates, Adam, and batch normalization.Batch-normalized classifiers failed more frequently, particularly with large learning rates and Adam.
  • How persistent is the MLI property?: 71% of SGD configurations with learning rate 1.0 were non-monotonic, while smaller learning rates always exhibited MLI.Large learning rates were necessary but not always sufficient for violating MLI; the authors suggest a possible connection to the catapult phase.
  • The effect of adaptive optimizers: Adam-trained MNIST and Fashion-MNIST models violated MLI 3.2 times more often than SGD-trained models, with average minimum ∆ values of 0.626 versus 0.016.Across autoencoders and classifiers, Adam produced more non-monotonic interpolations than SGD, especially with larger learning rates.
  • Weight distance versus monotonicity: Networks traveling farther in weight space were more likely to have non-monotonic interpolations, while short-distance networks consistently satisfied MLI.Larger learning rates increased weight movement across batch normalization and adaptive-optimizer settings.
  • Why does MLI hold?: Small Gauss lengths produced monotonic interpolations, whereas increasing Gauss length was associated with increasing non-monotonicity and larger learning rates produced much larger Gauss lengths.The observed relationship extended beyond the strict limits of the theoretical analysis.
  • Why does MLI hold?: A power-law relationship between average Gauss length and weight-space distance appeared across all explored experimental settings.This relationship connects the function-space and parameter-space measures used to analyze MLI.
  • What does MLI say about loss landscapes?: Unrelated random initializations showed the same interpolation behavior as the original initialization-to-solution pair, suggesting MLI is a global loss-landscape property.No barriers appeared between initialization pairs or initialization-to-optimum pairs, whereas barriers appeared between optima.

5. Conclusion

The study shows that MLI is not stable across neural-network training, yet often reflects broader structure in the loss landscape. It combines systematic empirical violations with theoretical analysis linking monotonicity to function-space geometry.

  • The study systematically produces networks that violate MLI and connects these failures to theoretical explanations of the property.
  • MLI persists across many datasets, architectures, optimization methods, and training mechanisms, but can be broken by mechanisms that move weights far from initialization.
  • The analysis indicates that MLI is not limited to initialization-to-solution paths, extending instead to arbitrary initialization-to-solution pairs in the loss landscape.
  • Together, the empirical and theoretical results highlight unresolved questions about the global geometry of neural-network loss landscapes.

Appendix A. Theoretical Gauss Length Analysis

The appendix develops a Gauss-length analysis of MLI for function-space interpolation, proving sufficient conditions for monotonicity and applying them to two-layer linear models. It also identifies limits of the guarantee and conditions under which MLI holds almost surely.

  • If an interpolation trajectory has Gauss length below π/2, its MSE loss is guaranteed to decrease monotonically along the path.
  • A violation of monotonicity implies Gauss length of at least π/2, because an increase in loss forces two tangent directions to form an angle of at least π/2.
  • The converse does not hold: curves can have arbitrarily large Gauss length while their loss decreases monotonically.
  • Two-layer linear models: For two-layer linear models trained with MSE, the appendix derives sufficient conditions for MLI and proves almost-sure MLI under specified assumptions.
  • Two-layer linear models: The model is a two-layer linear map f(x) = V Wx, with student-teacher labels enabling interpolation trajectories to reach the objective minimum.

Appendix B. Experiment Details

The experiments span diverse architectures, datasets, optimizers, initialization schemes, and training mechanisms. They show that MLI usually persists, while adaptive optimization and greater movement away from initialization are associated with more frequent violations.

  • Across varied architectures, datasets, activations, initialization methods, regularization, batch sizes, and normalization settings, common training choices usually have little effect on MLI.
  • Higher learning rates generally encourage non-monotonicity in CIFAR-10 and CIFAR-100 ResNets, although the correlation is weaker than for MNIST and Fashion-MNIST classifiers.
  • Adaptive optimizers such as RMSProp and Adam frequently produce non-monotonic interpolations, whereas SGD typically preserves MLI on MNIST and Fashion-MNIST.
  • Networks trained with Adam move farther from initialization than SGD-trained networks, and non-monotonicity occurs more often for configurations traveling farther in weight space.

C.4. Optimizer Ablations

Optimizer-switching experiments show that Adam is associated with larger parameter movement and Gauss length, while grafting experiments implicate update magnitude rather than direction as the main driver of non-monotonicity.

  • Switching from SGD to Adam makes all tested runs non-monotonic, whereas training with SGD alone yields monotonic interpolations.
  • Switching from Adam to SGD reduces both average Gauss length and distance traveled, linking the optimizer change to less curved and shorter interpolation trajectories.
  • Using Adam’s direction with SGD’s step magnitude keeps all runs monotonic, so Adam’s direction is not the primary influence on the trajectory.
  • Using SGD’s direction with Adam’s step magnitude makes all runs non-monotonic and yields an average distance traveled of 381.65, implicating update magnitude.
  • Across reconstruction and classification tasks, larger traveled distances are associated with larger non-monotonicity measures.

C.6. Additional Gauss length experiments

Gauss length and weight-space distance both track departures from monotonic interpolation: larger values generally correspond to larger non-monotonicity, though the relationship varies by task and architecture.

  • Larger Gauss lengths generally produce larger interpolation bumps, linking function-space path geometry to failures of the MLI property.For image reconstruction, small Gauss length corresponds to monotonic interpolation, while larger Gauss length correlates with MLI failure.
  • Larger distances moved in weight space likewise correspond to larger non-monotonicities in CIFAR-10 and CIFAR-100 ResNets.The same trend appears when plotting minimum Δ-monotonicity against weight-space distance.
  • Gauss length and weight distance show a clear power-law relationship for MNIST autoencoders.This relationship is weaker for CIFAR ResNets, where the correlation remains positive but is not clearly power-law.
  • Batch normalization increases variance in travelled weight distance, creating more configurations that move far enough to violate MLI.This provides a possible explanation for more frequent MLI violations under batch normalization.

C.9. Additional loss landscape experiments

Additional landscape experiments show that MLI extends across unrelated initializations, optima, and permutation-related parameterizations, while failures can involve a barrier followed by flat curvature.

  • A Fashion-MNIST classifier that fails MLI exhibits a wide loss barrier followed by an extremely flat-curvature region.
  • Permutation symmetries generate different weight-space paths while preserving the endpoint functions, enabling interpolation tests across equivalent parameterizations.
  • Random initializations can reproduce the original interpolation behavior, suggesting MLI is a global loss-landscape property rather than a property of one parameter pair.
  • Unrelated initial points and optima yield monotonic training-loss decreases and monotonic accuracy increases in ResNet20v1 projections.
  • Loss surfaces for RoBERTa show monotonic loss reduction along interpolations among the initialization and unrelated optima or initializations.

C.10. Additional MNIST results

Across additional MNIST and Fashion-MNIST experiments, learning rate and batch normalization matter more for MLI than depth or hidden size, while monotonic interpolation persists under unstable training and varying dataset size.

  • Batch normalization has a significant effect on interpolation behavior across settings, especially at large learning rates.
  • Varying depth and hidden size: Learning rate dominates changes in monotonicity, whereas hidden size and network depth show no significant effect.The heatmaps compare depth-3 networks across hidden sizes and width-1024 networks across depths.
  • Dataset size: Even with unstable, highly non-linear training dynamics, interpolation remains a smooth monotonic curve across dataset sizes from 30 to 3000 examples.
  • The experiments vary dataset size, hidden units, and label corruption to revisit whether MLI reflects relative ease of optimization.
  • The learning-dynamics evidence is inconclusive: some runs overcome large barriers while retaining MLI, whereas other projections show a barrier followed by lower curvature.

C.13. MLI on held-out data

Held-out loss generally follows training interpolation trends, although overconfidence can make test loss rise near the endpoint; the theoretical analysis gives high-probability MLI guarantees for sufficiently wide networks.

  • Held-out data: For MNIST classifiers, test loss rises near the endpoint despite small training loss because predictions become over-confident, while accuracy remains unchanged.
  • Held-out data: Test loss generally follows the overall training-loss interpolation trend for CIFAR classifiers, while often increasing near the endpoint.
  • Wide neural networks: Under mean-squared error and stated regularity assumptions, sufficiently wide fully connected networks satisfy MLI with high probability under suitably small learning rates.
  • Wide neural networks: The wide-network result relies on lazy training, where parameter movement remains close to initialization and the interpolation behaves within a locally convex region.
  • Wide neural networks: The proof linearizes the loss in function space and shows quadratic remainder terms are dominated by the negative first-order term as width increases.

D.2. A Noisy Quadratic Model

The noisy quadratic model links stochastic training dynamics to possible non-monotonic interpolation through inflection points near optima. Its endpoint derivative conditions reduce MLI verification to the converged endpoint, while simulations show that smaller learning rates increase the probability of detecting non-monotonicity.

  • Model motivation: The noisy quadratic model explains one possible source of non-monotonicity: an interpolation-curve inflection point with positive second derivative near the optimum.The model is motivated by prior work showing alignment between its predictions and experimental results on deep neural networks.
  • Interpolation criterion: For the quadratic loss, interpolation is monotonic exactly when the derivative with respect to α is negative at both endpoints.In one dimension, non-monotonicity occurs when the two interpolated points lie on opposite sides of the minimum.
  • Interpolation criterion: Because the learning rate makes the loss decrease in expectation at initialization, checking the derivative condition at the converged endpoint suffices.The initial derivative is already anti-aligned with the interpolation direction under the chosen learning-rate regime.
  • Stochastic dynamics: As training time tends to infinity, the iterate approaches a Gaussian distribution centered at zero with covariance ηK, arising from gradient noise.Empirical simulations verify that this limiting distribution is approximately symmetric about zero.
  • Simulation results: Smaller learning rates reduce the variance of the relevant endpoint quantity, increasing the probability that discretized checks detect non-monotonicity.Figure 29 empirically verifies this relationship, attributing the effect to an inflection point near the optimum that can be difficult to detect.
Loading 2104.11044v2…