Directional convergence and alignment in deep learning

Ziwei Ji; Matus Telgarsky

Source-linked AI summary

Directional convergence and alignment in deep learning

Ziwei Ji, Matus Telgarsky

arXiv:2006.06657v2Jun 11, 2020cs.LGcs.NEmath.OCstat.ML

TL;DR

The paper asks whether deep-network behavior stabilizes after weights diverge and perfect classification is reached. It proves directional convergence using o-minimal and unbounded nonsmooth Kurdyka–Łojasiewicz tools, then establishes gradient alignment under locally Lipschitz gradients. The resulting margin and saliency-map convergence is supported in homogeneous AlexNet and observed in non-homogeneous DenseNet experiments.

Problem
After perfect classification, weights diverge and prediction surfaces may continue changing, raising questions about the stability of trained-network behavior.
Method
The paper analyzes gradient flow for definable, positively homogeneous networks using unbounded nonsmooth Kurdyka–Łojasiewicz inequalities.
Results
The normalized parameters and prediction margins converge in direction, while locally Lipschitz gradients also converge and align with the gradient-flow path.
Takeaways & Limitations
The results imply margin-maximization consequences and convergence of saliency maps, with empirical margin convergence also observed for DenseNet outside the theory.
Takeaways & Limitations
The analysis is limited to binary classification and assumes homogeneity, leaving extensions to DenseNet, ResNet, and finite-time stochastic methods for future work.

Abstract

from arXiv · show

In this paper, we show that although the minimizers of cross-entropy and related classification losses are off at infinity, network weights learned by gradient flow converge in direction, with an immediate corollary that network predictions, training errors, and the margin distribution also converge. This proof holds for deep homogeneous networks -- a broad class of networks allowing for ReLU, max-pooling, linear, and convolutional layers -- and we additionally provide empirical support not just close to the theory (e.g., the AlexNet), but also on non-homogeneous networks (e.g., the DenseNet). If the network further has locally Lipschitz gradients, we show that these gradients also converge in direction, and asymptotically align with the gradient flow path, with consequences on margin maximization, convergence of saliency maps, and a few other settings. Our analysis complements and is distinct from the well-known neural tangent and mean-field theories, and in particular makes no requirements on network width and initialization, instead merely requiring perfect classification accuracy. The proof proceeds by developing a theory of unbounded nonsmooth Kurdyka-Łojasiewicz inequalities for functions definable in an o-minimal structure, and is also applicable outside deep learning.

1 Introduction

The paper addresses whether deep-network predictions remain unstable after perfect classification, proving directional convergence despite weights diverging. It further establishes gradient alignment under locally Lipschitz gradients and supports convergence empirically within and beyond its theoretical assumptions.

Motivation: The paper resolves concerns about continually changing prediction surfaces by proving stable convergence despite parameter norms diverging to infinity.The motivation is that perfect classification does not stop prediction surfaces from changing during training.
Directional convergence: For deep homogeneous networks, normalized parameters Wt/∥Wt∥ converge in direction, without assumptions on width or parameter initialization.The theory covers arbitrarily deep networks with linear, convolutional, ReLU, and max-pooling layers, but excludes skip connections and biases.
Directional convergence: Directional convergence implies convergence of normalized prediction margins, which are central to studying generalization and stability of other quantities.The analysis uses binary logistic or exponential loss under gradient flow, beginning from risk below 1/n.
Gradient alignment: When gradients are locally Lipschitz, they converge and align with the gradient-flow path, yielding consequences for margin maximization and saliency-map convergence.The gradient-flow path and the risk gradient converge to the same direction.
Proof strategy: The proof develops unbounded nonsmooth Kurdyka–Łojasiewicz inequalities for definable functions, addressing both nonsmooth layers and exponential or logistic losses.These tools are designed to handle ReLU and max-pooling together with cross-entropy-related losses and may apply beyond deep learning.
Experiments: Experiments show convergent margins for a homogeneous AlexNet covered by the theory and a standard DenseNet outside its assumptions.The synthetic experiment also compares evolving prediction surfaces with the frozen-activation NTK predictor, while CIFAR experiments use SGD.

2 Preliminaries and assumptions

The paper models locally Lipschitz, positively homogeneous, definable predictors and analyzes gradient flow using Clarke subdifferentials under a late-training risk condition.

Nonsmooth analysis: Clarke subdifferentials are nonempty, convex, and compact, and reduce to the ordinary gradient when the function is continuously differentiable.The analysis frequently uses the unique minimum-norm Clarke subgradient.
Definability: An o-minimal structure consists of sets closed under standard operations whose one-dimensional definable sets are finite unions of intervals and points.Polynomials and exponentials are definable in an appropriate structure, supporting common deep-learning layers and cross-entropy compositions.
Network model and assumptions: The predictor is locally Lipschitz, L-positively homogeneous, and definable in an o-minimal structure.Homogeneity permits linear, convolutional, ReLU, and max-pooling layers, but excludes skip connections and biases.

3 Directional convergence

Under the stated assumptions, the normalized gradient-flow path has finite length and therefore converges, yielding convergence of normalized margins. The proof combines a smoothed margin with unbounded nonsmooth Kurdyka–Łojasiewicz inequalities.

Directional convergence: The normalized flow f_Wt := Wt/∥Wt∥ converges because its swept curve has finite length under logistic or exponential loss.The assumptions imply ∥Wt∥→∞, so normalization isolates directional behavior.
Margin convergence: Normalized margins pi(Wt)/∥Wt∥L converge for every training example.Positive homogeneity identifies each normalized margin with the corresponding prediction evaluated on the normalized parameters.
Proof strategy: The proof tracks the smoothed margin and the spherical path length, using their derivative comparison to establish finite total path length.The smoothed margin is nondecreasing with a finite limit, while ζt denotes the length swept by the normalized flow.
Proof strategy: The finite-length conclusion follows by applying the asymptotic inequality after the flow enters a region with sufficiently large norm and smoothed margin near its limit.The resulting bound keeps the limiting path length finite.
Proof strategy: Unbounded nonsmooth Kurdyka–Łojasiewicz inequalities control the asymptotic regime despite locally Lipschitz, nonsmooth definable objectives.The argument uses separate inequalities in two cases determined by a ratio involving the flow and its subgradients.

4 Alignment between the gradient flow path and gradients

Under locally Lipschitz gradient conditions, gradients align asymptotically with the gradient-flow parameters. This alignment yields margin-maximization consequences in deep linear and shallow squared-ReLU networks, subject to stated assumptions.

Alignment: The negative loss gradient and parameters converge to the same direction when each prediction function has locally Lipschitz gradients.If the prediction functions are twice continuously differentiable, definability is not required.
Proof sketch: The proof addresses the difficulty that the exact minimum margin can be nondifferentiable and have discontinuous Clarke subdifferentials.A potential function is used to rule out later increases in the angle between parameters and the negative gradient.
Margin maximization: Directional convergence and alignment immediately support global margin-maximization results for deep linear and shallow squared-ReLU networks.The subsection treats these two examples as consequences of the main convergence theorems.
Deep linear networks: For deep linear networks, the limiting predictor is a unique linear max-margin predictor, with layer directions linked through unit vectors.The result follows by combining directional convergence and alignment with the layerwise gradient expressions.
Shallow squared-ReLU networks: In shallow squared-ReLU networks, the local guarantee characterizes the limiting margin using a linear problem over features induced by limiting neuron directions.The global guarantee additionally requires a finite-width covering condition to obtain a margin nearly matching the infinite-width global max-margin value.

5 Concluding remarks and open problems

The paper establishes directional convergence of normalized parameters and, under locally Lipschitz gradients, convergence and alignment of gradients. It identifies broader architectures, optimization methods, loss settings, and margin consequences as open directions.

Main conclusions: The normalized parameter vectors converge, while locally Lipschitz gradients also converge and align with the parameters.These are the paper’s stated main convergence conclusions.
Open problems: Future work includes weakening homogeneity to cover DenseNet and ResNet, analyzing finite-time gradient methods, and deriving convergence rates.The paper also notes that its analysis handles only binary classification, while the tools should directly allow cross-entropy.
Open problems: Further work could extend global margin-maximization results beyond the simple networks analyzed in Section 4.2 and study related generalization consequences.The passage presents these as directions rather than established results.

A Experimental setup

The experiments test directional convergence across synthetic and CIFAR data using gradient descent or SGD, homogeneous and non-homogeneous architectures, and margin visualizations.

Experimental goals: Experiments target a clear, reliable directional-convergence phenomenon using contour plots and margin plots.Margin plots include ResNet results in Figure 3.
Models and data: Synthetic experiments use a 10,000-node, two-layer squared-ReLU network with one trainable layer and a corresponding Neural Tangent Kernel classifier.The squared-ReLU network is 2-homogeneous and was selected to support the alignment guarantee and contrast with NTK behavior.
Models and data: CIFAR experiments use a homogeneous modified AlexNet, an unmodified DenseNet, and an unmodified ResNet.H-AlexNet combines ReLU, max-pooling, linear, and convolutional layers and is 5-homogeneous; DenseNet and ResNet lie outside the assumptions.
Training: Training uses vanilla gradient descent for Figure 1 and mini-batch SGD with batch size 512 for Figures 2 and 3, without weight decay or other regularization.Experiments were run to accuracy 10^-8 or greater; step sizes were scaled by 1/L(Wt−1) and reduced when induced updates were too large.
Training: SGD produces only small overall differences from smoother gradient-descent plots, while rigorous directional-convergence analysis for SGD remains future work.SGD is outside the paper’s gradient-flow assumptions.
Margin plots: Margins are sorted by final margin, tracked per example, and indexed by ln n/L(Wt) to reduce effects of step sizes and implementation choices.Homogeneous networks normalize by ∥Wt∥L, whereas DenseNet and ResNet margins use the median nonnegative margin across data.

B Results on o-minimal structures

The paper defines o-minimal structures through closure properties for sets in Euclidean spaces and defines functions through definable graphs.

Structure definition: An o-minimal structure contains collections S_n of subsets of R^n, with S_1 consisting of finite unions of open intervals and points.The structure also includes polynomial zero sets.
Closure properties: The collections are closed under finite unions, finite intersections, complements, and Cartesian products.These operations preserve membership in the corresponding S_n collections.
Closure properties: They are also closed under projection onto the first n coordinates.Projection closure is central to the definition and supports definability of function domains.
Definability: A set is definable when it belongs to S_n, while a function is definable when its graph belongs to the appropriate S_{n+m}.The paper works within an arbitrary fixed o-minimal structure.

B.1 Basic properties

The appendix develops closure, limit, monotonicity, curve-selection, and network-definability properties needed for the later analysis of definable functions.

Closure properties: Definability is preserved under algebraic combinations, coordinate operations, composition, inverse functions, and finite pointwise maxima or minima.These closure rules support first-order constructions and the representation of network layers.
Optimization-related properties: Infima, infimal convolutions, epigraphs, distance functions, closures, interiors, boundaries, and lower-semicontinuous envelopes remain definable under stated conditions.The infimum result requires definability and boundedness assumptions.
Network definability: Networks built from finite maxima or minima of polynomials are definable, covering linear, convolutional, ReLU, max-pooling layers, and skip connections.The proof uses induction through composition and the polynomial representation of layer coordinates.
One-dimensional behavior: Definable functions on intervals have endpoint limits, and their derivatives have constant signs on finitely many subintervals.These facts yield finite-length paths for convergent C1 definable curves.
Curve selection: Curve-selection lemmas provide C1 definable curves approaching points or escaping to infinity while tracking function limits.The at-infinity version applies to sequences with unbounded norm and convergent function values.

B.2 Clarke subdifferentials

The appendix establishes definability and chain-rule properties for Clarke subdifferentials and applies them to locally Lipschitz definable functions and gradient flow.

Clarke subdifferentials: For locally Lipschitz definable functions, the Clarke subdifferential and its unique minimum-norm subgradient are definable.The result is obtained by representing the relevant sets with definable constructions and applying Carathéodory’s theorem.
Chain rule: A chain rule holds almost everywhere along arcs for locally Lipschitz definable functions.The proof localizes the function, extends it through infimal convolution, and applies an existing chain-rule result.
Gradient flow: Along the gradient flow, the parameter derivative equals the negative minimum-norm Clarke subgradient almost everywhere.The same lemma also gives an almost-everywhere expression for the risk derivative.

B.3 Kurdyka- Lojasiewicz inequalities

This section develops bounded and unbounded Kurdyka–Łojasiewicz inequalities for locally Lipschitz definable functions. The unbounded analysis separates radial and spherical behavior and uses definability to control asymptotic Clarke critical values.

Asymptotic critical values: Locally Lipschitz definable functions have only finitely many asymptotic Clarke critical values.This finiteness supports choosing a sufficiently small interval without such values.
The bounded setting: For bounded domains, definability yields a definable desingularizing function satisfying a Kurdyka–Łojasiewicz inequality.The proof reduces the function range near zero to a definable interval and constructs a definable curve and inverse-function reparameterization.
The bounded setting: The bounded proof uses finitely many asymptotic critical values to ensure positivity of the auxiliary slope function near zero.After excluding critical values, the proof obtains φ(z) > 0 on a sufficiently small interval.
The unbounded setting: The unbounded setting requires two Kurdyka–Łojasiewicz inequalities according to the relationship between radial and spherical parts of the Clarke subdifferential.One inequality applies on a region Uϵ,c,η, while the other handles points outside that region through a definable change of variables.
The unbounded setting: For unbounded sequences, definable curves at infinity and inverse-function reparameterizations extend the bounded argument.The construction treats bounded and unbounded sequences separately and uses a C1 definable curve in the latter case.

C Omitted proofs from Section 3

These omitted proofs establish the technical properties linking homogeneous losses, normalized margins, Clarke subdifferentials, and gradient-flow dynamics. They show that normalized margin behavior can control the trajectory and its limiting direction.

Homogeneity and subdifferentials: For an L-positively homogeneous locally Lipschitz function, every Clarke subgradient satisfies the generalized Euler identity.The result extends Euler’s homogeneous function theorem beyond differentiable functions.
Directional convergence: Positive homogeneity gives f(x + δx) = (1 + δ)^L f(x), which supplies the scaling relation used in the convergence analysis.The argument applies this identity to compare radial perturbations and normalized margins.
Smoothed margins: The Clarke subdifferentials of the unnormalized and normalized smoothed margins are related through their radial and spherical components.Their spherical components determine simultaneous minimization of subgradient norms, while the normalized and unnormalized margins remain close.
Gradient-flow dynamics: The gradient-flow norm is increasing and stays bounded away from zero under the stated initialization assumption.This permits treating the normalized trajectory as an arc and controlling its curve length.
Directional convergence: The proof of directional convergence splits into two cases governed by complementary inequalities for radial and spherical subgradients.Both cases yield bounds on the evolution of the normalized margin and establish the core convergence lemma.

D Omitted proofs from Section 4

This section proves gradient-direction convergence and asymptotic alignment under locally Lipschitz gradient assumptions. The argument combines homogeneity, uniform Hessian bounds on the sphere, and control of the angle between parameters and gradients.

Homogeneous gradient estimates: The gradient of an L-positively homogeneous function with locally Lipschitz gradient is (L − 1)-positively homogeneous.This scaling law is used repeatedly to transfer estimates between arbitrary parameter norms and the unit sphere.
Regularity bounds: Local Lipschitzness of the component gradients makes the Jacobian locally Lipschitz and enables uniform Hessian bounds on the unit sphere.Compactness supplies a finite subcover and therefore a uniform constant Kσ.
Angle control: The technical estimates control the angle θ between the parameter vector and the loss gradient.The proof derives bounds from the normalized margin, radial and spherical gradient components, and homogeneity.
Scope of assumptions: If all component prediction functions are C2, the proof can avoid definability because the required chain rule holds directly.This marks a broader regularity-based scope for the alignment result.

E Global margin maximization proofs for Section 4.2

These proofs lift directional convergence and alignment from the full parameter vector to structured subsets, then characterize limiting layer and node configurations through margin-maximization conditions.

Subset alignment: For any partition of the parameters, normalized subset norms converge and each subset asymptotically aligns with its gradient.This subset-alignment lemma applies to both the deep linear and 2-homogeneous cases.
Deep linear networks: In deep linear networks, every layer norm diverges and all layers align with their gradients.The proof obtains layerwise rank-one directional limits represented by unit vectors vj.
Deep linear networks: The normalized end-to-end linear map asymptotically satisfies the optimality conditions for the unique maximum-margin vector.Uniqueness is guaranteed because the data are linearly separable under the stated loss assumption.
2-homogeneous networks: Accumulation points of the dual variables lie in the simplex and are supported on examples minimizing the prediction margin.Together with the primal conditions, Sion’s theorem yields a primal-dual optimal pair for the margin saddle-point problem.
2-homogeneous networks: In the 2-homogeneous case, positive limiting subset mass occurs exactly at nodes attaining the maximal margin value.This establishes the primal optimality conditions for the associated discrete saddle-point problem.