Source-linked AI summary
Conservative set valued fields, automatic differentiation, stochastic gradient method and deep learning
Jérôme Bolte, Edouard Pauwels
TL;DR
The paper tackles the need for flexible calculus in nonsmooth AI and numerical-analysis problems. It introduces conservative fields and path differentiability, develops their calculus and stratification-based representations, and applies them to automatic differentiation and stochastic optimization. The resulting framework covers broad nonsmooth function classes and supports analysis of practical differentiation and training procedures.
Problem
Existing nonsmooth calculus mainly covers convex-like or Clarke-regular problems, leaving important machine-learning and numerical-analysis settings outside its scope.
Method
The paper develops conservative set valued fields, path differentiability, a generalized calculus, and Whitney-stratification formulas for definable nonsmooth functions and automatic differentiation.
Results
Conservative fields are classical gradients almost everywhere, characterize broad path-differentiable classes, and provide a framework for analyzing nonsmooth automatic differentiation and stochastic algorithms.
Takeaways & Limitations
The framework links nonsmooth calculus with practical automatic-differentiation systems and the analysis of algorithms used to train nonsmooth deep neural networks.
Takeaways & Limitations
Fermat’s rule requires the convex envelope: a potential minimum need not belong to the original nonconvex field value.
Abstract
from arXiv · showhide
Modern problems in AI or in numerical analysis require nonsmooth approaches with a flexible calculus. We introduce generalized derivatives called conservative fields for which we develop a calculus and provide representation formulas. Functions having a conservative field are called path differentiable: convex, concave, Clarke regular and any semialgebraic Lipschitz continuous functions are path differentiable. Using Whitney stratification techniques for semialgebraic and definable sets, our model provides variational formulas for nonsmooth automatic differentiation oracles, as for instance the famous backpropagation algorithm in deep learning. Our differential model is applied to establish the convergence in values of nonsmooth stochastic gradient methods as they are implemented in practice.
1 Introduction
The paper introduces conservative set valued fields and path differentiability to extend nonsmooth calculus beyond convex-like regularity. It connects this framework to automatic differentiation, deep learning, and stochastic optimization.
- Motivation: The paper addresses nonsmooth problems not covered by convex, semiconvex, or Clarke-regular calculus because of calculus restrictions and algorithm decomposition.These problems arise in machine learning, numerical analysis, and nonsmooth dynamical systems.
- Conservative fields: Conservative fields are graph closed set valued mappings associated with locally Lipschitz functions through a generalized derivative relation.The framework uses Aumann integration and a zero-circulation formulation.
- Path differentiability: Functions admitting a conservative field are called path differentiable, including convex, concave, Clarke regular, semialgebraic Lipschitz, and Whitney stratifiable functions.This broadens the classes of functions covered by the differential model.
- Theory: The paper develops calculus and characterization results, including almost-everywhere agreement of conservative fields with classical gradients and variational stratification formulas for definable settings.These results extend qualitative properties such as Morse-Sard, nonsmooth Kurdyka-Lojasiewicz inequalities, and convergence of differential inclusions.
- Applications: The framework analyzes nonsmooth automatic differentiation and discrete stochastic algorithms used to train deep neural networks.The paper connects automatic differentiation, including backpropagation, with semialgebraic and Whitney-stratification ideas and illustrates the approach on prediction networks.
2 Conservative set valued fields
This section defines conservative fields through vanishing circulation along absolutely continuous loops and constructs their potential functions. Conservativity is equivalently characterized by a chain rule along absolutely continuous curves.
- Setting: The analysis focuses on locally Lipschitz functions and set valued maps in Euclidean spaces, with results generalizable to complete Riemannian manifolds.Absolutely continuous curves provide the path framework used for differential inclusions.
- Definition: A conservative field has a closed graph, nonempty compact values, and zero circulation for every absolutely continuous loop.The loop integral is understood in the Lebesgue sense.
- Characterization: For a path, the maximum and minimum integrals of directional pairings over the field coincide almost everywhere.This equality supports the equivalent vanishing-circulation and Aumann-integral formulations.
- Potential functions: A conservative field defines a path-independent potential function, uniquely determined up to an additive constant.The potential is obtained by integrating along any absolutely continuous path from the origin to the target point.
- Chain rule: Conservativity is equivalent to a chain rule stating that the derivative of the composed function along every absolutely continuous curve matches the field’s directional pairing.The maximum, minimum, and selected pairings agree almost everywhere along the curve.
3 A generalized differential calculus
The paper develops conservative set-valued fields as a generalized differential calculus, characterizes path differentiability, and extends chain and sum rules to conservative mappings.
- Conservativity and Clarke subdifferentials: Conservative fields are gradients almost everywhere, and the Clarke subdifferential is minimal among convex-valued conservative fields.The framework also yields a Fermat rule in which zero belongs to the convex hull of the field at local extrema.
- Conservativity and Clarke subdifferentials: The convex envelope in Fermat’s rule is necessary: a global minimum can have 0 outside the field itself while 0 belongs to its convex hull.The absolute-value example uses D(0) = {−1, 1}.
- Path differentiability: A locally Lipschitz function is path differentiable when it is a potential of a conservative field.Equivalent characterizations connect path differentiability with conservativity of the Clarke subdifferential and its chain rule.
- Path differentiability: Convex, concave, Clarke regular, prox regular, semialgebraic, and more generally definable Lipschitz functions satisfy sufficient conditions for path differentiability.The paper notes that many functions arising in applications are path differentiable, although generic 1-Lipschitz functions are not.
- Conservative mappings and calculus: Conservative mappings support coordinate projections and composition: products of conservative mappings remain conservative, and generalized Jacobians compose through the chain rule.For g composed with F, the resulting conservative field is represented by DF(x)^T Dg(F(x)).
- Conservative mappings and calculus: The outer sum rule enlarges component fields to produce a conservative field for the sum of locally Lipschitz functions.This rule is identified as fundamental for mini-batch stochastic gradient methods.
4 Tameness and conservativity
The paper uses definability and Whitney stratification to connect conservative fields with variational projection formulas and qualitative properties. These results support applications to nonsmooth analysis and automatic differentiation.
- Tameness: Definability in o-minimal structures provides a tame framework that includes semialgebraic functions and applications such as deep learning with hyperbolic activations or entropic losses.Semialgebraic assumptions suffice for major applications including ReLU networks with square loss.
- Variational stratification: A variational stratification partitions the domain into Whitney strata where f is smooth and projection formulas relate D(x) to the active-stratum gradient.The definition uses the gradient of f restricted to the active strata containing x.
- Variational stratification: Every definable conservative field with a definable potential admits a Cr variational stratification, proved by controlling projection-formula failures on a small set.The proof extends to definable finite-dimensional manifolds and uses stratification arguments.
- Scope and limitations: Definability of a potential does not imply definability of its conservative field, while definability of the field does not generally imply definability of the potential in the same structure.Whether the potential becomes definable in a larger o-minimal structure is left open.
- Geometric and dynamical properties: For definable conservative fields, the paper establishes finite D-critical values, a nonsmooth Kurdyka-Lojasiewicz inequality, and finite length for bounded inclusion curves.The finite-length result applies when the solution curve is bounded.
5 Automatic differentiation
The paper develops a conservative-field model for nonsmooth automatic differentiation, representing closed-form computations through directed graphs and generalized chain rules. Forward and backward autodiff fields are conservative, and definable constructions recover gradients almost everywhere.
- Motivation: Automatic differentiation can fail to provide gradients when closed formulas contain nonsmooth elementary functions.Existing approaches using piecewise linear approximations may have high computational complexity.
- Conservative-field framework: Conservative fields provide a flexible nonsmooth differentiation model that supports the chain-rule structure used by automatic differentiation.The model is designed for composed functions whose components are simple closed-form functions.
- Computational model: The paper models a closed-form function as an operational evaluation program built from elementary functions and a directed parent graph.Each intermediate function depends on previously computed variables, defining the final output recursively.
- Forward mode: Forward autodiff computes a conservative field by selecting componentwise generalized derivatives and propagating them through the evaluation graph.The resulting field contains all outputs obtained across the allowed choices of component derivatives.
- Forward and backward modes: Forward and backward automatic differentiation fields are conservative for the computed function.The backward mode computes essentially the same quantity as forward mode with a lower memory and time footprint.
- Definable case: For definable component functions and fields, automatic differentiation returns the gradient almost everywhere outside finitely many smooth manifolds of dimension at most p −1.Automatic differentiation need not be convex valued, and definable selections such as Clarke subgradients can be used.
6 Algorithmic consequences and deep learning
The paper extends nonsmooth analysis to automatic differentiation and applies it to backpropagation and mini-batch stochastic gradient descent in deep learning. It proves convergence properties for bounded iterates while showing that nondifferentiable intermediate activations occur with nonzero probability.
- Mini-batch stochastic approximation: Mini-batch SGD with backpropagation has almost surely nonempty accumulation points contained in the critical set, with the loss constant on those points.The result assumes step sizes α_k = o(1/log(k)) and considers the event that the iterates remain bounded.
- Scope: The analysis is presented for feedforward networks, while the abstract automatic-differentiation and definability framework also covers more complex architectures.The stated convergence theorem specifically concerns bounded sequences under the mini-batch backpropagation algorithm.
- Deep neural networks and nonsmooth backpropagation: The deep-learning loss terms admit an algorithmic representation compatible with the paper’s automatic-differentiation framework.Each loss term is formed from the network activations and elementary loss function, enabling the backpropagation construction.
- Deep neural networks and nonsmooth backpropagation: Backpropagation computes a conservative field for each definable deep-learning loss built from definable activations and conservative maps.This supplies the nonsmooth differential object used to analyze the actual backpropagation oracle rather than replacing it with a conventional subgradient.
- Deep neural networks and nonsmooth backpropagation: Random feedforward ReLU networks can evaluate ReLU(0) during forward propagation with nonzero probability for some architectures.Thus, almost-everywhere differentiability of the final piecewise-polynomial output does not ensure that the classical chain rule applies at intermediate layers.
- Training nonsmooth neural networks: For ReLU networks with mean squared error, bounded mini-batch SGD sequences approach the BPJ-critical set with probability one.The same convergence result extends to activation and loss functions definable in a common o-minimal structure.
7 Conclusion
The conclusion advocates conservative fields as a flexible alternative to fixing a subdifferential in advance. It proposes a protocol combining decomposition-aware field selection, definability checks, and Lyapunov analysis, with potential extensions to other optimization algorithms.
- Conclusion: Conservative fields avoid the limitations that can arise from choosing a fixed subdifferential before analyzing a nonsmooth nonconvex problem.The conclusion links this flexibility to analysis and representation, including automatic differentiation.
- Protocol: The proposed protocol starts from the automatic-differentiation optimization problem and matches the conservative field to the decomposition requirements of the algorithm.Mini-batches for SGD are given as an example of decomposition requirements.
- Protocol: The protocol checks that the relevant objects are definable in a common structure, including semialgebraic, globally subanalytic, or log-exp settings.These structures are listed in descending order of frequency in the authors’ applications.
- Protocol: A Lyapunov or dissipative analysis is then used to obtain asymptotic properties for the algorithm under consideration.The conclusion specifically points to the Benaïm-Hofbauer-Sorin approach for this step.
- Extensions: The framework is suggested for mini-batch stochastic approximation methods based on discretized continuous-time systems, including heavy ball momentum, ADAM, and INDIAN.These examples are identified as methods commonly proposed or studied in the cited works.