Source-linked AI summary
Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization
Satrajit Chatterjee
TL;DR
The paper asks how gradient descent generalizes when neural networks can fit randomized labels as well as real data. It proposes the Coherent Gradients hypothesis and reports that experiments agree with its predictions, while limiting the study to simple SGD baselines.
Problem
The paper asks how gradient descent finds a well-generalizing map among maps that fit real data, despite neural networks’ ability to fit randomized labels.
Source: §1 · p. 1Method
The paper proposes that similar examples produce similar gradients that reinforce in shared directions, and uses heuristic arguments and intervention experiments to examine this hypothesis.
Source: §1 · p. 2Results
The authors report that experimental results agree with the predictions of the Coherent Gradients hypothesis.
Source: §3.3 · p. 8Takeaways & Limitations
The authors connect strong gradient directions with stability and suggest that suppressing weak directions can reduce overfitting.
Source: §1 · p. 2The study is limited to vanilla SGD on MNIST using fully connected networks.
Source: §1 · p. 3
Abstract
from arXiv · showhide
An open question in the Deep Learning community is why neural networks trained with Gradient Descent generalize well on real datasets even though they are capable of fitting random data. We propose an approach to answering this question based on a hypothesis about the dynamics of gradient descent that we call Coherent Gradients: Gradients from similar examples are similar and so the overall gradient is stronger in certain directions where these reinforce each other. Thus changes to the network parameters during training are biased towards those that (locally) simultaneously benefit many examples when such similarity exists. We support this hypothesis with heuristic arguments and perturbative experiments and outline how this can explain several common empirical observations about Deep Learning. Furthermore, our analysis is not just descriptive, but prescriptive. It suggests a natural modification to gradient descent that can greatly reduce overfitting.
1 INTRODUCTION AND OVERVIEW
The paper asks how gradient descent selects well-generalizing maps when neural networks can also fit randomized labels. It proposes that similar examples produce reinforcing gradients, biasing updates toward changes that benefit many examples.
1 INTRODUCTION AND OVERVIEW
- The paper frames its question against networks that fit randomized labels and asks how gradient descent finds a well-generalizing map among maps that fit real data.It also notes that trees can fit randomized data by putting examples into separate leaves, producing larger trees than on real data.Source: §1 · p. 1
- The Coherent Gradients hypothesis proposes that similar examples have similar gradients, whose reinforcement makes parameter updates favor changes that benefit many examples.The authors name this proposed mechanism the Coherent Gradients hypothesis.Source: §1 · p. 2
- In a two-example thought experiment, a shared gradient component is reinforced, making the overall gradient stronger in the direction that helps both examples.The other components are example-specific and orthogonal to one another.Source: §1 · p. 2
- The notion of example similarity changes during training, starting mostly task-independent at random initialization and becoming task-dependent as parameters change.Even at initialization, syntactically close examples may be treated similarly.Source: §1 · p. 2
- The authors connect strong gradient directions to algorithmic stability: one example’s presence or absence affects them less than weak directions, and stability is linked to generalization.They suggest suppressing weak directions as a way to reduce overfitting.Source: §1 · p. 2
- The study uses vanilla SGD on MNIST with fully connected networks and proposes intervention experiments, replicating observations (a)–(c) while outlining explanations for (d)–(f).The authors state that the similarity notion changes with network parameters, making direct experimental verification challenging.Source: §1 · p. 3
2 EFFECT OF REDUCING SIMILARITY BETWEEN EXAMPLES
The section tests whether reducing example similarity through label noise weakens learning and gradient alignment, then examines how pristine and corrupt examples contribute to loss reduction. The results broadly match the predictions, while the authors describe some contribution comparisons as suggestive rather than conclusive.
2.1 SETUP
- The experiment adds 25%, 50%, 75%, or 100% randomized label noise to MNIST while keeping the test labels unchanged and other training aspects fixed.The setup uses a fully connected one-hidden-layer network and vanilla SGD; pristine labels remain unchanged, while corrupt labels are randomized.Source: §2.1 · p. 3, §2 · p. 3
2.2 QUALITATIVE PREDICTIONS
- The heuristic predicts that greater agreement among per-example gradients speeds loss reduction; with mixed labels, pristine examples should learn faster, though fewer pristine examples weaken their learning rate.The authors relate this to prior observations that sufficiently many clean examples can permit learning despite noisy labels greatly outnumbering them [Rolnick et al., 2017].Source: §2.2 · p. 4
2.3 AGREEMENT WITH EXPERIMENT
- As label noise increased, learning to a given accuracy took longer; pristine examples were learned faster than corrupt ones, and their learning rate fell as their number decreased.All five variants reached 100% training accuracy but showed different generalization; the authors also report that early stopping can limit the impact of weak gradients.Source: §2.3 · pp. 4–5, Figure 1 · p. 5
2.4 ANALYZING STRONG AND WEAK GRADIENTS
- At 25% and 50% noise, pristine examples initially contributed more to point-in-time loss reduction, then contributions crossed over; at 75%, corrupt examples' larger count could dominate despite weaker individual gradients.The analysis estimates contributions from sampled per-example gradients and compares actual designations with randomized pristine/corrupt designations.Source: §2.4 · p. 6, Figure 2 · p. 6
- In the first 10 training steps, the mean cumulative pristine–corrupt contribution difference appeared somewhat significant, but not overwhelmingly so.The authors note that analysis over the full population of examples and trainable parameters would be useful.Source: Figure 3 · p. 7, §2.4 · p. 7
3 EFFECT OF SUPPRESSING WEAK GRADIENT DIRECTIONS
The section tests whether suppressing weak gradient directions through winsorization reduces overfitting, while examining the trade-off with fitting the training data. Experiments broadly follow the predictions, but heavy winsorization can stop updates from being descent steps.
3 EFFECT OF SUPPRESSING WEAK GRADIENT DIRECTIONS
- The intervention suppresses weak gradient directions by clipping per-example gradient outliers independently for each trainable parameter.The c-winsorized gradient replaces the usual gradient in each parameter update.Source: §3 · p. 7, §3.1 · p. 8
3.1 SETUP
- The implementation uses minibatch examples to set clipping thresholds, making c correspond to the number of high and low outliers clipped per minibatch.The smaller network and training schedule also differ from the earlier baseline.Source: §3.1 · p. 8
3.2 QUALITATIVE PREDICTIONS
- The predictions are that larger c reduces overfitting but also makes training accuracy harder to fit, while small c can leave residual overfitting.Winsorization is expected to suppress only the most extreme outliers, not eliminate all weak directions.Source: §3.2 · p. 8
3.3 AGREEMENT WITH EXPERIMENT
- For c > 1, training accuracy does not exceed the dataset’s proper accuracy, and the overfit curve grows more slowly as c increases.Figure 4 varies label noise by column and winsorization by row; overfit adjusts for test labels that were not randomized.Source: §3.3 · p. 8, Figure 4 · p. 9
- With large winsorization, training and test accuracy eventually decline, partly because updates cease to be descent steps.At that point, the procedure is no longer gradient descent.Source: §3.3 · p. 8
4 DISCUSSION AND RELATED WORK
The discussion frames generalization as a property of optimization dynamics and transitions, rather than solely of the solutions reached. It connects this view to stability while distinguishing it from complexity-based accounts and related descriptive work.
4 DISCUSSION AND RELATED WORK
- The discussion notes that no entirely satisfactory explanation of deep-learning generalization has emerged, despite substantial work on optimization landscapes and SGD stochasticity.It points to similar qualitative generalization with full-batch training in a modified experiment, while noting the absence of large-scale full-batch studies known to the authors.Source: §4 · p. 9
- The authors view optimization as a search whose feasibility means preserving generalization throughout parameter transitions.This shifts attention from solution quality to whether the search dynamics preserve generalizability.Source: §4 · p. 9
- Classical complexity measures can fail to explain practical generalization, motivating newer measures that the authors also judge unsatisfactory.Networks can memorize random labels and still generalize on real data, making VC-dimension and Rademacher-complexity bounds vacuous in this setting.Source: §4 · pp. 9–10
- The authors favor algorithmic stability as a framework, arguing that practical non-vacuous bounds may need to account for the training data.They note that formal results on GD and SGD do not explain practical settings such as multiple epochs and non-convex objectives, and that stability-based analysis could also guide algorithms.Source: §4 · p. 10
- Related work studies learned-function biases or gradient statistics, whereas this paper proposes a mechanism for detecting commonality and links it to interventions.The authors contrast their causal-mechanism focus with low-frequency learning [Rahaman et al. (2019)] and descriptive statistics such as stiffness [Fort et al. (2019)] and gradient confusion [Sankararaman et al. (2019)].Source: §4 · p. 10
5 DIRECTIONS FOR FUTURE WORK
The proposed future work includes testing Coherent Gradients in larger architectures, studying how example similarity evolves, and exploring explanations and algorithms built from the hypothesis.
5 DIRECTIONS FOR FUTURE WORK
- Testing whether Coherent Gradients holds for BERT and ResNet requires more computationally efficient tests, and the evolution of example similarity remains to be studied.The authors suggest non-parametric similarity methods and characterizations of early-learned examples as useful directions.Source: §5 · p. 10
- The authors suggest that Coherent Gradients may explain why adversarial initializations fit examples case by case and fail to generalize.They propose that semantically similar examples initially appear different, so their gradients share less in common.Source: §5 · p. 10
- They propose combining the Lottery Ticket Hypothesis [Frankle & Carbin, 2018] with Coherent Gradients to explain why wider networks generalize better.Wider networks may offer more initial gradient directions that improve many examples, which are then learned preferentially.Source: §5 · p. 11
- Open questions include efficient winsorized SGD with generalization or privacy guarantees, comparison with Abadi et al. (2016), and algorithms for discrete networks.Source: §5 · p. 11