Source-linked AI summary
Fisher information and the geometry of memorization in neural networks
Daniel T. Bernstein, Chase Goddard, Luca Di Carlo, Vudtiwat Ngampruetikorn, David J. Schwab
TL;DR
The paper asks whether Fisher information can distinguish generalizing structure from memorized or corrupted training features. It probes Fisher-based projections across controlled-noise vision, synthetic in-context learning, and a linear model, finding that the geometry separates memorization from generalization and can partially recover task generalization.
Problem
The paper investigates whether Fisher geometry can distinguish generalizable structure from idiosyncratic or corrupted training features, an important distinction for understanding model generalization.
Source: §1 · p. 1Method
The authors study weight projections onto Fisher-governed subspaces across controlled-noise vision, synthetic in-context regression, and an analytically tractable linear model.
Source: §1.1 · p. 2, §1 · p. 2Results
Across several settings, Fisher geometry separates memorized from generalized information, and suppressing memorized directions can partially recover task generalization otherwise lost to overfitting.
Source: §1 · p. 2, §6 · p. 10Takeaways & Limitations
The findings support a geometric account in which learning a task and memorizing data can be decoupled in some models.
Source: §6 · p. 10The empirical scope is ResNet-20 classifiers on CIFAR-10 variants and small synthetic-task transformers; layerwise separation may change in larger models, and cross-layer correlations are not analyzed.
Source: §6 · p. 10
Abstract
from arXiv · showhide
A fruitful approach towards understanding generalization in machine learning involves characterizing how models separate generalizable structure from idiosyncratic or even corrupted features of training data. Fisher information has emerged as a diagnostic tool for this separation, leveraging loss geometry to distinguish generalization from memorization. Here we investigate this connection in vision models trained on CIFAR-10 with controlled noise that can be learned only through memorization. We show that projecting model weights onto the top Fisher eigenvectors decouples general task performance from noisy sample memorization, and we investigate how this decoupling presents differentially across layers. We show that the same phenomena are present in small transformers trained to solve a synthetic in-context learning task, and are reproduced in closed form in a simple, analytically solvable linear model. Our results reveal that Fisher information captures aspects of the geometric structure underlying how neural networks allocate capacity between generalization and memorization.
1 Introduction
The paper tests whether Fisher geometry separates sample-specific memorization from features shared across the training distribution. It examines how that separation depends on data and layers, across image, synthetic in-context, and linear settings.
1 Introduction
- Sample-specific directions should be suppressed in the averaged Fisher, while directions encoding shared features add coherently, making its top eigenspectrum a candidate site of separation.The experiments test whether projecting onto top Fisher eigenspaces preserves shared structure while suppressing information tied to individual examples.Source: §1 · pp. 1–2
1.1 Contributions
- Low-eigenvalue Fisher subspaces encode memorized data across image-noise, layer-wise, and in-context regression settings.Source: §1.1 · p. 2
- De-memorization depends on which samples are used to compute the Fisher, so its eigenspace structure is data-dependent.Source: §1.1 · p. 2
- In synthetic in-context regression, de-memorization extends to latent tasks, can improve task generalization, and identifies components responsible for task memorization.Source: §1.1 · p. 2
- Single-sample Fisher projections probe individual-example de-memorization, while participation ratios distinguish how sample types occupy the eigenspectrum.Source: §1.1 · p. 2
- The analytically tractable barcode model reproduces de-memorization in a simple linear setting.It is designed to analyze finite-sample correlations with features that are uninformative across the population.Source: §1.1 · p. 2
2 Related Work
Related work connects loss geometry and Fisher information to memorization and generalization, alongside research on noisy labels, in-context learning, benign overfitting, and unlearning.
Curvature-based signatures of memorization
- Merullo et al. [4] report that top Fisher modes encode reasoning-like structure while lower-curvature modes encode memorized samples; other curvature tools target mislabeled examples and influential training data.Source: §2 · pp. 2–3
Memorization and noisy labels
- Networks can fit arbitrary labels and random inputs, often learn shared structure first, and may benefit from memorizing atypical examples.Source: §2 · p. 3
Loss geometry, sharpness, and Fisher information
- Fisher information is a local metric of model sensitivity related to loss curvature, while prior work also links flat directions to more global connections between minima.Source: §2 · p. 3
In-context learning and task overfitting
- Transformers can first generalize to held-out regression tasks and later overfit finite training-task sets, with memorization localized to MLP layers.Source: §2 · p. 3
Benign overfitting and data coarse-graining
- In benign-overfitting settings, removing uninformative feature directions after fitting noise can reduce finite-sample risk; the barcode model isolates such population-uninformative features correlated with potential label noise.Source: §2 · p. 3
Unlearning
- Unlearning seeks to force forgetting of specific training information, often for privacy or harmful-output concerns, whereas the model editing studied here does not enable fine-grained deletion.Source: §2 · p. 3
3 Experiments
The experiments force memorization with Gaussian inputs or flipped labels, then project weights onto top Fisher subspaces across layers; a synthetic in-context learning task tests latent-task memorization. Separation depends on layer and the data used to compute the Fisher, while MLP projections in the in-context task improve held-out-task performance at a cost to memorized-task performance.
3.1 Which layers store memorized information?
- At k≈45, CIFAR-10 train and test accuracy are near maximum while less than 30% of Gaussian samples are classified correctly in the first layer.The train and test curves rise in tandem, distinguishing structured CIFAR-10 data from the memorized Gaussian inputs.Source: §3.1 · p. 4, Figure 1 · p. 4
- In the last layer, CIFAR-10 train, CIFAR-10 test, and Gaussian accuracy rise together, so selective removal is no longer possible.The representations of the two data distributions have converged by this layer.Source: §3.1 · p. 4, Figure 1 · p. 4
3.2 Memorization and generalization
- With 10% label noise, first-layer top Fisher subspaces distinguish clean and flipped-label samples under both Fisher datasets, and true-label accuracy on flipped samples rises above chance.The projection does not completely remove the memorized samples.Source: §3.2 · p. 5, Figure 2 · p. 5
- In the penultimate convolutional layer, clean-data Fisher preserves stronger separation than full-training-set Fisher, showing that the Fisher geometry depends on its evaluation data.With the full Fisher, the accuracy curves move closer together.Source: §3.2 · p. 5, Figure 2 · p. 5
- For the block-diagonal full-model Fisher, clean-label data yields top eigenspaces selective for the true task, whereas full-training data mixes memorization and generalization information.The layer-wise separation is not retained by the full-model approximation when corrupted labels are included.Source: §3.2 · p. 6, Figure 3 · p. 6
3.3 In-context learning
- Roughly half the gap to best early-stopped performance is recovered by MLP Fisher projection, which improves held-out task performance while reducing memorized-task performance.In the second transformer block, attention-layer projections instead decrease performance on both task types.Source: §3.3 · pp. 6–7, Figure 4 · p. 7
4 Contributions of Individual Samples
Individual-sample Fisher contributions distinguish structured images from memorized Gaussian inputs in the eigenspectrum, while single-sample projections preferentially reduce recovery of noisy samples.
4.1 Sample Contributions to the Eigenspectrum
- Participation ratios quantify how broadly samples contribute to each eigenspectrum mode, with restricted ratios counting only CIFAR-sample contributions.Equal contributions from all samples yield a ratio of 1; a mode contributed to by one sample yields 1/n.Source: §4.1 · p. 7
- In input-noise experiments, clean CIFAR-10 images dominate top Fisher modes while memorized Gaussian inputs contribute mainly to bottom modes; label-noise samples contribute at a constant ratio throughout.Total participation ratios fall by roughly one order of magnitude from top to bottom modes in the input-noise case.Source: §4.1 · p. 7
4.2 Single-Sample Projections
- Sample projection computes a sample’s single-sample Fisher and projects out its top eigenvectors; noisy samples are less likely to be recovered than their counterparts.Figure 5C–D reports self-hit rates after removing the top-k subspace of the corresponding sample Fisher.Source: §4.2 · p. 7, Figure 5 · p. 8
- After single-sample projection, clean images retain higher classification accuracy than Gaussian inputs, while corrupted-label samples recover true labels more often than assigned labels.True-class recovery is non-monotonic in the number of removed dimensions, and collateral degradation is generally small except for Gaussian inputs after projecting a Gaussian sample.Source: §4.2 · p. 8
5 A barcode model of label-noise memorization
The paper analyzes label-noise memorization with a finite-rank linear model containing informative signal features and uninformative barcode features. Post-hoc Fisher projection can reduce prediction risk, but when signal and barcode directions mix, the best retained rank depends on the noise regime.
5 A barcode model of label-noise memorization
- The linear model yields exact expressions for risk reduction from projection and identifies a regime where an interior-optimal projection rank appears.The figure compares theoretical curves with simulations and illustrates prediction MSE against retained Fisher rank.Source: §5 · p. 8, Figure 6 · p. 9
- The model forms each input by concatenating anisotropically scaled informative sign features with isotropic Gaussian barcode features, while training labels receive random per-dimension sign flips.Prediction risk is evaluated on fresh samples from the clean generative model.Source: §5 · pp. 8–9
- Post-hoc barcode deletion reduces prediction risk, and the model’s theoretical risk-gain prediction agrees with numerical simulations across label-flip probabilities.The analysis deletes the barcode block after fitting both informative and barcode features, rather than removing barcodes before fitting.Source: §5 · pp. 9–10
- At low flip rates, full retained rank is best; at higher rates, an interior-optimal rank emerges as memorization variance trades off against leakage of informative features.When informative-feature variance is below 1 + d/(n-r), its empirical eigenvector merges into the barcode bulk, so discarding low-eigenvalue directions also sacrifices predictive information.Source: §5 · p. 10
6 Discussion and Outlook
The discussion frames memorization and generalization as separated by Fisher geometry, with interventions that can remove noisy-sample recall while preserving structured-data task performance. It also emphasizes that this geometric account has important scope limits and leaves cross-layer correlations unmodeled.
6 Discussion and Outlook
- Fisher geometry separates memorized from generalized information, and single-layer interventions can de-memorize noisy samples without degrading task performance on structured data.Layerwise effects also accumulate in the Fisher information of the full model.Source: §6 · p. 10
- The geometric signature depends strongly on Fisher-evaluation data: denoised training data makes memorization easier to disentangle, and breakdown occurs in particular layers.The discussion also reports cases where removing components improves performance on tasks benefiting from generalization, without retraining.Source: §6 · p. 10
- The empirical scope is ResNet-20 on CIFAR-10 variants and small transformers on synthetic in-context regression; larger models and other modalities may differ, and cross-layer correlations are ignored.The authors caution that the results are an exploration, not a scale-independent model-editing prescription.Source: §6 · p. 10
- The authors present the results as evidence for models that can reason without retaining particular information, supported by examples separating task learning from data memorization.They describe the work as demonstrating a geometric mechanism for this decoupling.Source: §6 · p. 10
A.1 Model Training
The experiments use separately specified ResNet-20 and transformer setups, while supplemental figures extend layerwise Fisher projections across input-noise, label-noise, and control settings.
A.1 Model Training
- ResNet-20 models train for 200 epochs with batch size 128, SGD, and learning-rate reductions at epochs 100 and 150.BatchNorm parameters are re-estimated after each projection.Source: §A.1 · p. 13
- The in-context-learning transformer has three GPT-2-style blocks, hidden dimension 32, eight attention heads per block, and dropout 0.1.Training and test sequences contain 20 (x, y) pairs, with x in R^5.Source: §A.1 · p. 13
B Supplemental figures
- Supplemental figures show layerwise projections for input noise and label noise, using both clean-sample and full-training-set Fishers, plus SVD and random-basis controls.Figure 8 covers the transformer blocks; its Block 2 MLP improves test performance at the cost of training-task performance, while other layers worsen on both tasks.Source: B Supplemental figures · p. 14, Figure 7 · p. 14, Figure 8 · p. 15
C Barcode model derivations
The derivation estimates signal and barcode coefficients by squared-loss minimization, then derives risk gain from barcode deletion under stated distributional assumptions.
C.1 Estimating block coefficients
- The block-coefficient estimators are obtained by minimizing squared loss between corrupted targets and their signal- and barcode-block predictions.The derivation substitutes the resulting expression into the loss and sets its derivative to zero.Source: §C.1 · p. 18
C.2 Risk reduction from barcode deletion
- The risk-gain derivation averages over training data and a fresh test point, using a zero-mean, identity-covariance barcode independent of the other variables.It conditions on the design and averages over the noise before evaluating the expression.Source: §C.2 · p. 18
- For m > d + 1, the projected barcode covariance has a Wishart form whose inverse-Wishart mean completes the derivation of Eq. (7).The derivation uses m = n − r remaining degrees of freedom after the signal rows are zeroed out.Source: §C.2 · p. 18
C.3 Partial-overlap prediction MSE
The partial-overlap analysis explains how imperfect spectral separation mixes signal and barcode directions, making projection rank a trade-off between preserving signal and removing noise.
C.3 Partial-overlap prediction MSE
- Below the population threshold, empirical eigenvectors mix signal and barcode directions, so projection rank K trades off signal loss against noise removal.A signal coordinate separates from the barcode bulk only above the population threshold.Source: §C.3 · p. 19
- The post-hoc projected predictor applies the rank-K projection to the fitted estimator’s input.The estimator includes an infinite-sample coefficient on corrupted targets.Source: §C.3 · p. 19
- The prediction MSE includes target mismatch, a mixing penalty amplified for sub-threshold signals, and noise variance retained by the projection.The mixing penalty peaks at overlap A_jj = 1/2 and vanishes when eigenvectors are pure coordinate axes.Source: §C.3 · p. 19
- An interior optimal rank requires heterogeneity or mixing; increasing K below the separation threshold recovers signal while admitting noise.With ideally separated pure signal and barcode directions, the optimal rule keeps signal directions and discards barcode directions.Source: §C.3 · p. 19
D.1 Model accuracy in single-sample projection experiments
Table 2 reports bulk accuracy after sample projection and specifies the accuracy scale and BatchNorm control procedure.
D.1 Model accuracy in single-sample projection experiments
- Table 2 reports bulk accuracy after sample projection.The listed accuracies are percentages.Source: Table 2 · p. 20
- BatchNorm is reset after each projection and recalibrated by cumulative averaging on equal batches of 250 images.Source: Table 2 · p. 20
- The k = 0 column is the matched unchanged-weight BatchNorm control.Source: Table 2 · p. 20