Beyond neural scaling laws: beating power law scaling via data pruning

Motivation

• Data pruning can potentially replace weak power-law improvements from collecting random data with exponential error scaling using carefully selected training examples.The motivation is that power-law gains demand rapidly increasing data, compute, and energy, whereas pruning targets redundant examples.Source: §1 · pp. 1–2

Theory

• A statistical-mechanics theory for perceptron learning quantitatively matches numerical experiments and predicts how pruning strategy should depend on initial dataset abundance.With abundant data, retain hard examples; with scarce data, retain easy examples.Source: §1 · pp. 2–3, Figure 1 · p. 2
• The theory predicts exponential scaling with pruned dataset size when the pruning fraction increases according to a Pareto-optimal schedule as initial dataset size grows.This condition enables the predicted improvement beyond ordinary power-law scaling.Source: Figure 1 · p. 2, §1 · p. 3

Empirical validation

• Experiments show signatures of exponential error scaling for pruned datasets across ResNets trained on SVHN, CIFAR-10, and ImageNet, plus Vision Transformers fine-tuned on CIFAR-10.These tests examine whether the theoretical predictions extend beyond the perceptron setting.Source: §1 · p. 3

Metric benchmark

• A large-scale ImageNet benchmark of 10 pruning metrics finds that most perform poorly, while the strongest exceptions are computationally intensive.The results highlight the difficulty of scaling existing high-performing pruning metrics to ImageNet.Source: §1 · p. 3

Self-supervised metric

• A new cheap self-supervised pruning metric requires no labels yet performs comparably to the best supervised metrics that require labels and substantially more computation.This enables pruning potentially before new datasets are labeled, including through pretrained foundation models.Source: §1 · p. 3

2.1 Pruning metrics: not all training examples are created equal

• Existing pruning approaches include EL2N, forgetting, margins, memorization, influence, clustering, and ensemble-based scores, but expensive computation and limited ImageNet-scale exploration have constrained their use.[10] retained only the hardest 50% of CIFAR-10 and 75% of CIFAR-100 without loss in final test accuracy; [21] and reached 90% and 80% retained ImageNet data without sacrificing accuracy.Source: §2.1 · pp. 3–4
• Prior pruning metrics retain hard examples after ranking training points by difficulty, but most require labels and lack ImageNet-scale validation or theory.The paper introduces an unsupervised metric benchmarked at ImageNet scale and develops margin-pruning theory predicting exponential scaling and a possible advantage for retaining easy examples when data is scarce.Source: §2.1 · pp. 3–4

2.2 Neural scaling laws and their potential inefficiency

• Neural test loss commonly follows slow power laws in model size, data, and compute, with small exponents making resource-intensive improvements potentially inefficient.For a large transformer language model, ν = 0.095 means a tenfold data increase reduces cross-entropy loss by only about 0.6 nats; jointly scaling model size and data 10× under 100× more compute reduces it by about 0.5 nats.Source: §2.2 · p. 4

2.3 Statistical mechanics of perceptron learning

• Statistical-mechanics analyses established power-law perceptron learning and studied active selection of labeled inputs, whereas this work analyzes practical margin-based pruning from a fixed dataset.Earlier message-passing methods target Gaussian inputs and perceptrons and are difficult to generalize to real-world settings.Source: §2.3 · pp. 4–5

Analytic predictions

• The optimal pruning strategy switches with initial data: retain easiest examples when data are scarce, but hardest examples when abundant.Easy examples provide coarse-grained information and reduce overfitting in the limited-data regime, whereas hard examples better resolve the teacher’s decision boundary when data are plentiful.Source: §3 · p. 5

Empirical prediction

• The predicted transition between easiest- and hardest-example pruning also appears in ResNet18 experiments on pruned CIFAR-10 datasets.The transition becomes sharper at smaller retained fractions f, suggesting that the perceptron-theory prediction extends beyond perceptron learning.Source: §3 · p. 5

Analytic predictions

• Pareto-optimal pruning can achieve at least exponential test-error scaling in pruned dataset size by increasing the information gained per retained example.Aggressively pruning larger initial datasets produces an optimal fraction f_opt that decreases with pruned dataset size; the theory predicts faster-than-exponential decay when probe information is included.Source: §3 · p. 6

Information-theoretic perspective

• Random pruning follows the classical ε ∝ α_prune^-1 power law because each additional example contributes progressively less information about the decision boundary.For fixed pruning fraction f, the information gain eventually decays as a power law; under more aggressive Pareto-optimal pruning it can converge to I(∞) = 1 nat/example.Source: §3 · p. 6

Imperfect pruning metrics

• With an imperfect metric, scaling initially appears exponential but crosses over to a power law once pruning exceeds a minimum useful fraction f_min(θ).For θ = 10° (θ = 20°), the asymptotic minimum fractions are 24% (46%); as θ approaches 0, f_min(θ) approaches 0.Source: §3 · pp. 6–7

Scaling experiments

• Better-than-power-law scaling emerged on SVHN, CIFAR-10, and ImageNet, especially with larger initial datasets and more aggressive pruning.The experiments verified predictions that retaining hardest examples can help at large dataset sizes, while optimized pruning traces a lower envelope beyond power-law scaling.Source: §4 · p. 7

Transfer learning

• Fine-tuning on 10% of CIFAR-10 matched or exceeded fine-tuning on all CIFAR-10 after ImageNet21K pre-training.Pre-trained models therefore permit more aggressive downstream data pruning while maintaining or improving performance.Source: §4 · p. 8
• Pre-training on 50% of ImageNet matched or exceeded CIFAR-10 performance from pre-training on all ImageNet.This shows that pruning upstream pre-training data can preserve performance on a different downstream task.Source: §4 · p. 8

ImageNet-scale pruning

• At ImageNet scale, training on 80% approximated training on 100% across supervised and self-supervised prototype pruning comparisons.The figure compares existing supervised metrics, memorization, and supervised and self-supervised prototype metrics using top-5 validation accuracy.Source: Figure 5 · p. 8

Benchmark design

• The benchmark systematically evaluated eight supervised metrics plus two new prototypicality metrics, spanning influence scores, EL2N, DDD, memorization, ensemble active learning, and forgetting.The study addressed uncertainty about how pruning methods scale to ImageNet and which method performs best.Source: §5 · p. 8

Metric comparison

• Metric rankings were substantially diverse, although EL2N, DDD, and memorization were similar with pairwise Spearman correlations above 0.7.Performance differences remained marked despite these ranking similarities.Source: §5 · p. 8

Results

• At about 80% of ImageNet, only a few metrics matched full-dataset training, while most outperformed random pruning and memorization performed strongly.ImageNet pruning may be harder because the dataset is already curated to remove uninformative examples.Source: §5 · pp. 8–9

Experimental controls

• All pruning metrics amplified class imbalance and degraded performance, so the ImageNet experiments used a simple 50% class-balancing ratio.Baselines without class balancing were reported separately in Appendix H.Source: §5 · p. 9

• Existing ImageNet pruning metrics either scale poorly or require substantial compute, and all require labels, motivating simple scalable self-supervised alternatives.Label dependence limits pruning for large-scale foundation models trained on massive unlabeled datasets [12].Source: §6 · p. 9
• The self-supervised prototype metric matches or exceeds memorization until only 70–80% of ImageNet data is kept, without labels and with simpler, cheaper computation.It ranks difficulty by cosine distance to the nearest k-means centroid in SWaV embedding space: prototypical examples are easy, while distant examples are hard.Source: §6 · p. 9
• Supervised prototypes outperform other supervised metrics and largely match memorization, while the best self-supervised and supervised metrics achieve similar performance.Supervised prototypes average class embeddings into one prototype per class; memorization is prohibitively expensive to compute.Source: §6 · p. 9
• Prototype-metric performance is robust to the cluster count: k can vary one order of magnitude above or below ImageNet’s 1000 classes without affecting results.Source: §6 · p. 9
• Across visualized examples, easy images are highly similar and redundant, whereas hard images appear as idiosyncratic outliers under both prototype and memorization metrics.Figure 6 compares the four easiest and hardest images for ImageNet’s black swan class under both metrics.Source: §6 · p. 9, Figure 6 · p. 9

7 Discussion

• Data pruning achieves faster-than-power-law, potentially exponential error scaling, and the self-supervised metric discards 20% of ImageNet without sacrificing performance.Its performance is on par with the best and most compute-intensive supervised metric.Source: §7 · p. 10
• Exponential scaling depends on high-quality pruning metrics, most existing metrics scale poorly to ImageNet, and future work should develop scalable metrics.Pruning can reduce computation, although matching full-dataset iterations may add epochs; gains saturate before training time approaches full-data training, and class balancing is essential.Source: §7 · p. 10
• Preliminary analyses found no substantial differential effects across ImageNet classes, but deployment-specific fairness tests remain necessary for pruned and unpruned models.The paper also analyzes pruning effects on out-of-distribution performance.Source: §7 · p. 10
• Scalable unsupervised pruning on larger datasets could enable more aggressive pruning and reusable foundation datasets whose selection cost is amortized across downstream models.The proposed direction targets massive unlabeled datasets used for foundation-model training.Source: §7 · p. 10

A A theory of data-pruning for the perceptron: detailed derivations

• The appendix provides reproducible Colab code for the theory figures and numerical simulations.The implementation is supplied at the linked notebook in the passage.Source: A A theory of data-pruning for the… · p. 14

A.1 Problem setup

• The perceptron theory studies teacher-generated Gaussian classification in the high-dimensional limit N, P →∞ with fixed O(1) sample ratio α_tot = P/N.Inputs are isotropic Gaussian vectors, labels are generated by a randomly oriented teacher, and pruning introduces non-isotropic structure.Source: §A.1 · p. 14
• The theory models pruning as retaining examples with the smallest probe-student margins, yielding distribution p(z) along the probe direction while preserving isotropy in its nullspace.It allows arbitrary p(z) and probe–teacher overlap before specializing to small-margin pruning with a truncated Gaussian.Source: §A.1 · p. 15
• A new student is trained from scratch on the pruned data using the maximal-margin solution, with generalization error ε_g = cos^-1(R)/π determined by student–teacher overlap R.This setup covers algorithms such as support vector machines and SGD on separable data.Source: §A.1 · p. 15

A.3 Replica calculation of the generalization error; A.3.1 Replica symmetry ansatz; A.4 Quenched entropy

• Replica theory computes the quenched Gardner entropy by replicating solution volumes, introducing overlaps with the teacher and probe student, and applying replica symmetry.The Gardner volume shrinks to the unique max-margin solution at critical κ; the quenched entropy separates into entropic and energetic contributions.Source: §A.3 · pp. 15–17, §A.3.1 · p. 22, §A.4 · p. 24

A.5 Perfect teacher-probe overlap; A.5.1 Saddle point equations

• For perfect teacher-probe overlap, the saddle-point equations reduce as q approaches one and recover the classical teacher-student perceptron when pruning vanishes.The reduced equations solve numerically for R and κ under arbitrary teacher-direction data distributions and fit Fig. 1A simulations.Source: §A.5.1 · pp. 24–25, §A.5 · p. 24

A.6 Information gain per example

• Aggressive pruning makes information gain per example approach a finite rate, yielding at least exponential error decay; for R →1, the asymptotic gain is 1 nat/example.Classical unpruned information gain vanishes as I(α) ∼1/α, whereas Pareto-optimal pruning maintains finite gain.Source: §A.6 · pp. 25–26

A.7 Imperfect teacher-probe overlap

• With imperfect teacher-probe overlap, pruning eventually stops concentrating data near the teacher boundary, forcing information gain to zero and reverting generalization to a power law.The resulting lower envelope is characterized by a minimum pruning fraction fmin(θ), which approaches zero as the probe angle shrinks.Source: §A.2 · p. 15, §A.7 · p. 26

A.8 Optimal pruning policy

• The optimal pruning policy switches from retaining easy, large-margin examples at small αprune to hard, small-margin examples at large αprune, with an intermediate window interpolating between them.Calculus of variations gives a delta-function optimum, while a realistic Gaussian window shifts from infinity toward zero as αprune grows and lowers the Pareto frontier.Source: §A.8 · pp. 26–28, Figure 7 · p. 27
• The theoretically optimal delta distribution can recover the teacher exactly for any αprune, but this requires infinite data and precise selection, limiting practical relevance.The realistic analysis therefore models pruning as retaining examples within a finite window a < z < b.Source: §A.8 · pp. 27–28

A.7 Imperfect teacher-probe overlap; A.9 Exact saddle point equations

• The full equations over R, q, and ρ extend the analysis to arbitrary probe-teacher angles and provide the numerical predictions underlying the structured-data results.They are obtained by extremizing the quenched entropy and integrating by parts to obtain three saddle-point equations.Source: §A.7 · p. 26, §A.9 · pp. 28, 30

A.2 Main result and overview; A.9 Exact saddle point equations

• Solving the exact saddle-point equations numerically yields excellent agreement with simulations on structured data, while the general formulation gives εg = cos−1(R)/π for arbitrary α, p(z), and θ.The equations determine R, ρ, and κ, and produce the theory curves in Figs. 1–3.Source: §A.2 · p. 15, §A.9 · p. 31

• Synthetic perceptrons used Gaussian inputs, teacher-generated labels, fixed dimension N = 200, and datasets with P = αN examples across α = 100.1–100.5.Datasets retained fractions f of smallest-margin examples, and maximum-margin solutions were found with CVXPY quadratic programming, averaging 100 independent draws.Source: B Model training method details &… · p. 31
• ImageNet scaling experiments trained ResNet-50 models across kept-data fractions 0.1–1.0, matched to 105 full-dataset epochs, requiring 97 models.Training used VISSL, eight NVIDIA V100 GPUs, SGD, learning rate 0.1, Nesterov momentum 0.9, weight decay 0.001, and scheduled decays after 30, 60, 90, and 100 epochs.Source: B Model training method details &… · p. 31
• The main ImageNet pruning benchmark trained ResNet-50 models for five kept-data fractions across 11 metrics, requiring 55 models, with repeated runs used to estimate variation.At full data, performances were averaged into one datapoint; Figure 5C additionally used five models per datapoint and metric.Source: B Model training method details &… · pp. 31–32, Table 1 · p. 32
• CIFAR-10 and SVHN used ResNet-18 models, with EL2Ns pruning scores averaged over 10 probe initializations before fresh 200-epoch training on each pruned dataset.Training used SGD with learning rate 0.1, Nesterov momentum 0.9, weight decay 0.0005, and learning-rate drops after epochs 60, 120, and 160.Source: B Model training method details &… · p. 32
• Transfer experiments evaluated pruning downstream CIFAR-10 fine-tuning data for ImageNet21k-pretrained ViTs and upstream ImageNet1k pretraining data for ResNet-50s.EL2Ns scores were averaged across 10 random seeds; ViTs used Adam for 10 fine-tuning epochs, while ResNet-50s used RMSProp for 10 epochs on all CIFAR-10.Source: B Model training method details &… · p. 32

C Breaking compute scaling laws via data pruning

• Data pruning breaks compute scaling laws for perceptrons and yields preliminary evidence of similar benefits for ResNets on CIFAR-10 and ImageNet.Perceptron convergence time scales as T = 0.96α_prune + 0.80; with fixed-epoch training, pruned ResNet learning curves appear strikingly similar while using proportionally less compute.Source: Figure 8 · p. 33, C Breaking compute scaling laws via… · p. 33

E Extremal images according to different metrics

• Additional visualizations compare easy and hard images selected by different pruning metrics across multiple ImageNet classes.Figures 12–19 show extremal images for classes 100, 200, 300, and 500, while avoiding classes 0 and 400 because they predominantly contain people.Source: E Extremal images according to… · p. 33, A B C · p. 34, Figure 12 · p. 36

D Additional scaling experiments

• Additional ImageNet metrics show signatures of breaking power-law scaling, while a weak SVHN pruning metric retains a power-law lower bound that a stronger metric surpasses.The additional ImageNet effects are less dramatic than for memorization; the stronger SVHN probe was trained for 40 epochs versus 4 epochs for the weaker probe.Source: D Additional scaling experiments · p. 33, Figure 9 · p. 34, Figure 10 · p. 34

A B C

• Fitting the empirical ResNet18-on-SVHN Pareto frontier shows that an exponential form fits substantially better than a power law, indicating at least exponential scaling.The frontier was estimated from 300 additional models trained around the frontier region.Source: Figure 11 · p. 35

F Impact of number of clusters k on self-supervised prototypes

• Prototype performance is impaired by very small or very large k, but remains relatively high when k stays within one order of magnitude of the dataset’s class count.For ImageNet-1K, the default is k = 1000; ±0.54% corresponds to approximately two standard deviations from repeated training variation.Source: F Impact of number of clusters k on… · p. 44, Table 2 · p. 44

G Impact of ensemble prototypes

• Ensembling prototype scores from five self-supervised models neither improves nor harms pruning performance, so ensembles are unnecessary for the reported results.Table 3 averages four independent runs for pruning fractions of 10%, 20%, and 30%; observed differences remain within one standard deviation.Source: Table 3 · p. 44, G Impact of ensemble prototypes · p. 44

Class balance metric

• Class imbalance is measured by averaging pairwise minority-to-majority image ratios across two randomly selected classes, producing a 0–100% score where higher is better.A 90% score means the smaller class contains, on average, 90% as many images as the larger class.Source: H Relationship between pruning and… · p. 45

Class imbalance effects

• Class-conditional score distributions explain the effect: many classes have mostly low or mostly high scores, so threshold-based pruning preferentially removes particular classes.The underlying class-conditional histograms are shown in Figs. 22 and 23.Source: H Relationship between pruning and… · p. 45, Figure 22 · p. 48, Figure 23 · p. 49

Class balancing

• ImageNet experiments use 50% class balancing, ensuring every class retains at least half the images expected under equal pruning across classes.Results without class balancing are worse than with 50% balancing, confirming that vanilla pruning amplifies class imbalance.Source: H Relationship between pruning and… · p. 45, Figure 21 · p. 47

Class imbalance effects

• Pruning increases class imbalance for both easy- and hard-image pruning across all metrics except random pruning, with imbalance worsening as the pruning fraction grows.This pattern is shown in Fig. 20 for pruning fractions up to 0.9.Source: H Relationship between pruning and… · p. 45, Figure 20 · p. 46

Class-conditional accuracy

• Pruning caused a very small systematic reduction in accuracy for classes that were least accurately predicted by models trained on the full dataset.The effect was observed as blue lines slightly above red unity lines when class-conditioned accuracy was low.Source: I Effect of pruning on… · p. 50

Limitations

• The fairness assessment is preliminary because it concerns individual ImageNet classes rather than a specific deployment scenario, leaving broader use-case-specific evaluation for future work.The current models had no specific deployment setting beyond ImageNet testing.Source: I Effect of pruning on… · p. 50

Class-conditional accuracy

• Across all 1000 ImageNet classes, 80% datasets pruned with EL2N, memorization, or self-supervised prototypes produced class-conditional accuracy quite similar to full-data training.This comparison used ResNet-50 models and included two supervised metrics and one self-supervised metric; the class-wise comparison is shown in Fig. 24.Source: Figure 24 · p. 50, I Effect of pruning on… · p. 50

J Interaction between data pruning and training duration

• Training the same number of epochs after pruning reduces iterations, so performance reflects both less data and shorter training.This is the default ImageNet setting used throughout the main paper.Source: J Interaction between data pruning and… · p. 50
• The matched iterations factor interpolates between default training at 0 and full-dataset-equivalent iterations at 1.For example, pruning 50% of the dataset requires doubling training duration to match the full dataset’s iterations.Source: J Interaction between data pruning and… · p. 50
• Training longer slightly improves performance, but a matched iterations factor of around 0.4–0.6 may already provide the full benefit while any factor below 1.0 saves training time.Table 4 evaluates 20% ImageNet pruning using the self-supervised prototype metric and reports top-5 accuracy.Source: Table 4 · p. 51, J Interaction between data pruning and… · p. 51

Evaluation setup

• The OOD analysis uses the model-vshuman toolbox’s 17 datasets, including image distortions and style changes, on ResNet-50 models trained with full or pruned ImageNet.Pruned models use the self-supervised prototype metric; best-case discards easy examples, while worst-case discards hard examples.Source: K Out-of-distribution (OOD) analysis… · p. 51

Human-like OOD behavior

• Pruned models somewhat better match human accuracies than the VISSL baseline, outperform it on error consistency, but the baseline remains best on overall consistency.These human-like behavior metrics induce different model rankings.Source: K Out-of-distribution (OOD) analysis… · p. 51, Table 6 · p. 53

Shape–texture bias

• Best-case pruning produces slightly higher shape bias, whereas worst-case pruning shifts models toward texture bias; torchvision’s strong texture bias is attributed to augmentation differences.Shape and texture bias are assessed using category-level averages, with model colors matching Figure 25.Source: K Out-of-distribution (OOD) analysis… · pp. 51–52, Figure 26 · p. 53

OOD accuracy

• Best-case pruning achieves OOD accuracies very similar to the VISSL baseline, and does not hurt OOD performance versus an accuracy-matched baseline on the investigated datasets.Even pruning 40% of ImageNet produced almost the same OOD accuracy as the full-dataset baseline despite the prototype metric’s impaired IID performance; worst-case pruning performed lower.Source: K Out-of-distribution (OOD) analysis… · p. 51, Figure 25 · p. 52