PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond neural scaling laws: beating power law scaling via data pruning

Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, Ari S. Morcos

arXiv:2206.14486v6cs.LGcs.AIcs.CVstat.ML

TL;DR

Power-law scaling with dataset size makes further improvements costly, and prior pruning work leaves open whether data selection can achieve faster scaling at practical scale. The paper develops analytic pruning theory, tests it across modern vision models and datasets, benchmarks 10 ImageNet metrics, and introduces a self-supervised metric. It reports exponential-scaling signatures and comparable performance from a cheap unlabeled-data pruning method, while noting that exponential scaling depends on high-quality metrics that remain difficult to scale.

  • Problem

    Power-law scaling makes dataset-driven improvements increasingly costly, while it remains unclear whether data pruning can beat power-law scaling and work on large unlabeled datasets.

  • Method

    The paper combines statistical-mechanics theory of margin-based pruning, experiments across datasets and models, ImageNet benchmarking, and a self-supervised pruning metric.

  • Results

    Exponential scaling is possible in theory and practice, while the new self-supervised metric performs comparably to the best supervised metrics without labels.

  • Takeaways & Limitations

    Data pruning may reduce reliance on extremely large randomly collected datasets by enabling smaller, carefully selected training datasets.

  • Takeaways & Limitations

    Achieving exponential scaling requires a high-quality pruning metric, but most metrics scale poorly to ImageNet and the strongest scalable metrics require substantial computation.

Abstract

from arXiv · show

Widely observed neural scaling laws, in which error falls off as a power of the training set size, model size, or both, have driven substantial performance improvements in deep learning. However, these improvements through scaling alone require considerable costs in compute and energy. Here we focus on the scaling of error with dataset size and show how in theory we can break beyond power law scaling and potentially even reduce it to exponential scaling instead if we have access to a high-quality data pruning metric that ranks the order in which training examples should be discarded to achieve any pruned dataset size. We then test this improved scaling prediction with pruned dataset size empirically, and indeed observe better than power law scaling in practice on ResNets trained on CIFAR-10, SVHN, and ImageNet. Next, given the importance of finding high-quality pruning metrics, we perform the first large-scale benchmarking study of ten different data pruning metrics on ImageNet. We find most existing high performing metrics scale poorly to ImageNet, while the best are computationally intensive and require labels for every image. We therefore developed a new simple, cheap and scalable self-supervised pruning metric that demonstrates comparable performance to the best supervised metrics. Overall, our work suggests that the discovery of good data-pruning metrics may provide a viable path forward to substantially improved neural scaling laws, thereby reducing the resource costs of modern deep learning.

1 Introduction

Power-law scaling makes further error reductions increasingly costly, motivating whether intelligently pruning redundant training data can yield exponential scaling. The paper develops theory, tests its predictions across datasets, benchmarks pruning metrics, and introduces a scalable self-supervised alternative.

  • Motivation: Power-law error scaling can make modest improvements require substantially more data, compute, or energy.Examples include a tenfold data increase for a language-model loss reduction and billions of additional vision-training examples for a few accuracy points.
  • Motivation: The paper asks whether selecting training examples intelligently can achieve exponential rather than power-law scaling with dataset size.Its approach relies on pruning datasets to smaller sizes while preserving performance.
  • Theory: The optimal pruning strategy depends on the initial dataset size: retain hard examples with abundant data but easy examples with scarce data.The analytic prediction concerns the relationship between the initial data amount and which examples should be retained.
  • Theory: Exponential scaling is possible with respect to pruned dataset size when the pruning fraction is chosen along an increasing Pareto-optimal schedule.This prediction is derived theoretically and tested empirically.
  • Experiments: The predicted better-than-power-law behavior appears in ResNets trained on SVHN, CIFAR-10, and ImageNet, and in Vision Transformers fine-tuned on CIFAR-10.These experiments test the theory beyond the perceptron setting.
  • Metric benchmarking: A benchmark of 10 ImageNet pruning metrics found that most perform poorly at scale, while the exceptions require substantial computation.The study motivated development of a cheaper scalable metric.
  • Metric benchmarking: A new self-supervised pruning metric performs comparably to the best supervised metrics without requiring labels and much more compute.The result supports pruning datasets before they are labeled.
  • Implications: The results suggest that collecting smaller amounts of carefully selected data could be more efficient than collecting extremely large random datasets.The authors frame this as motivation for further study of foundation datasets.

2 Background and related work

Prior data-pruning work orders examples by difficulty or importance, but its ImageNet-scale efficacy, label requirements, theoretical basis, and ability to beat power-law scaling remain limited. This paper situates its contribution across pruning metrics, neural scaling laws, and statistical-mechanics analyses of learning.

  • Pruning metrics: Existing pruning metrics generally rank examples from easiest to hardest, typically retaining hard examples and discarding easy ones.The framework provides a common description for several metric families.
  • Pruning metrics: EL2N pruning retained the hardest examples and enabled training on 50% of CIFAR-10 and 75% of CIFAR-100 without loss in final test accuracy.Its ImageNet performance had not yet been explored.
  • Pruning metrics: Forgetting scores distinguish examples learned early and retained from examples repeatedly learned and forgotten, while memorization and influence quantify example-specific learning effects.Memorization and influence are computationally expensive, and memorization does not account for interactions.
  • Related approaches: Active learning selects new inputs for labeling iteratively, whereas data pruning performs one-shot subset selection for training from scratch.Coreset methods and early clustering approaches exist but are computationally expensive or insufficiently explored at ImageNet scale.
  • Open gaps: A large-scale summary found that only one prior metric had tested well on ImageNet, all prior metrics required labels, and none proposed exponential scaling.The paper addresses these gaps with ImageNet benchmarking, an unsupervised metric, and analytic theory.
  • Scaling laws: Neural test loss often follows a weak power law in parameters, training examples, or compute, with exponents that can be close to zero.The paper connects these empirical scaling observations to statistical-mechanics analyses of learning.
  • Theory context: The theory uses the perceptron student-teacher setting, while prior active-learning analyses focused on specialized algorithms that are difficult to generalize to real-world settings.This work instead analyzes margin-based pruning algorithms used in practice.

3 An analytic theory of data pruning

The analytic theory models pruning in a student–teacher perceptron and predicts when retaining easy or hard examples is optimal. With a perfect metric, pruning can initially produce exponential error scaling, whereas imperfect metrics eventually force a power-law crossover.

  • Theory and setup: The theory uses teacher-margin ranking to retain either the hardest or easiest examples, then trains a new perceptron on the pruned dataset.Large margins denote easy examples and small margins denote hard examples.
  • Optimal pruning strategy: With scarce initial data, retaining easy examples is better because they provide coarse-grained information and reduce overfitting.With abundant data, hard examples become preferable because they provide finer information about the teacher’s decision boundary.
  • Empirical check: The predicted transition between easy-example and hard-example strategies also appears in ResNet18 experiments on pruned CIFAR-10 subsets.This supports the theory’s qualitative behavior beyond the perceptron setting.
  • Scaling predictions: For fixed pruning fractions, error initially decreases exponentially with pruned dataset size but eventually returns to the universal power law ε ∝ α_prune^-1.The asymptotic power law means fixed-fraction pruning offers no lasting advantage.
  • Scaling predictions: More aggressive pruning of larger initial datasets can maintain a finite information rate, allowing at least exponential error decay and potentially faster-than-exponential decay.This requires a Pareto-optimal pruning strategy that removes redundant examples.
  • Metric quality: An imperfect metric causes exponential scaling to cross over to power-law scaling because retaining too few examples eventually excludes useful information about the teacher.The minimum usable fraction depends on metric error: θ = 10° permits 24% retention, whereas θ = 20° permits 46%.

4 Data pruning can beat power law scaling in practice

Experiments test the theory across neural-network training and transfer-learning settings. They observe signatures of improved scaling and show that pruning can reduce both fine-tuning and pre-training data while matching or exceeding full-data performance.

  • Empirical scaling: The theory’s predictions are tested on ResNets trained from scratch on SVHN, CIFAR-10, and ImageNet, plus Vision Transformers fine-tuned on CIFAR-10.These experiments show signatures of exponential scaling with pruned dataset size.
  • Transfer learning: Pruning can improve transfer learning in both downstream fine-tuning and upstream pre-training phases.The experiments use pruned CIFAR-10 fine-tuning sets and pruned ImageNet1K pre-training sets.
  • Transfer learning: Fine-tuning on only 10% of CIFAR-10 can match or exceed performance from fine-tuning on all CIFAR-10 after ViT pre-training on ImageNet21K.The comparison uses EL2N-based pruning.
  • Transfer learning: Pre-training on as little as 50% of ImageNet1K can match or exceed CIFAR-10 performance from pre-training on all ImageNet1K.ResNet50 models are subsequently fine-tuned on the full CIFAR-10 dataset.

5 Benchmarking supervised pruning metrics on ImageNet

The ImageNet benchmark finds substantial differences among pruning metrics and shows that most do not scale well to this dataset. Metrics that perform well are generally computationally intensive, motivating simpler scalable alternatives.

  • Benchmark scope: The study compares 10 data-pruning metrics on ImageNet and finds substantial diversity in their induced example rankings.EL2N, DDD, and memorization are relatively similar, with rank correlations above 0.7.
  • Benchmark results: Only a few metrics match full-dataset performance when selecting a substantially smaller ImageNet training subset of about 80%.Most metrics still outperform random pruning, with memorization showing particularly strong performance.
  • Practical constraints: Most pruning metrics do not scale well to ImageNet, while the few that do require substantial computation and labels for every image.These requirements limit their use for large unlabeled foundation-model datasets.
  • Practical constraints: All evaluated pruning metrics amplify class imbalance, so the ImageNet experiments use a 50% class-balancing ratio.The paper provides additional baselines without class balancing in Appendix H.

6 Self-supervised data pruning through a prototypicality metric

The paper introduces a simple self-supervised pruning metric based on image prototypicality and finds it scalable, label-free, and competitive with supervised metrics. Its cluster structure partially aligns with ImageNet classes, while easy examples appear redundant and hard examples idiosyncratic.

  • Many pruning metrics scale poorly to ImageNet, while those that scale typically require substantial compute and labels.
  • The self-supervised metric ranks examples by their cosine distance to the nearest k-means cluster centroid in a pretrained embedding space.
  • Its clusters align with some ImageNet classes but disperse examples from other classes across multiple clusters.
  • Performance remains robust when the number of clusters differs from the 1,000 ImageNet classes by one order of magnitude.
  • Under the metric, easy images are highly similar and redundant, whereas hard images resemble idiosyncratic outliers.
  • Figure 6 compares cluster-class probabilities with the easiest and hardest examples selected by self-supervised and supervised metrics for black swans.

7 Discussion

The paper argues that data pruning can replace slow power-law error scaling with faster exponential scaling, but this depends on high-quality metrics and careful evaluation of efficiency trade-offs. It proposes scalable pruning for large datasets and future foundation datasets while identifying class balance and fairness as practical boundaries.

  • Data pruning is reported to achieve faster exponential rather than slow power-law scaling of error with dataset size.
  • Exponential scaling requires a high-quality pruning metric, and most existing metrics scale poorly to ImageNet.
  • Pruned datasets can reduce computational cost, but matching full-dataset iterations may add epochs and create an accuracy–training-time trade-off.
  • Class balancing is essential for maintaining performance on data subsets, with appropriate balancing levels left for future work.
  • Deployment-specific fairness tests remain necessary because preliminary ImageNet analysis found no substantial differential effects across classes.
  • The authors propose scalable unsupervised pruning for massive unlabeled datasets and envision reusable foundation datasets whose pruning cost is amortized across downstream models.

A.2 Main result and overview

The theory derives self-consistent equations for generalization error under arbitrary pruning distributions and validates them against simulations. It further characterizes information gain and shows that the optimal pruning policy changes with the pruning-data ratio, interpolating between easy- and hard-example selection.

  • Main result: Self-consistent equations determine generalization error for any data ratio, pruning distribution, and probe–teacher angle.The order parameters R, ρ, and κ yield εg = cos^-1(R)/π.
  • Replica calculation: The statistical-mechanics derivation uses Gardner volumes, replica methods, auxiliary variables, and order parameters for student, teacher, and probe overlaps.Normalization of the student and isotropy around the probe student are imposed during the calculation.
  • Validation: The resulting saddle-point equations fit numerical simulations excellently and recover the classical teacher–student perceptron when pruning is removed.The no-pruning limit is f → 1 and γ → ∞.
  • Information gain: Pruning changes information gain from the classical decay I(α) ∼ 1/α toward a finite per-example rate under aggressive pruning.In the limit f → 0 and γ → 0, the information gain approaches a finite rate, with I(∞) = 1 nat/example for R → 1.
  • Scaling boundary: When data ultimately stops concentrating near the teacher’s decision boundary, information gain vanishes and the generalization-error envelope returns to power-law scaling.A minimum pruning fraction fmin bounds the achievable error envelope in this regime.
  • Optimal pruning policy: The optimal policy retains hard examples at large αprune but easy examples at small αprune, with an intermediate pruning window connecting the regimes.The optimal distribution and window shift from large margins toward small margins as αprune increases.

B Model training method details & dataset information

The experiments combine synthetic perceptron data with CIFAR-10, SVHN, and ImageNet training, using specified architectures, optimizers, hardware, and pruning procedures. ImageNet benchmarking evaluates pruning metrics across retained-data fractions against an unpruned baseline.

  • Perceptron experiments: Perceptron experiments use Gaussian inputs labeled by a randomly drawn teacher, with N = 200 and datasets sized as P = αN.Datasets retain fractions of the smallest-margin examples, and perceptrons are optimized for max-margin separation using quadratic programming.
  • ImageNet: ImageNet experiments use ResNet-50 through VISSL, trained on a single node with 8 NVIDIA V100 32GB GPUs and SGD.The main benchmark trains one model for each of five retained-data fractions across 11 metrics, totaling 55 models.
  • ImageNet: The ImageNet benchmark reports top-5 validation accuracy for ResNet-50 models against an unpruned baseline of 90.848.Performance variation at full retention is averaged into one datapoint, with additional repeated models used to reduce noise for Figure 5C.
  • Transfer learning: Transfer-learning experiments prune either CIFAR-10 fine-tuning data for pretrained ViTs or ImageNet1k pretraining data before CIFAR-10 fine-tuning.These settings use different pretrained models, optimizers, learning rates, and probe-training durations.

C Breaking compute scaling laws via data pruning

The paper evaluates whether pruning can improve compute scaling by relating convergence and fixed-epoch training costs to pruned dataset size. Additional experiments examine metric strength, image rankings, and prototype-metric hyperparameters.

  • Breaking compute scaling laws: Data pruning can afford exponential compute savings for perceptrons, with preliminary evidence of similar savings for ResNets on CIFAR-10 and ImageNet.The study tracks computational complexity while repeating learning experiments across pruning levels.
  • Breaking compute scaling laws: ResNet18 on CIFAR-10 and ResNet50 on ImageNet are evaluated at fixed epoch counts to test whether convergence time depends mainly on retained example count.This differs from prior experiments that fixed the number of iterations.
  • Additional scaling experiments: Additional EL2Ns and self-supervised-prototype experiments show signatures of better-than-power-law scaling, though less dramatically than memorization.The experiments include EL2Ns with 0% and 50% class balancing and self-supervised prototypes with class balancing.
  • Metric strength: On SVHN, a weak four-epoch probe leaves the learning-curve envelope lower-bounded by a power law, whereas a stronger 40-epoch probe breaks through it.The stronger metric achieves lower generalization error, illustrating a crossover from power-law to improved scaling.
  • Metric behavior: Extreme-image figures compare examples ranked easy or hard by each pruning metric across multiple ImageNet classes, including cliff dwelling.Top rows show most-pruneable images and bottom rows show least-pruneable images.
  • Metric hyperparameters: The self-supervised prototype metric’s performance depends on k, the number of clusters used for embedding-space k-means.Table 2 reports top-5 accuracy after pruning 20% of ImageNet without class balancing.

H Relationship between pruning and class (im-)balance

Pruning generally increases class imbalance, whether it removes easy or hard examples, which can degrade class-specific performance if unaddressed. The ImageNet experiments therefore apply class balancing.

  • Observed imbalance: Pruning strongly increases class imbalance for easy- and hard-image pruning across metrics, except random pruning.The effect is measured with a class-balance score designed to account for all ImageNet classes rather than only the extremes.
  • Observed imbalance: Metric scores differ across classes, helping explain why pruning can preferentially discard images from particular classes.Class-conditional score distributions are visualized across metrics, with every tenth class shown for readability.
  • Mitigation: The ImageNet experiments use 50% class balancing to ensure every class retains at least half the images expected under equal pruning.This provides a floor on per-class representation, while the authors note that other approaches could also be used.

I Effect of pruning on class-conditional accuracy and fairness

The paper compares class-conditional accuracy after pruning ImageNet data and treats this as an initial fairness analysis. It finds a small systematic disadvantage for harder classes and limits broader conclusions to deployment-specific settings.

  • Class-conditional accuracy: Class-conditional accuracies are compared between full-data and 80%-retained ResNet-50 models pruned with EL2N, memorization, or self-supervised prototypes.The analysis covers all ImageNet classes and includes two supervised and one self-supervised metric.
  • Class-conditional accuracy: Pruning yields a slight systematic reduction in accuracy for harder classes relative to easier classes, though the effect is small.This comparison is reported across the evaluated pruning metrics.
  • Scope boundary: The fairness assessment is limited because it evaluates individual ImageNet classes without a specific deployment scenario.The authors leave fairness evaluation in other deployment settings for future work.

J Interaction between data pruning and training duration

The paper examines whether pruning can reduce training iterations while preserving performance, finding that modestly longer training recovers most of the benefit of full-dataset iteration counts.

  • A matched iterations factor of 0 represents equal epochs and proportionally fewer iterations on smaller datasets, whereas 1 matches full-dataset iterations.
  • Training longer slightly improves performance, with a matched iterations factor of around 0.4–0.6 often sufficient to obtain the full benefit.
  • Any matched iterations factor below 1.0 retains reduced training time relative to training on the full dataset.
  • Table 4 evaluates longer-training settings for models trained after pruning 20% of ImageNet using the self-supervised prototype metric.

K Out-of-distribution (OOD) analysis of dataset pruning

The OOD analysis evaluates pruned and full-data ResNet-50 models across 17 datasets, finding that best-case pruning preserves OOD accuracy and modestly shifts models toward human-like shape bias without consistently improving every human-alignment metric.

  • OOD accuracy: The OOD benchmark averages accuracy across 17 datasets and compares 12 ResNet-50 models, including full-ImageNet torchvision and VISSL baselines, pruned models, and human data.
  • OOD accuracy: Best-case pruning achieves OOD accuracies very similar to the VISSL full-data baseline, including after the self-supervised metric prunes 40% of the data.
  • OOD accuracy: The analysis concludes that pruning did not hurt OOD performance on the investigated datasets compared with an accuracy-matched baseline, although more analysis is needed to assess consistency.
  • Human-like behavior: Two models pruned with the self-supervised prototype metric somewhat more closely match human accuracies than the VISSL baseline, while overall and error consistency produce different rankings.
  • Shape versus texture bias: Best-case pruning produces slightly higher shape bias than the baseline, whereas worst-case pruning shifts toward texture bias; the torchvision outlier is attributed to augmentation differences.
  • Benchmark summaries: Tables 5 and 6 summarize models with the highest OOD robustness and the most human-like behavior, respectively.
Loading 2206.14486v6…