PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning on a Data Diet: Finding Important Examples Early in Training

Mansheej Paul, Surya Ganguli, Gintare Karolina Dziugaite

arXiv:2107.07075v2cs.LG

TL;DR

Deep learning’s growing datasets raise questions about which examples matter for generalization and how to identify them efficiently. The paper introduces early-training GraNd and EL2N scores, showing that EL2N can prune 50% of CIFAR-10 examples without affecting accuracy.

  • Problem

    The paper asks which training examples are important for generalization and how to identify them as datasets grow increasingly costly to train.

  • Method

    The paper uses GraNd and EL2N scores, computed from local early-training information, to rank examples for dataset pruning and study training dynamics.

  • Results

    50% of CIFAR-10 examples can be pruned without affecting accuracy, while EL2N rankings generalize across architectures and hyperparameter configurations.

  • Takeaways & Limitations

    Simple scores computed very early in training can identify examples for substantial data pruning while preserving test accuracy in the evaluated vision benchmarks.

  • Takeaways & Limitations

    At extreme pruning levels, GraNd and EL2N can sharply reduce performance, likely because retaining high-error examples provides poor coverage of the data distribution.

Abstract

from arXiv · show

Recent success in deep learning has partially been driven by training increasingly overparametrized networks on ever larger datasets. It is therefore natural to ask: how much of the data is superfluous, which examples are important for generalization, and how do we find them? In this work, we make the striking observation that, in standard vision datasets, simple scores averaged over several weight initializations can be used to identify important examples very early in training. We propose two such scores -- the Gradient Normed (GraNd) and the Error L2-Norm (EL2N) scores -- and demonstrate their efficacy on a range of architectures and datasets by pruning significant fractions of training data without sacrificing test accuracy. In fact, using EL2N scores calculated a few epochs into training, we can prune half of the CIFAR10 training set while slightly improving test accuracy. Furthermore, for a given dataset, EL2N scores from one architecture or hyperparameter configuration generalize to other configurations. Compared to recent work that prunes data by discarding examples that are rarely forgotten over the course of training, our scores use only local information early in training. We also use our scores to detect noisy examples and study training dynamics through the lens of important examples -- we investigate how the data distribution shapes the loss surface and identify subspaces of the model's data representation that are relatively stable over training.

1 Introduction

The paper develops early-training scores for identifying important or difficult examples, enabling substantial dataset pruning while studying how selected subpopulations shape loss surfaces and training dynamics. It also shows that the highest-scoring examples can become superfluous, especially with label noise or aggressive pruning.

  • Motivation: Overparameterized models and ever-larger datasets increase computational demands, motivating methods that identify important training data for resource-constrained and active-learning settings.The paper frames dataset pruning as an empirical question across standard vision benchmarks.
  • Correction: This version corrects erroneous conclusions about GraNd pruning at initialization caused by a Flax bug.The correction specifically affects GraNd scores computed at initialization.
  • Scoring methods: GraNd scores rank each example by its expected loss-gradient norm, which bounds the expected loss change caused by removing that example.The score is defined for training pairs (x_i, y_i) and applies to removal effects on arbitrary examples.
  • Scoring methods: Within the first few epochs, EL2N approximates GraNd and enables pruning 50% of CIFAR10 or 25% of CIFAR100 without loss in test accuracy.EL2N is the norm of predicted class probabilities minus the one-hot label encoding.
  • Important examples: Excluding a small subset of the highest-EL2N examples improves performance, with a stronger boost under corrupted labels.The paper also finds that very high-scoring examples become superfluous as label noise increases and in high-pruning regimes.
  • Training dynamics: A linearly connected-mode analysis shows that low-EL2N subpopulations determine their converged empirical-risk surface much earlier than high-score subpopulations.The paper uses this method to study when final subpopulation performance is determined during training.

2 Which samples are important for learning?

The section defines GraNd as the expected gradient norm of an example and motivates it as a proxy for that example’s influence on loss reduction. It also introduces EL2N as an error-vector approximation to GraNd and relates both scores to learning speed and forgetting events.

  • GraNd score: GraNd is the expected L2 norm of an example’s loss gradient at a training time under random initialization.It is defined as χ_t(x, y) = E_{w_t} ||g_t(x, y)||_2.
  • Influence on learning: An example’s gradient norm bounds its contribution to decreasing the loss on other examples during a gradient step.Because the bound’s constant is example-independent, examples with small expected GraNd scores have bounded influence on learning.
  • EL2N score: EL2N approximates GraNd by measuring the expected L2 norm of the prediction error vector.The approximation follows when logit gradients are roughly orthogonal and similarly sized across logits and examples.
  • Learning dynamics: Examples learned faster and maintaining small error tend to have smaller GraNd scores, while examples with many forgetting events tend to have higher scores.In the simplified setting, the rescaled prediction error upper-bounds both 0–1 loss and the number of forgetting events.

3 Empirical Evaluation of GraNd and EL2N Scores via Data Pruning

Empirical evaluations across architectures and vision datasets show that early-training EL2N and GraNd scores can identify high-value examples for data pruning. EL2N is especially effective after a few epochs, while extreme pruning harms coverage and initialization-only GraNd scores perform poorly.

  • Data pruning experiments: Experiments train ResNet18 and ResNet50 models on CIFAR-10, CIFAR-100, and CINIC-10, averaging scores across ten independent runs before retraining on selected subsets.Final accuracies are measured after retraining from new random initializations using only the selected data.
  • Pruning at initialization: At initialization, GraNd scores can select subsets in every setting, but the resulting pruning performance is poor.This contrasts with the strong performance of scores computed after limited early training.
  • Pruning early in training: After a few epochs, EL2N scores identify examples whose selected subsets achieve test accuracy on par with or better than the full dataset.Early-training EL2N remains competitive with forgetting scores even though forgetting integrates information across the training trajectory.
  • Pruning limits: Extreme EL2N or GraNd pruning causes a sharp performance drop, likely because selecting high-error examples excludes substantial test-relevant subpopulations.The hypothesized failure is poor coverage of the data distribution when only a small number of difficult examples are retained.
  • A property of the data: EL2N rankings appear dataset-specific rather than network-specific: ResNet18 and ResNet50 on CIFAR-10 show similar curves and prune the same amount of data.Scores averaged across initializations or trajectories are hypothesized to reduce dependence on particular weights and better capture dataset properties.

4 Identifying noise examples

The section shows that the highest-scoring examples are not always the most important for accuracy: their apparent value depends on label noise, which shifts the optimal training subset toward lower-score examples. Consequently, selecting only the highest-scoring samples can be suboptimal, and excluding high-score examples without validation is risky.

  • Identifying noise examples: Highest-scoring examples are not necessarily the most important for achieving high accuracy, challenging the hypothesis that score directly measures example importance.The section explicitly frames this as a refutation and demonstrates the role of label noise.
  • Identifying noise examples: 10% randomized labels shift the best subset toward excluding more examples than in the clean-label setting.Figure 2 compares ResNet18 trained on 40% subsets of CIFAR-10 with clean labels and 10% randomized labels; scores were computed at epoch 10.
  • Identifying noise examples: Performance generally improves as an EL2N-score window moves to higher percentiles, except when the window includes examples affected by label noise.The experiment keeps a fixed P% of data while sliding the percentile window upward.
  • Identifying noise examples: With low Bayes error, using only the highest-scoring samples can be optimal, but high-score examples should not be excluded without a validation set.The caution is especially relevant when label noise may be present.

5 Optimization landscape and the training dynamics

The paper connects EL2N-based example difficulty to neural-network optimization dynamics: high-scoring examples induce faster NTK evolution, except for the very highest-scoring noisy examples, while low- and high-score subsets exhibit different loss-landscape behavior. This contrasts finite-width training dynamics with the fixed-NTK kernel-regression picture of the infinite-width limit.

  • Optimization landscape: Infinite-width networks follow kernel regression with a fixed Neural Tangent Kernel (NTK) determined at initialization, whereas finite networks have different early-training dynamics.The NTK is defined through the logits’ Jacobians at initialization.
  • Optimization landscape: Higher EL2N scores correspond to higher NTK submatrix velocities, and this relationship is stable across the training times examined.Velocity is estimated using cosine distance between NTK Gram matrices one epoch apart on contiguous EL2N-ranked subsets.
  • Optimization landscape: With 10% label noise, NTK velocity drops sharply for the very highest-scoring examples, which are hypothesized to be too difficult or unrepresentative.The paper links removing these examples to improved final-predictor accuracy.
  • The training dynamics: The study measures error barriers between independently minibatched training trajectories spawned from the same weights to analyze linear mode connectivity on example subsets.Subset-specific barriers enable comparisons of training dynamics and modes across subpopulations.
  • The training dynamics: Low-EL2N subsets reach near-zero error barriers rapidly, whereas high-EL2N subsets retain high barriers, indicating different loss-landscape behavior.The comparison uses subsets with the smallest scores, largest scores, and random examples.

6 Related Work

The work relates to methods based on forgetting, proxy networks, coresets, training dynamics, example difficulty, and influence. Its distinguishing feature is estimating example importance or difficulty from local information early in training rather than statistics accumulated over a full run.

  • Forgetting-based selection: Forgetting scores identify rarely forgotten examples whose removal preserves accuracy, whereas these methods use local information early in training.The paper connects both approaches to analyzing training dynamics through training examples and argues that early local loss-landscape properties reflect importance.
  • Proxy-network selection: Proxy networks select small subsets of important-for-training examples and find that selected examples remain important when training a SOTA network.Coleman et al. also study a shortened-training proxy that reuses the SOTA network’s architecture.
  • Coresets: Coreset methods seek possibly weighted subsets that provably approximate the full training objective, but most guarantees require special structure such as convexity.The paper distinguishes these guarantees from the nonconvex setting of deep neural-network training and notes coreset applications in active learning and robustness.
  • Training-dynamics statistics: For noisy-example detection, AUM tracks training dynamics throughout training, unlike this work’s focus on instantaneous information in the early phase.AUM exploits differences between clean and mislabeled samples and is similar to forgetting scores because both use whole-run information.
  • Example difficulty: VoG and prediction depth estimate example difficulty using checkpoint gradient variance or final-network representations, while this work highlights a strong early-training signal.Prediction depth is defined by the first layer where a k-Nearest Neighbor classifier correctly classifies an example using subsequent-layer representations.
  • Influence and memorization: Removing an example without increasing generalization error suggests low influence on test data, linking the work to sample-based explainability and memorization theory.The related theory models data as mixtures of populations and studies memorization when distributions are long-tailed.

7 Discussion · A Ethical and societal consequences · B.1 Resources used

The work presents early-training data-pruning methods that can preserve test accuracy while also enabling analysis of how training examples shape deep-learning dynamics. It discusses energy use, accuracy-related fairness limitations, and the computational resources required.

  • 7 Discussion: The study introduces methods to significantly prune training data without sacrificing test accuracy.The methods use local information very early in training and sometimes at initialization.
  • 7 Discussion: The proposed methods use local information very early in training, sometimes even at initialization.
  • 7 Discussion: The resulting methods provide scientific insights into how different subsets of training examples drive deep-learning dynamics.
  • A Ethical and societal consequences: The empirically driven work consumed considerable energy, while potentially enabling theory to guide experiments more efficiently.
  • A Ethical and societal consequences: Focusing mostly on accuracy can hide disparate effects on marginalized groups.The authors hope uncovering the influence of training examples and sub-populations will lead to methods that decrease bias.
  • B.1 Resources used: All experiments ran on a single 16GB NVIDIA Tesla V100 GPU.The project used about 15000 GPU hours on an internal ServiceNow cluster.

B.2 Training details · B.3 Experimental details · C Example Images

The experiments use standardized CIFAR-10, CIFAR-100, and CINIC-10 pipelines with low-resolution ResNet models and fixed SGD schedules. Scores and outcomes are evaluated across independent runs, while example inspection shows that low-score images are canonical and high-score images are harder to identify.

  • B.2 Training details: The study uses CIFAR-10, CIFAR-100, and CINIC-10, with CINIC-10 combining training and validation into 180000 images and testing on 90000 images.All datasets are normalized by training-set channel statistics and use padding, random cropping, and horizontal flipping.
  • B.2 Training details: The models are low-resolution ResNet18-v1 and ResNet50-v1 variants, replacing the original initial layers with a single 3×3 convolution of stride 1.This modification targets the 32×32 resolution of CIFAR and CINIC images.
  • B.2 Training details: Networks train with SGD, learning rate 0.1, Nesterov momentum 0.9, weight decay 0.0005, and 200 total epochs.Batch sizes are 128 for CIFAR-10/CIFAR-100 and 256 for CINIC-10; the learning rate drops by 5 after epochs 60, 120, and 160.
  • B.3 Experimental details: Reported accuracies average 4 independent runs, with the 16th–84th percentile shown for variability across runs.Score evaluation uses newly initialized models with seeds different from those used to calculate scores.
  • B.3 Experimental details: All EL2N, GraNd, and forgetting scores are averaged across 10 independent runs.This averaging is the stated procedure for calculating each score family.
  • B.3 Experimental details: Random-label experiments permute labels for 10% of examples, then sort examples by score and select subsets according to the specified offset and size.The procedure drops the lowest-scoring offset, keeps the next subset, and drops all following images.
  • C Example Images: For ResNet18 on CIFAR-10, minimum GraNd and EL2N examples tend to be simple, canonical class representations from typical angles.GraNd is computed at initialization and EL2N at epoch 10; maximum-score examples are harder to identify.

D Comparisons to Label Independent Scores · E Additional Experiments

The section compares label-dependent and label-independent variants of GraNd and EL2N, testing whether labels contribute to pruning performance. In CIFAR10 experiments, only label-dependent scores significantly outperformed random pruning when retaining 50% of training examples.

  • D Comparisons to Label Independent Scores: The GraNd score is defined for classification examples (x, y), with class index k appearing in its formulation.This formulation motivates examining whether class-label information contributes to the score.
  • D Comparisons to Label Independent Scores: GraNd is theoretically label-independent at initialization because the weights contain no label information.The paper notes that the finite-initialization-sample approximation may nevertheless introduce practical label effects.
  • D Comparisons to Label Independent Scores: EL2N differs from GraNd by consisting only of label prediction error.The section also defines three label-independent score variants for comparison.
  • D Comparisons to Label Independent Scores: 10 independent ResNet18 networks were trained on CIFAR10, with six scores computed at initialization and epoch 20.The six scores comprise three score types evaluated at two initialization points, averaging expectations across the 10 networks.
  • D Comparisons to Label Independent Scores: 50% of CIFAR10 training examples with the smallest scores were pruned to create six corresponding datasets.Three newly randomly initialized ResNet18 networks were then trained on each pruned dataset, and final test accuracy was averaged.
  • D Comparisons to Label Independent Scores: Only label-dependent scores performed significantly better than the random baseline.The comparison included a 50% subset selected by maximum EL2N scores at epoch 20, a random 50% subset, and the full training set.

E.1 Sensitivity analysis of GraNd and EL2N scores · E.2 Comparison between scores from different architectures on the same dataset

Averaging GraNd and EL2N scores across initializations substantially improves pruning reliability, with 10–20 runs generally sufficient. Across ResNet architectures, early EL2N and GraNd pruning remains effective, while network depth has only a small effect.

  • E.1 Sensitivity analysis of GraNd and EL2N scores: 0.75 is the average Spearman rank correlation between single-run and 10-run-averaged GraNd scores at initialization.This comparison uses ResNet18 on CIFAR-10.
  • E.1 Sensitivity analysis of GraNd and EL2N scores: Averaging GraNd scores over 10 initializations significantly outperforms single-initialization scoring when selecting 50% of CIFAR-10.The comparison is against both single-initialization GraNd selection and random sampling.
  • E.1 Sensitivity analysis of GraNd and EL2N scores: EL2N scores calculated at epoch 10 show similar averaging benefits to GraNd scores at initialization.The comparison is conducted on ResNet18 trained on CIFAR-10.
  • E.1 Sensitivity analysis of GraNd and EL2N scores: Averaging over 10–20 runs suffices for accurate importance rankings, while additional runs provide insignificant benefit.The results suggest the scores reflect dataset properties rather than specific network weights.
  • E.2 Comparison between scores from different architectures on the same dataset: EL2N pruning at epoch 20 is competitive with forgetting-score pruning at epoch 200 across the tested architectures.The experiment compares ResNet18 and ResNet50 on CIFAR-10.
  • E.2 Comparison between scores from different architectures on the same dataset: GraNd-based pruning performs significantly better than the random baseline on both tested networks.The comparison uses ResNet18 and ResNet50 trained on CIFAR-10.
  • E.2 Comparison between scores from different architectures on the same dataset: Network depth has a small effect on the data-pruning results.This conclusion follows the comparison of ResNet18 and ResNet50 on CIFAR-10.

E.3 EL2N scores generalize across architectures … F.1 Noisy Examples in Low Pruning Regime

EL2N scores remain effective when transferred across architectures and hyperparameter configurations, indicating that they capture information intrinsic to the dataset. The sections also relate score agreement to pruning behavior and show that, with lower pruning, retaining difficult or noisy high-score examples no longer improves performance.

  • E.3 EL2N scores generalize across architectures: EL2N scores calculated with one architecture prune datasets for other architectures without changing final test accuracy relative to same-architecture scores.This supports the claim that EL2N captures dataset-intrinsic rather than network-specific information.
  • E.3 EL2N scores generalize across architectures: VGG13 networks trained on EL2N-selected CIFAR10 subsets perform identically for scores computed with ResNet18 or VGG13 and significantly better than random subsets.Scores were computed at epoch 20 by averaging 10 independently initialized networks, and subsets retained the highest scores across n = 20, 30, 40, 50, and 60.
  • E.4 Calculating EL2N scores during hyperparameter optimization: EL2N scores from networks trained during hyperparameter searches perform as well as scores averaged over networks with optimal hyperparameters across architectures, datasets, and pruning levels.The comparison uses nine grid-search runs versus nine optimal-hyperparameter runs, with scores calculated at epoch 20.
  • E.4 Calculating EL2N scores during hyperparameter optimization: Using hyperparameter-search networks to calculate EL2N scores could reduce future compute and energy use by enabling smaller training datasets.The paper identifies using such scores to accelerate hyperparameter optimization as a future research direction.
  • E.5 Correlations between scores: EL2N and forgetting scores have the highest Spearman rank correlation among the compared scores, consistent with their most similar pruning performance.The comparison uses GraNd at initialization, EL2N at epoch 20, and forgetting at epoch 200 for ResNet18 on CIFAR10.
  • F Noise / F.1 Noisy Examples in Low Pruning Regime: At 50% pruning with no randomized labels, discarding the highest-score examples no longer boosts performance, although decreasing marginal gains remain.This lower-pruning experiment contrasts with the 60% pruning regime and suggests that retaining difficult or noisy high-score examples does not hurt when enough data remain.

F.2 Scores for Noisy Examples · G Comparison to memorization threshold

EL2N scores tend to flag corrupted-label images, increasing after labels are corrupted, but they do not appear correlated with memorization values. The analysis uses the Fig. 2 noisy-label experiment and compares epoch-20 scores with memorization values for 1015 CIFAR100 examples.

  • F.2 Scores for Noisy Examples: The noisy-label analysis examines how adding noise affects EL2N scores in the experiment from Fig. 2.Results are shown in Fig. 17.
  • F.2 Scores for Noisy Examples: These findings suggest that EL2N scores can identify images with corrupted labels.The paper presents the higher scores for corrupted labels and post-corruption increases as two supporting results.
  • F.2 Scores for Noisy Examples: EL2N scores for images with corrupted labels tend to be higher than scores for images with regular labels.Figure 17 compares the score distributions for true and corrupted labels.
  • F.2 Scores for Noisy Examples: After an example’s label is corrupted, its EL2N score tends to increase relative to its pre-corruption score.Figure 17 also compares scores before and after noise is added.
  • G Comparison to memorization threshold: The memorization comparison uses memorization values for 1015 CIFAR100 examples and EL2N scores computed at epoch 20.The experiment replicates the referenced setting by training a ResNet50 on CIFAR-100.
  • G Comparison to memorization threshold: EL2N scores and memorization values do not appear to be correlated in the comparison shown in Fig. 18.The memorization values come from [38], while EL2N scores are computed for the provided examples.
Loading 2107.07075v2…