Source-linked AI summary
Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt
Sören Mindermann, Jan Brauner, Muhammed Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N. Gomez, Adrien Morisot, Sebastian Farquhar, Yarin Gal
TL;DR
Training on web-scale data is slow, while existing selection methods can prioritize redundant, noisy, or less task-relevant examples. RHO-LOSS estimates which points would most reduce holdout loss and selects points that are learnable, worth learning, and not yet learnt. Across datasets and architectures it speeds training and improves accuracy, including 2% higher final accuracy on Clothing-1M than uniform selection.
Problem
Training can take weeks or months, and existing methods may prioritize hard examples that are noisy or less task-relevant.
Method
RHO-LOSS estimates how much each candidate would reduce generalization loss without actually training on it, selecting approximately optimal points.
Results
2% higher final accuracy than uniform selection on Clothing-1M, while RHO-LOSS consistently outperforms prior art and speeds training across datasets, modalities, architectures, and hyperparameters.
Takeaways & Limitations
RHO-LOSS prioritizes points that are not yet learnt, learnable, and worth learning, avoiding redundant, noisy, and less relevant points.
Takeaways & Limitations
The effect of subset selection on performance for minority groups remains important to understand because selection may prioritize or deprioritize rare groups.
Abstract
from arXiv · showhide
Training on web-scale data can take months. But most computation and time is wasted on redundant and noisy points that are already learnt or not learnable. To accelerate training, we introduce Reducible Holdout Loss Selection (RHO-LOSS), a simple but principled technique which selects approximately those points for training that most reduce the model's generalization loss. As a result, RHO-LOSS mitigates the weaknesses of existing data selection methods: techniques from the optimization literature typically select 'hard' (e.g. high loss) points, but such points are often noisy (not learnable) or less task-relevant. Conversely, curriculum learning prioritizes 'easy' points, but such points need not be trained on once learned. In contrast, RHO-LOSS selects points that are learnable, worth learning, and not yet learnt. RHO-LOSS trains in far fewer steps than prior art, improves accuracy, and speeds up training on a wide range of datasets, hyperparameters, and architectures (MLPs, CNNs, and BERT). On the large web-scraped image dataset Clothing-1M, RHO-LOSS trains in 18x fewer steps and reaches 2% higher final accuracy than uniform data shuffling.
1. Introduction
Web-scale training is costly because datasets contain noisy, redundant, and difficult-to-use examples, while existing selection strategies prioritize either easy or hard points. RHO-LOSS instead targets points that are learnable, task-relevant, and not yet learnt, improving training efficiency and accuracy.
- Training state-of-the-art models on web-scraped data can take weeks or months, making training time a core development constraint.
- Web-scraped datasets contain noisy samples with incorrect or ambiguous labels and redundant samples that can often be omitted without losing performance.
- Curriculum learning prioritizes easy points but lacks a mechanism to skip examples that are already learnt, whereas hard-example selection can prioritize noisy or less task-relevant points.
- RHO-LOSS estimates which points would most reduce generalization loss and selects points that are learnable, worth learning, and not yet learnt.
- 18x fewer training steps are needed for RHO-LOSS to reach target accuracy than uniform selection on Clothing-1M, while final accuracy is 2% higher.
2. Background: Online Batch Selection
Online batch selection samples a large candidate batch, ranks its examples with a label-aware function, and trains on only the highest-ranked subset. This replaces uniform mini-batch selection with selective gradient updates.
- At each step, online batch selection uniformly pre-samples a large batch Bt of size nB > nb from the training data.
- A label-aware selection function ranks the candidates, and the top nb points form the training batch bt.
- The model performs a gradient step on the mini-batch loss over the selected points before sampling the next large batch without replacement.
3. Reducible Holdout Loss Selection
RHO-LOSS approximates the holdout-loss reduction from training on each candidate without actually training separately on every candidate. Its score combines current training loss with an irreducible holdout loss to favor useful, learnable examples.
- RHO-LOSS selects candidate points that would approximately minimize negative log-likelihood on a holdout set drawn from the training distribution.
- The selection function is derived with probabilistic modelling so candidate points can be evaluated without training the current model on each one.
- The reducible holdout loss is the difference between a point's training loss and its irreducible holdout loss, L[y | x; Dho].
- An irreducible-loss model estimates holdout loss efficiently and can be smaller and reused across many target-model runs, amortizing its cost.
- Understanding reducible loss: Redundant points have low training loss, while noisy and less relevant points have high irreducible holdout loss; all therefore receive low reducible loss.
- Selecting multiple points concurrently: For batch selection, RHO-LOSS chooses the top nb-scoring points from a randomly pre-sampled candidate set Bt, approximately assuming points have little effect on one another's scores.
4. Experiments
Across seven datasets and varied training settings, RHO-LOSS accelerates training by selecting points that are non-noisy, task-relevant, and non-redundant. Its benefits persist with cheap irreducible-loss models, across target architectures and hyperparameters, and increase on noisy or web-scraped data.
- Experiments: RHO-LOSS was evaluated on seven datasets against uniform sampling, loss, gradient norm, gradient norm with importance sampling, and Selection-via-Proxy.The evaluation covered controlled and real-world conditions, including Clothing-1M with noisy web-scraped labels.
- Impact of Approximations: RHO-LOSS approximations retained similar point rankings while replacing Bayesian inference with a single model and one SGD step, achieving Spearman coefficients of 0.75 and 0.76.These approximations move the method toward standard neural-network fitting with AdamW.
- Cheap Irreducible Loss Models & Robustness: A small CNN irreducible-loss model used 21x fewer parameters and 29x fewer forward-pass FLOPs than ResNet-18 while accelerating training as much or more.The smaller model had lower final accuracy than the target model, but still supported effective selection.
- Cheap Irreducible Loss Models & Robustness: A single small CNN irreducible-loss model accelerated seven target architectures and reused selection across nearly all tested batch sizes, learning rates, and weight-decay settings.The few settings without speedup were settings where uniform training performed very poorly.
- Properties of RHO-LOSS & Other Selection Functions: RHO-LOSS prioritized non-noisy, task-relevant, non-redundant points, whereas loss and gradient norm selected more corrupted-label and low-relevance examples than uniform sampling.All methods selected fewer redundant points than uniform sampling, with redundancy measured by the proportion already classified correctly.
- Speedup: 18x fewer steps were required on average to reach uniform selection’s peak accuracy across Clothing-1M architectures, while final accuracy improved from 70% to 72%.On a ResNet-50, RHO-LOSS reached that accuracy in 2 epochs and used 2.7x fewer FLOPs; speedups also grew when 10% label noise was added.
5. Related Work
RHO-LOSS is positioned as an alternative to selection functions based on computational efficiency, point difficulty, uncertainty, or gradient variance. Existing approaches differ in whether they use labels and which examples they prioritize.
- Time-efficient data selection: Selection methods can reduce cost through low-precision or parallelized forward passes, since selection needs losses rather than backpropagation activations.A forward pass typically takes roughly three times less time than a forward-backward pass, with further acceleration possible.
- Compute-efficient data selection: Selection costs can be reduced by reusing prior losses, predicting losses with a small model, or performing one-time core-set selection.These approaches target the computational overhead of repeatedly evaluating candidate points.
- Data selection functions: Data selection functions prioritize different point properties: hard examples use loss or uncertainty, while other methods select points using label information.The paper frames RHO-LOSS as an alternative selection function rather than as a separate training objective.
- Variance reduction methods: Online batch selection can reduce stochastic-gradient variance by importance-sampling points with high approximate gradient norms and down-weighting them for debiasing.These methods are also used in reinforcement learning.
6. Conclusion
The conclusion presents RHO-LOSS as a theoretically grounded selection function for reducing excessive training time. It reports larger speedups on noisy and web-scraped data than on clean data and recommends combining the method with cheaper selection techniques.
- 6. Conclusion: RHO-LOSS enables substantial speedups on clean data and even larger speedups on noisy and web-scraped data.The conclusion connects these speedups to a theoretically grounded selection function.
- 6. Conclusion: The selection function should be combined with methods for cheap and fast selection to maximize speedups.The paper identifies this combination as a practical direction for reducing selection overhead.
Ethics Statement
The reported experiments include vision and NLP evaluations, including settings with noisy data and no holdout data. The supplied passages do not provide an ethics-specific discussion beyond these experimental conditions.
- Experimental scope: Vision evaluations measure gradient steps needed to reach target test accuracy, with lower values indicating faster training.Figure 4 covers web-scraped noisy data, clean datasets, and added 10% label noise; results use means and minima–maxima ranges across three seeds.
- Experimental scope: NLP evaluations cover CoLA grammatical acceptability and SST2 sentiment classification using gradient steps to reach target test accuracy.Figure 5 reports means and standard deviations across four or more random seeds, using half of each dataset for training.
- No-holdout evaluation: Without holdout data, RHO-LOSS performs best in both epochs required and final accuracy.Table 3 reports results averaged across two to three seeds, with lower epoch counts preferred.
B. Experiment Details
The experiments span multiple architectures, optimizers, data-augmentation settings, and irreducible-loss-model procedures. Implementation choices include separate BatchNorm statistics for selection and model training.
- Architectures: Experiments use MLPs for QMNIST, adapted ResNet variants for vision datasets, a pretrained ResNet-50 target on Clothing-1M, and pretrained ALBERT v2 for NLP.The irreducible-loss model on Clothing-1M is a randomly initialized ResNet-18.
- Hyperparameters: Vision models use AdamW with specified default hyperparameters, while NLP uses ALBERT v2 with a 2 · 10^-5 learning rate and weight decay of 0.02.The selection batch is configured as 10% of the points, and all weights are fine-tuned for NLP.
- Hyperparameters: Experiments use between 2 and 10 random seeds.The number of seeds varies across experiments.
- Data augmentation: CIFAR-10, CIFAR-100, and CINIC-10 use random crops and horizontal flips for both irreducible-loss-model training and main training.Irreducible losses are computed once at the start using unaugmented images to reduce compute.
- Irreducible loss model training: Irreducible-loss-model checkpoints are selected by lowest holdout loss on the training holdout set rather than highest accuracy.The paper reports that holdout loss typically reaches its minimum early, saving compute.
- BatchNorm: BatchNorm statistics are computed separately across the large selection batch and the small training batch.The paper notes that these choices can substantially affect performance.
C. Robustness to Noise
RHO-LOSS is evaluated under multiple label-noise patterns, including uniform and structured noise, to test whether selection remains robust when labels are corrupted.
- RHO-LOSS remains robust across varied label-noise patterns, whereas other selection methods degrade.Results are summarized across three random seeds, with lines showing means and shaded areas showing minima and maxima.
- The experiments add uniform noise to 10% of training points and structured noise affecting easily confused classes.
D. Irreducible Holdout Loss Approximation
The irreducible holdout loss approximation replaces a jointly updated holdout term with a fixed holdout-only term, reducing computation and preventing deterioration during target-model training.
- The approximation avoids recomputing the holdout term whenever the acquired training set changes, substantially reducing computation.It approximates the model trained on acquired and holdout data with one trained only on the holdout set.
- Updating the irreducible loss model on acquired data is empirically unnecessary: approximation does not reduce target-training speed or final accuracy across CIFAR-10, CIFAR-100, and CINIC-10.The comparison is reported in Table 4, which records epochs needed to reach target test accuracy; NR means the target was not reached.
- The original selection function subtracts holdout loss conditioned on both acquired data and holdout data, whereas the approximation uses holdout-only loss.
- The approximation prevents repeated selection of undesirable points when stochastic gradient descent is used instead of Bayesian updating.With a constant second term, previously selected undesirable points become less likely to remain highly favored after the target loss decreases.
- The approximation prevents deterioration of the irreducible loss model during target-model training.The irreducible loss model is trained on a holdout set before target training and is not updated on acquired target data.
E. Experimental Details for Assessing Impact of Approximations
The approximation study uses QMNIST with 10% uniform label noise, controlled model configurations, and batchwise acquisition followed by training on the acquired points.
- The experiments use QMNIST with uniform label noise applied to 10% of the dataset and a batch size of 1000.
- The deep ensemble contains five 3-layer MLPs with 512 hidden units, while the weaker irreducible loss model is an MLP with 256 hidden units.
- For Approximation 0, models acquire 10% of points per batch, then train to convergence on each acquired batch for up to five epochs.The irreducible loss model trains on holdout data union acquired data, while the target model trains only on acquired data.
F. Ablation of percentage selected
The study varies the percentage of evaluated points selected per batch and compares RHO-LOSS with uniform sampling and active-learning acquisition methods on MNIST and CIFAR10.
- The percentage of evaluated points selected for training is a RHO-LOSS hyperparameter set to 0.1 in the reported experiments without tuning.On two-thirds of datasets, other values further improve performance.
- Lower selection percentages typically trade lower training time for greater compute cost.The parameter is intended to let practitioners choose their preferred training-time and computation tradeoff.
- The selected-per-batch percentage affects datasets differently, as shown in the ablation experiments.
- Active-learning methods use label-free acquisition, whereas RHO-LOSS is label-aware.
- RHO-LOSS accelerates training across MNIST and CIFAR10, while active-learning baselines accelerate MNIST but not CIFAR10.The curves compare RHO-LOSS, uniform sampling, and active-learning baselines.