PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Adaptive Training: beyond Empirical Risk Minimization

Lang Huang, Chao Zhang, Hongyang Zhang

arXiv:2002.10319v2cs.LGcs.CVstat.ML

TL;DR

Deep networks trained with ERM can overfit corrupted data, motivating a method that uses model predictions to adapt training labels and weights. Self-adaptive training improves generalization across corruption settings and applications, while its performance can degrade for extremely small underfitting models.

  • Problem

    ERM may overfit corrupted training data, making robust learning for clean test performance from noisy labels an important problem.

  • Method

    Self-adaptive training progressively corrects problematic labels using moving-average model predictions and dynamically downweights erroneous data.

  • Results

    Self-adaptive training improves generalization under random and adversarial noise, achieves up to 9.3% absolute CIFAR accuracy improvement, and reaches up to 50% relative improvement in selective classification.

  • Takeaways & Limitations

    The method provides a generic, computationally efficient approach for improving deep-network generalization under varied data corruptions.

  • Takeaways & Limitations

    The method can underperform ERM with extremely small models that underfit the training data, particularly models 10× smaller than standard ResNet-18.

Abstract

from arXiv · show

We propose self-adaptive training---a new training algorithm that dynamically corrects problematic training labels by model predictions without incurring extra computational cost---to improve generalization of deep learning for potentially corrupted training data. This problem is crucial towards robustly learning from data that are corrupted by, e.g., label noises and out-of-distribution samples. The standard empirical risk minimization (ERM) for such data, however, may easily overfit noises and thus suffers from sub-optimal performance. In this paper, we observe that model predictions can substantially benefit the training process: self-adaptive training significantly improves generalization over ERM under various levels of noises, and mitigates the overfitting issue in both natural and adversarial training. We evaluate the error-capacity curve of self-adaptive training: the test error is monotonously decreasing w.r.t. model capacity. This is in sharp contrast to the recently-discovered double-descent phenomenon in ERM which might be a result of overfitting of noises. Experiments on CIFAR and ImageNet datasets verify the effectiveness of our approach in two applications: classification with label noise and selective classification. We release our code at https://github.com/LayneH/self-adaptive-training.

1 Introduction

The paper identifies how ERM overfits corrupted training data and motivates self-adaptive training, which incorporates model predictions to improve robust generalization without extra computational cost.

  • ERM can nearly memorize randomly corrupted training data, achieving near-perfect training accuracy while exhibiting sharply different generalization behaviors across corruption types.
  • Early stopping can improve generalization, but identifying the right epoch is unclear and stopping may substantially hurt performance on clean validation data.
  • 80% clean-training accuracy exceeds the noisy set’s 60% clean-data fraction, suggesting model predictions can amplify useful information beyond the observed clean labels.
  • Self-adaptive training improves generalization under label-wise and instance-wise random noise and produces a single-descent error-capacity curve, unlike ERM’s possible noise-related double descent.
  • Across CIFAR and ImageNet applications, the method advances prior results in label-noise and selective classification, including up to 9.3% higher CIFAR accuracy and 50% relative improvement in selective classification.
  • Self-adaptive training progressively corrects problematic labels with moving-average predictions and downweights erroneous data through re-weighting.

2 Improved Generalization of Deep Networks

Self-adaptive training addresses ERM’s tendency to overfit corrupted data by combining model-prediction-based target updates with confidence-based sample re-weighting. Across random and adversarial noise settings, it improves generalization, removes double descent, and requires little additional computation.

  • Self-Adaptive Training: The method avoids early predictions’ instability and can completely change erroneous training labels when accumulated predictions become informative.The initial Es epochs allow the model to capture informative signals before target updates begin.
  • Self-Adaptive Training: Self-adaptive training combines exponentially smoothed model predictions with sample re-weighting to correct unreliable targets during training.Targets are fixed initially, then updated using a moving average; sample weights are based on the largest target probability.
  • Model Capacity: Self-adaptive training produces a monotonically decreasing error-capacity curve, whereas ERM exhibits double descent and generally higher test error under 15% label corruption.The experiment varies ResNet-18 width on CIFAR10; width 64 corresponds to the standard ResNet-18.
  • Limitations: The method can perform worse than ERM for extremely small underfitting models, but ERM wins only in cases roughly 10× smaller than standard ResNet-18.Ambiguous predictions can hinder training when model capacity is insufficient.
  • Adversarial Noise: Under adversarial training, self-adaptive training mitigates later overfitting and improves TRADES robust accuracy by 1%∼3%.TRADES’ robust accuracy peaks near its first learning-rate decay and decreases afterward without early stopping.

3 Application I: Classification with Label Noise

Self-adaptive training improves classification from corrupted labels by correcting problematic targets with model predictions while retaining ERM’s computational budget. Across CIFAR and ImageNet experiments, it improves generalization, recovers clean labels, and outperforms established methods.

  • Main results: Self-adaptive training outperforms state-of-the-art methods by 1%–9% on CIFAR10 and CIFAR100 while using the same computational budget as ERM.Unlike Joint Opt, DAC, and SELF, it does not require multiple training iterations.
  • Ablation study: Early stopping is a sub-optimal baseline, while combining exponential moving averages with sample re-weighting substantially improves performance under all tested noise rates.Removing either component hurts performance, with removing EMA causing a significant drop.
  • Experiments on ImageNet dataset: On ImageNet, self-adaptive training consistently improves ERM, including a 2% gain when 40% of training labels are corrupted.Performance is reported as top1 accuracy on the validation set.
  • Further inspection on self-adaptive training: With 40% corrupted labels, recovered accuracy reaches 94.6% on CIFAR10 and 81.1% on ImageNet.Recovered labels are obtained from moving-average targets and compared with clean labels.
  • Further inspection on self-adaptive training: Average sample weights are higher for samples whose recovered labels match their clean labels than for samples in non-diagonal label blocks.This pattern indicates that the method assigns less weight to likely erroneous data.

4 Application II: Selective Classification

Selective classification trades coverage against accuracy by allowing abstention, and self-adaptive training uses model predictions to learn when to abstain. Across CIFAR10 and Dogs vs. Cats, it improves over prior methods under varied coverage rates.

  • Selective classification trades prediction coverage against classification accuracy by allowing the classifier to output “don’t know” for uncertain samples.
  • A selective classifier combines a conventional c-class classifier f with a selection function g that reveals input uncertainty.
  • Self-adaptive training adds an abstention class and replaces g(·) with f(·)c, using model predictions as a learning guideline.
  • When ti,yi is small, the loss encourages abstention; when ti,yi approaches 1, it recovers standard cross-entropy classification.
  • Up to 50% relative improvement over all compared methods is achieved across coverage rates on CIFAR10 and Dogs vs. Cats.

5 Related Works

Related work examines how deep networks fit noise and how their risk changes with capacity, while robust-learning methods address corrupted data through training dynamics or model predictions.

  • Prior analyses show that deep networks can overfit random noise, challenging traditional explanations of their generalization.
  • Other studies report double-descent risk curves that challenge the conventional U-shaped bias-variance account.
  • Robust-learning research finds that networks fit clean samples before noisy ones, making early stopping a way to mitigate label-noise problems.
  • Some methods incorporate model predictions into training for learning from noisy data.

6 Conclusion

The paper analyzes deep-network generalization under corruption and proposes Self-Adaptive Training to incorporate model predictions during training. It reports improved generalization and advances in label-noise and selective-classification applications.

  • The paper characterizes intrinsic failure cases of standard ERM under data corruptions by analyzing deep-network training dynamics.
  • Self-Adaptive Training incorporates model predictions into the training process to improve generalization under various corruptions.
  • The approach is applied to classification with label noise and selective classification, where it significantly advances the state of the art.

Broader Impact

The authors position self-adaptive training as a low-cost approach for robust learning from corruptions common in uncurated datasets. They caution that most evaluations use artificial corruptions, limiting representativeness for natural noise.

  • Potential corruptions are common in real-world, uncurated large-scale datasets because data acquisition is error-prone.
  • Self-adaptive training is designed to handle varied noise types and levels without extra computational cost when the noise scheme is unknown.
  • Most studies evaluate artificial corruptions, which may not represent natural noise in practice, although ImageNet contains annotation errors.

A Experimental Setups

The experiments use fixed optimization settings and adjust self-adaptive training hyperparameters according to model capacity.

  • Models are trained with Adam for 4,000 epochs using a fixed learning rate of 0.0001, batch size 128, common augmentation, and zero weight decay.
  • For standard ResNet-18 with width 64, the approach uses Es = 40 and α = 0.9, adjusting them for other models according to capacity.

A.2 Adversarial training

The adversarial-training setup builds on TRADES and evaluates self-adaptive training by replacing its natural-error term, alongside specified CIFAR10 training configurations.

  • TRADES decomposes robust error into natural error and boundary error, balancing natural accuracy against adversarial robustness with 1/λ.
  • Self-adaptive training is evaluated by replacing TRADES’ first cross-entropy term while retaining the adversarial robustness term.
  • CIFAR10 experiments use WRN-34-10, 100 epochs, batch size 128, and PGD-generated adversarial examples under the official TRADES implementation.
  • The supplied setup also specifies ResNet-50 training with SGD, batch size 768, warmup, cosine annealing, and standard image augmentation.
  • Selective-classification experiments use VGG-16 with batch normalization and dropout, trained for 300 epochs with SGD.

B.1 ERM may suffer from overfitting of noise

The passages show that ERM can fit corrupted data while exhibiting unstable generalization, motivating model-prediction-based training rather than relying solely on early stopping.

  • ERM can achieve nearly perfect accuracy on noisy training data while exhibiting different generalization behaviors across corruption types.
  • Self-adaptive training substantially diminishes ERM’s epoch-wise double-descent phenomenon on CIFAR10 with 15% randomly corrupted labels.
  • Clean-training and validation accuracy can exceed the clean-data proportion during the transition between underfitting and overfitting.
  • Early stopping may be sub-optimal because it can hurt clean-set performance under corruption schemes beyond corrupted labels.
  • Model predictions can capture and amplify useful signals in heavily corrupted training data, including corruption types beyond label noise.

B.2 Improved generalization of self-adaptive training on random noise

Across random-noise settings, self-adaptive training improves generalization over ERM and remains effective under extreme noise and limited augmentation, while its error-capacity behavior differs from ERM’s double descent.

  • Self-adaptive training improves generalization over ERM even when random noise dominates the CIFAR10 training data at an 80% noise rate.
  • Without data augmentation, both methods incur higher errors, but self-adaptive training still provides gains over standard ERM.
  • The study reports parameter sensitivity across datasets and noise rates and evaluates test accuracy under varied uniform label-noise levels.
  • Self-adaptive training exhibits a single-descent error-capacity curve, contrasting with ERM’s reported epoch-wise double-descent behavior.

B.4 Cooperation with symmetric cross entropy

Self-adaptive training can cooperate with Symmetric Cross Entropy (SCE), combining label correction with a noise-robust loss without extra computational cost. This combination yields further gains, including a 1.5% improvement under 60% label noise on CIFAR100.

  • SCE combines standard cross entropy with its reversed version, using training targets and model predictions.
  • Self-adaptive training cooperates with SCE without extra cost and can obtain further performance improvements.
  • Setup: w1 and w2 are set to 1 and 0.1, respectively, for all experiments.
  • Results: 1.5% improvement is achieved when 60% label noise is injected into the CIFAR100 training set.
Loading 2002.10319v2…