Source-linked AI summary
Learning to Reweight Examples for Robust Deep Learning
Mengye Ren, Wenyuan Zeng, Bin Yang, Raquel Urtasun
TL;DR
The paper tackles training-set bias and label noise without manually tuning reweighting schedules or regularization parameters. It learns example weights online from gradient directions by minimizing clean validation loss, and reports robust performance across class imbalance and corrupted-label settings, with added computational overhead.
Problem
Deep networks can overfit training-set biases and label noise, while existing example-reweighting methods require additional hyperparameter tuning and may rely on conflicting loss-based assumptions.
Method
An online meta-learning algorithm takes a gradient step on current mini-batch example weights to minimize loss on a small clean unbiased validation set.
Results
The method ranks first on UNIFORMFLIP and BACKGROUNDFLIP, wins more than 3% on CIFAR-100 against the state-of-the-art method, and remains robust as corruption increases.
Takeaways & Limitations
The approach improves robustness to class imbalance, noisy labels, and their combination while applying directly to deep architectures without additional hyperparameter search.
Takeaways & Limitations
The method requires approximately 3× regular training time because it adds training, validation, and backward-on-backward passes.
Abstract
from arXiv · showhide
Deep neural networks have been shown to be very powerful modeling tools for many supervised learning tasks involving complex input patterns. However, they can also easily overfit to training set biases and label noises. In addition to various regularizers, example reweighting algorithms are popular solutions to these problems, but they require careful tuning of additional hyperparameters, such as example mining schedules and regularization hyperparameters. In contrast to past reweighting methods, which typically consist of functions of the cost value of each example, in this work we propose a novel meta-learning algorithm that learns to assign weights to training examples based on their gradient directions. To determine the example weights, our method performs a meta gradient descent step on the current mini-batch example weights (which are initialized from zero) to minimize the loss on a clean unbiased validation set. Our proposed method can be easily implemented on any type of deep network, does not require any additional hyperparameter tuning, and achieves impressive performance on class imbalance and corrupted label problems where only a small amount of clean validation data is available.
1. Introduction
The paper addresses training-set bias from distribution mismatch, class imbalance, and label noise by learning example weights from a small clean validation set rather than relying on manually designed loss-based rules.
- Training-set distribution mismatch can produce biased models when training data differ from the evaluation distribution.
- Class imbalance can cause poor recognition of rarely observed classes despite their practical importance.
- Label noise can hurt generalization because deep networks may fit arbitrary label-flipping noise.
- Loss-based reweighting methods assign example weights using training losses, including boosting, hard-negative mining, and self-paced learning.
- Training-loss heuristics conflict across noisy-label and class-imbalance settings, motivating a method based on an unbiased clean validation objective.
- The proposed online meta-learning method dynamically reweights each training batch using a small unbiased validation set and improves robustness to training-set biases.
2. Related Work
Prior work reweights examples through sampling, loss-based heuristics, or structured objectives, while this paper frames reweighting as online meta-learning driven by validation performance.
- Importance sampling and boosting reweight examples to match distributions or emphasize harder cases.
- Robust losses and self-paced learning instead downweight high-loss examples or prioritize easier examples during training.
- Class imbalance and noisy labels have been addressed through resampling, cost-sensitive weighting, structured margins, and practical noise-robust methods.
- The proposed approach uses validation loss as a meta-objective, operates online during ordinary training, and avoids additional hyperparameters and an expensive offline stage.
3. Learning to Reweight Examples
The method derives an online approximation to validation-driven example reweighting, using gradient information and automatic differentiation during standard deep-network training. It provides implementation details, overhead analysis, and convergence results under stated assumptions.
- Learning to Reweight Examples: The algorithm derives an online approximation to a meta-learning objective that can fit regular supervised training and any deep network type.
- Objective and assumptions: A small unbiased clean validation set supplies the objective used to select training-example weights.
- Objective and assumptions: The method minimizes a weighted training loss instead of the equally weighted average loss, with nonnegative example weights treated as validation-selected hyperparameters.
- Online approximation: At each iteration, it compares training-example descent directions with the validation-loss descent direction to estimate weights online.
- Online approximation: Batch normalization constrains nonzero weights to have unit L1 norm, preserving the original training step scale and canceling the meta learning-rate parameter.
- Network implementation: For multilayer networks, meta-gradients combine similarities between layer inputs and between training and validation gradient directions.
- Network implementation: The implementation uses backward-on-backward automatic differentiation after unrolling the training gradient graph, with approximately 3× regular training time.
4. Experiments
Experiments evaluate the method on class imbalance, noisy labels, and their combination using MNIST and CIFAR benchmarks. Across these settings, the method remains effective with small clean validation sets and is less sensitive to noise type and overfitting.
- Experimental setup: The experiments cover class imbalance, noisy labels, and combined imbalance-noise settings on MNIST and CIFAR image-classification benchmarks.The study uses deep CNNs and compares against established reweighting, noise-robustness, and custom baseline methods.
- Class imbalance: At a 200:1 class-imbalance ratio, the method increases error by only around 2%, while resampling and hard negative mining perform much worse.The method retains samples based on their usefulness for validation loss rather than discarding examples by class or training loss.
- Noisy labels: The method ranks first on both UNIFORMFLIP and BACKGROUNDFLIP, and wins by more than 3% over the state-of-the-art method on CIFAR-100.These results indicate performance across different label-noise types under the reported experimental settings.
- Weight behavior: The model assigns most noisy images zero weight and reliably detects images flipped to the background class.This behavior is observed using unseen hyper-validation images and batches restricted to a single non-background class.
- Robustness to overfitting noise: Early stopping improves S-MODEL by over 10% on CIFAR-10, while baseline models overfit noise and the proposed model maintains validation accuracy until termination.The comparison includes a S-MODEL configured with oracle noise-ratio information unavailable to the proposed method.
- Clean validation-set size: With 15 validation images for all classes, performance drops only 2%, and it does not improve after more than 100 validation images.Fine-tuning baselines on 15 images causes a significant performance drop, with performance catching up around 1,000 validation images.
5. Conclusion
The paper proposes an online meta-learning algorithm that automatically reweights training examples to improve robustness across class imbalance and noisy-label settings. It is designed to apply directly to any deep learning architecture without additional hyperparameter tuning.
- The proposed method automatically reweights training examples through online meta-learning rather than manually designed reweighting objectives.
- The algorithm shows superior performance on class imbalance, noisy labels, and their combination.
- The method can be directly applied to any deep learning architecture and trained end-to-end without additional hyperparameter tuning.
A. Reweighting in an MLP
This section introduces the complete derivation for calculating example weights in an MLP network.
- The paper derives how to calculate example weights for an MLP network.
B. Convergence of our method
Under smoothness and bounded-gradient assumptions, the method guarantees that validation loss decreases monotonically and reaches a stationary point in expectation.
- The convergence analysis assumes a Lipschitz-smooth validation loss and training-example losses with σ-bounded gradients.
- If α_t ≤ 2n/(Lσ^2), the validation loss monotonically decreases for every sequence of training batches.
- In expectation, validation loss remains unchanged if and only if its gradient is zero.
- The proof models a small clean validation set with M examples and adds it to N−M training examples to form the training dataset.
- For simplicity, the analysis assumes the validation data is a subset of the training data and uses mini-batches of size n.
- The validation loss cannot increase under the analyzed update because the derived inequality gives G(θ^(t+1)) ≤ G(θ^t).
C. Convergence rate of our method
The convergence-rate analysis establishes that the algorithm reaches a point with a minimum expected gradient-related quantity under the stated conditions.
- Theorem 2 analyzes Algorithm 1 under the previously stated conditions and derives a convergence guarantee over T steps.
- The analysis allows a constant learning rate α < 2n/(Lσ^2) or a positive decaying sequence bounded by that constant.
- The proof uses batches that contain particular validation examples with non-zero sampling probabilities.
- The resulting conclusion is that the algorithm can achieve the minimum of the analyzed quantity.