Source-linked AI summary
Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples
Haw-Shiuan Chang, Erik Learned-Miller, Andrew McCallum
TL;DR
Existing reweighting strategies favor easy or hard examples, but their usefulness depends on whether data are noisy or clean. This paper introduces two lightweight uncertainty-based methods for SGD and reports consistent accuracy and robustness improvements across six datasets and multiple architectures. The methods are presented as compatible with common training techniques, though implementation constraints limit direct sampling comparisons in some sequence-labeling settings.
Problem
Existing methods favor easier or harder examples in different situations, while the noise level of a training dataset is often unknown.
Method
The paper reweights classification examples using prediction-probability variance across SGD iterations or closeness of the correct-class probability to the decision threshold.
Results
The methods reduce generalization error by 1%–18% across six datasets and multiple architectures, including settings with few or no annotation errors.
Takeaways & Limitations
Active bias consistently improves accuracy and robustness when the classifier does not easily memorize all training samples.
Takeaways & Limitations
Sequence-labeling constraints require words in a sentence to be trained together, so some experiments compare loss modification with SGD-Scan rather than changing sampling distributions.
Abstract
from arXiv · showhide
Self-paced learning and hard example mining re-weight training instances to improve learning accuracy. This paper presents two improved alternatives based on lightweight estimates of sample uncertainty in stochastic gradient descent (SGD): the variance in predicted probability of the correct class across iterations of mini-batch SGD, and the proximity of the correct class probability to the decision threshold. Extensive experimental results on six datasets show that our methods reliably improve accuracy in various network architectures, including additional gains on top of other popular training techniques, such as residual learning, momentum, ADAM, batch normalization, dropout, and distillation.
1 Introduction
The paper motivates re-weighting training examples by uncertainty because preferring easy or hard samples works in different data conditions. It proposes lightweight uncertainty-based methods and reports accuracy gains across datasets and architectures.
- Easy-example preference can improve robustness on noisy or outlier-containing tasks, whereas hard-example sampling can accelerate SGD on cleaner data.
- Samples predicted correctly with high confidence may add little information, while consistently misclassified samples may be too difficult or noisy.
- The proposed methods emphasize uncertain samples according to previous prediction history, favoring examples sometimes predicted correctly and sometimes incorrectly.
- The paper proposes prediction-probability variance and decision-threshold closeness as lightweight methods for emphasizing uncertain samples during mini-batch SGD.
- 1%–18% lower generalization error is reported across six datasets and multiple architectures, including logistic regression, fully connected, convolutional, and residual networks.
2 Related work
Related work addresses SGD’s gradient variance and learning-rate tuning through optimization, sampling, and loss-reweighting methods. The paper positions its approach as balancing hard-example acceleration with easy-example robustness when dataset noise is unknown.
- Momentum, SVRG, and Prox-SVRG use information from previous iterations to stabilize stochastic optimization and reduce gradient variance.
- ADAGRAD adapts learning rates across dimensions, while ADAM combines adaptive techniques to address SGD learning-rate tuning.
- Importance-sampling methods reweight samples while adjusting learning rates to preserve unbiased gradient estimation and convergence guarantees.
- Other approaches bias the loss toward easier examples or gradually increase the loss of hard examples to improve robustness to outliers.
3 Methods
The paper introduces active-bias methods that emphasize uncertain training samples during mini-batch SGD, using prediction variance or proximity to the decision threshold. These methods build on sampling and weighting baselines while requiring lightweight, history-based uncertainty estimates.
- Baselines: Active bias re-weights or samples training examples according to uncertainty rather than using only uniform SGD or difficulty-based criteria.The baselines include uniform sampling, difficulty- and easiness-based sampling or weighting, and importance sampling.
- Prediction Variance: SGD-SPV selects examples using estimated prediction variance plus a confidence interval, balancing exploration and exploitation.The variance is estimated from stored prediction probabilities, with a smoothness constant preventing low-variance samples from being permanently excluded.
- Prediction Variance: SGD-WPV incorporates prediction variance into the loss through normalized sample weights, emphasizing examples with greater estimated uncertainty.The weighting method uses a normalization constant so the average sample weight remains fixed.
- Logistic Regression Analysis: For logistic regression, Gaussian-posterior and first-order Taylor approximations link prediction variance to boundary proximity and suggest reduced parameter uncertainty when high-variance samples are emphasized.The analysis states that prediction variance scales with p(yi|xi,w)^2(1-p(yi|xi,w))^2 and is larger near the boundary.
- Implementation: Burn-in training is used before applying active bias because early uncertainty estimates can emphasize incorrect examples and slow convergence.The implementation uses previous prediction histories, avoiding extra forward or backward passes; average per-sample overhead is O(E), where E is the number of epochs.
- Threshold Closeness: SGD-STC selects samples whose correct-class probability is close to the decision threshold, providing a simpler uncertainty-based alternative.The approach is motivated by the finding that prediction variance is larger near the classification boundary.
4 Experiments
Experiments across six datasets and multiple architectures show that active bias improves robustness and generalization, while its applicability depends on model complexity and task constraints. The methods require little additional computation and can complement existing training techniques.
- Experimental setup: Experiments span six datasets and shallow and deep architectures, using mini-batch training and existing implementations.The evaluated models include logistic regression, fully connected networks, CNNs, and residual networks; CNN and residual-network experiments use momentum or ADAM.
- Limitations and interpretation: The methods are vulnerable when a complicated model quickly reaches 100% training accuracy on a small dataset, because uncertain examples may be outliers.The experiments also note that emphasizing difficult examples can increase training accuracy without necessarily improving testing error.
- MNIST: On MNIST with 10% label reassignment, SGD-WPV improves SGD-Scan further, whereas SGD-SD seriously overfits the corrupted data.On the clean, high-accuracy task, selecting difficult instances can help, while uncertainty-based weighting also provides a boost.
- Image and classification experiments: Active bias improves performance across architectures, including about 0.5% gains in both 27-layer and 63-layer CIFAR-100 residual networks.The reported gains occur without changing the architectures or depending strongly on network depth and burn-in epochs.
- Sequence tagging tasks: The proposed weighting methods achieve the lowest error rate on both CoNLL 2003 and OntoNote 5.0 after simplifying models that were too complex for the tasks.The smaller models lose only around 1%–2% micro F1 relative to the original larger models.
- Distillation: Active bias remains compatible with distillation, with distillation plus SGD-WTC achieving the best MNIST performance.SGD-WTC and SGD-WD achieve similar or better improvements than adding distillation to SGD-Scan.
5 Conclusion
The paper presents active bias as a lightweight, compatible way to emphasize uncertain examples during neural-network training. Its experiments indicate that choosing the appropriate bias improves generalization across easy, noisy, and challenging settings when the classifier does not simply memorize the training data.
- Conclusion: Active bias is a simple, computationally lightweight sibling to momentum, dropout, batch normalization, and distillation that emphasizes uncertain examples.The methods are identified as SGD-*PV and SGD-*TC.
- Conclusion: When tasks are easy, emphasizing difficult examples works well; when datasets are challenging or noisy, emphasizing easier samples often performs better.The conclusion frames the appropriate bias in terms of training and testing accuracy.
- Conclusion: Active bias consistently produces more accurate and robust networks when classifiers do not memorize all training samples easily.The stated condition is high training accuracy together with low testing accuracy as the memorization scenario to avoid.
A Implementation details
The implementation estimates sample statistics from prior prediction histories and removes transient outliers before computing uncertainty-related quantities. This keeps the method lightweight, with overhead that can become constant when only recent epochs are used.
- General framework: The framework uses SGD-Scan by default when no sampling distribution is specified and assigns unit weight when no sample weight is specified.The threshold ϵ is set to the average of the current estimation.
- Statistics estimation: Prediction-history estimates exclude early transient behavior and remove current predictions whose deviation exceeds twice the median deviation.The same outlier-removal procedure is applied to difficulty, easiness, prediction variance, and threshold closeness.
- Computational overhead: Using only previous predictions avoids extra forward or backward passes, with average per-sample-per-epoch overhead O(E).E denotes the total number of training epochs.
- Computational overhead: For very large datasets and epoch counts, restricting outlier removal to recent epochs makes the overhead constant.For word-level uncertainty in sequence tagging, the implementation uses the latest five epochs.
B Experiment details
The experiments examine MNIST convergence, dataset properties, and testing performance across sampling methods. Changing the sampling distribution accelerates training more, while changing the loss function can yield better final results; training accuracy does not necessarily predict testing improvement.
- MNIST convergence curves compare error rates under different sampling and loss-function choices.The figures concern MNIST without noise.
- Changing the sampling distribution accelerates training more, whereas changing the loss function can produce better final error rates.
- Higher training accuracy from emphasizing difficult examples does not necessarily imply improved testing error.The paper reports average testing performance over the last 10 epochs in addition to the best testing performance within each trial.
- The dataset statistics include average sentence lengths of 11, 14, and 18 for Question Type, CoNLL 2003, and OntoNote 5.0.For the sequence-tagging datasets, each word is an instance with a label.
- Testing error rates are reported for the last 10 epochs across sampling methods, with sequence-tagging values expressed as 1−F1.Standard errors below 0.01% are omitted in the relevant tables.
C Proof sketch of Equation (5)
The supplied passages indicate that the proof sketch invokes assumptions associated with Equations (3) and (4) and connects the derivation to the loss function, but they provide no complete proof details.
- The proof sketch refers to an assumption made in connection with Equations (3) and (4).
- The derivation proceeds toward the loss function.
- The supplied experiment tables report training error rates for sampling methods, but do not state the proof of Equation (5).