Source-linked AI summary
ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, Colin Raffel
TL;DR
Semi-supervised learning seeks to exploit abundant unlabeled data when labeled examples are limited. ReMixMatch adds distribution alignment and augmentation anchoring, achieving state-of-the-art accuracy across labeled-data amounts, including 93.73% with 250 CIFAR-10 labels.
Problem
Semi-supervised learning needs effective ways to leverage unlabeled data when labeled examples are limited or expensive to obtain.
Method
ReMixMatch improves MixMatch with distribution alignment, augmentation anchoring, and an online-learned CTAugment policy for strong augmentations.
Results
93.73% accuracy with 250 CIFAR-10 labels establishes state-of-the-art performance across all labeled-data amounts.
Takeaways & Limitations
ReMixMatch reduces the labeled data needed for strong semi-supervised performance, reaching MixMatch’s 4,000-label CIFAR-10 accuracy with 250 labels.
Abstract
from arXiv · showhide
We improve the recently-proposed "MixMatch" semi-supervised learning algorithm by introducing two new techniques: distribution alignment and augmentation anchoring. Distribution alignment encourages the marginal distribution of predictions on unlabeled data to be close to the marginal distribution of ground-truth labels. Augmentation anchoring feeds multiple strongly augmented versions of an input into the model and encourages each output to be close to the prediction for a weakly-augmented version of the same input. To produce strong augmentations, we propose a variant of AutoAugment which learns the augmentation policy while the model is being trained. Our new algorithm, dubbed ReMixMatch, is significantly more data-efficient than prior work, requiring between $5\times$ and $16\times$ less data to reach the same accuracy. For example, on CIFAR-10 with 250 labeled examples we reach $93.73\%$ accuracy (compared to MixMatch's accuracy of $93.58\%$ with $4{,}000$ examples) and a median accuracy of $84.92\%$ with just four labels per class. We make our code and data open-source at https://github.com/google-research/remixmatch.
1 INTRODUCTION
ReMixMatch improves semi-supervised learning by combining distribution alignment with augmentation anchoring and CTAugment. It achieves state-of-the-art accuracy across labeled-data amounts, including 93.73% accuracy with 250 CIFAR-10 labels versus 88.92% previously.
- Key contributions: Distribution alignment matches aggregated unlabeled-data predictions to the marginal ground-truth class-label distribution.It can be added straightforwardly to MixMatch.
- Key contributions: Augmentation anchoring uses weakly augmented predictions as targets for multiple strongly augmented versions of each unlabeled input.It replaces MixMatch’s consistency-regularization component.
- Key contributions: CTAugment learns an augmentation policy alongside model training to generate strong augmentations for semi-supervised learning.It is introduced as a control-theory-based variant of AutoAugment.
- Empirical validation: 93.73% accuracy with 250 labels on CIFAR-10 exceeds the previous state-of-the-art of 88.92%.Fully supervised classification with 50,000 labels reaches 96.09%.
2 BACKGROUND
Semi-supervised learning uses unlabeled data to improve performance on labeled data, commonly through guessed labels or label-free objectives. ReMixMatch builds on MixMatch, which combines guessed labels with fully supervised training techniques.
- Background: Semi-supervised learning aims to use unlabeled data to improve performance on labeled data through guessed labels or label-free objectives.The background focuses on methods relevant to ReMixMatch, especially MixMatch components.
- Consistency Regularization: Consistency regularization enforces unchanged model outputs when inputs are perturbed.This approach includes stochastic transformations and perturbations, the Π-Model, adversarial perturbations, and dropout-based perturbations.
- Entropy Minimization: Entropy minimization encourages low-entropy, high-confidence predictions on unlabeled data so classes remain well-separated.One implementation adds a loss term minimizing the entropy of the predicted class distribution.
- Standard Regularization: Standard regularization can apply to labeled and unlabeled data, including weight decay and MixUp training on interpolated inputs and labels.Weight decay minimizes the L2 norm of parameters, while MixUp uses linear interpolants.
- MixMatch: MixMatch generates guessed labels for unlabeled examples and trains jointly on original labeled data and those guessed labels using supervised techniques.It averages predictions across weak augmentations, sharpens the resulting distribution, and applies MixUp to labeled and guessed-label examples.
3 REMIXMATCH
ReMixMatch extends MixMatch with distribution alignment and augmentation anchoring, using stronger online-learned augmentations to improve semi-supervised learning. The method adjusts unlabeled predictions toward the labeled class distribution and enforces consistency across augmentations.
- Distribution Alignment: Distribution alignment rescales each unlabeled prediction by the ratio of the labeled marginal p(y) to the running unlabeled marginal, then renormalizes it.The running unlabeled marginal averages predictions over the previous 128 batches, while p(y) is estimated from labeled examples seen during training.
- Distribution Alignment: ReMixMatch uses the adjusted prediction as an unlabeled label guess before sharpening and subsequent processing.This integrates distribution alignment directly into the MixMatch pipeline for processed labeled and unlabeled examples.
- Augmentation Anchoring: Augmentation anchoring enforces consistency between multiple strongly augmented unlabeled inputs and a weakly augmented version’s prediction.The algorithm applies strong augmentation K times to each unlabeled example and computes the reference prediction from its weak augmentation.
- CTAugment: CTAugment dynamically infers transformation magnitudes online without reinforcement learning or a supervised proxy task, enabling more aggressive augmentations in semi-supervised training.It uniformly samples transformations and adapts magnitude distributions during model training.
- Augmentation Anchoring: K = 8 improves augmentation anchoring, which permits replacing MixMatch’s unlabeled-data mean squared error with standard cross-entropy while maintaining stability.MixMatch performed best at K = 2, whereas augmentation anchoring benefited from K = 8.
4 EXPERIMENTS
Experiments evaluate ReMixMatch on standard semi-supervised benchmarks using a consistent Wide ResNet-based codebase and training setup. ReMixMatch achieves strong data efficiency across CIFAR-10, SVHN, STL-10, and extremely low-label settings.
- Experimental setup: ReMixMatch is evaluated with a Wide ResNet-28-2 architecture and the same codebase and training algorithm across experiments.The model has 1.5 million parameters and follows Realistic Semi-Supervised Learning evaluation recommendations.
- CIFAR-10: ReMixMatch sets the state of the art on CIFAR-10 for every tested number of labeled examples and is 16× more data efficient than MixMatch.At 250 labeled examples, ReMixMatch reaches the same accuracy as MixMatch at 4,000 labeled examples.
- SVHN: ReMixMatch reaches state-of-the-art performance on SVHN with 250 labeled examples and falls within the margin of error otherwise.The comparison uses results shown in Table 1 under the stated experimental setup.
- STL-10: On STL-10, ReMixMatch reduces the error rate by a factor of two compared to MixMatch.This comparison uses the same WRN-37-2 network with 23.8 million parameters and five of the ten predefined labeled folds.
- Extremely low-label settings: With four labels per CIFAR-10 class, ReMixMatch obtains a 15.08% median-of-five error rate after changing λr from 0.5 to 2.The same change enables SVHN training with 40 total labels, yielding 3.48% error, or 2.81% with the extra dataset.
5 CONCLUSION
ReMixMatch extends MixMatch with augmentation anchoring and distribution alignment, substantially reducing the labeled data needed for semi-supervised classification. On CIFAR-10, it beats MixMatch using 250 labeled examples compared with MixMatch’s 4,000.
- 5 CONCLUSION: ReMixMatch introduces augmentation anchoring and distribution alignment to MixMatch.These techniques are presented as the basis for the method’s improved data efficiency.
- 5 CONCLUSION: ReMixMatch reduces the quantity of labeled data needed by a large factor compared to prior work.The conclusion frames this reduction as part of a broader shift in semi-supervised learning.
- 5 CONCLUSION: 250 labeled examples allow ReMixMatch to beat MixMatch at 4,000 labeled examples on CIFAR-10.The passage uses this comparison as a concrete example of ReMixMatch’s data efficiency.
A PROOF OF EQUATION 2
The proof of Equation 2 follows Bridle et al. (1992), beginning from the relevant definition and successively rewriting terms and integrals as expectations. It concludes with expressions involving conditional prediction probabilities and their expectations, including Equation 11.
- The proof follows closely the approach of Bridle et al. (1992) and begins with a definition.
- The derivation rewrites the terms and then expresses both integrals as expectations.
- The resulting expressions use p(y|x), p(yi|x), and their expectations over x, culminating in Equation 11.
B FULL 40 LABEL RESULTS
This section reports ReMixMatch results using only 40 labeled examples, sorting error rates across five training-data splits. Because so few labels are sampled randomly, substantial variance is expected.
- Experiment setup: ReMixMatch is evaluated with 40 labeled examples across five different training-data splits.The splits are 40-label subsets of the training data.
- Results: The results are sorted by error rate across the five splits.Table 4 presents the sorted error rates for ReMixMatch with 40 labeled examples.
- Results: High variance is expected because the labeled examples are randomly selected in such a small-label regime.The experiment uses only five randomly chosen 40-label subsets.
C TRANSFORMATIONS INCLUDED IN CTAUGMENT
CTA augment’s transformation parameters are discretized into 17 equal bins, except for Rescale’s M parameter, which uses six interpolation options.
- C TRANSFORMATIONS INCLUDED IN CTAUGMENT: 17 equal bins discretize all listed transformation parameters, while Rescale’s M parameter has six options: anti-alias, bicubic, bilinear, box, hamming, and nearest.The six Rescale options are anti-alias, bicubic, bilinear, box, hamming, and nearest.
D MEASURING THE EFFECT OF DISTRIBUTION ALIGNMENT
The experiment tests whether distribution alignment makes the marginal distribution of model predictions match the true marginal class distribution. On CIFAR-10 with 250 labels, the KL divergence between these distributions is significantly smaller throughout training with distribution alignment.
- Distribution alignment aims to match the model’s marginal prediction distribution ˜p(y) to the true marginal class distribution p(y).The effect is evaluated using the KL divergence between ˜p(y) and p(y).
- The comparison tracks KL divergence over training with and without distribution alignment on CIFAR-10 using 250 labels.The corresponding comparison is shown in Figure 3.
- The KL divergence between ˜p(y) and p(y) is significantly smaller throughout training with distribution alignment.
E CTAUGMENT PARAMETERS EFFECTS
On CIFAR10 with 250 labels, CTAugment’s exponential weight decay ρ has little effect, whereas depth and threshold significantly affect results. The default settings perform well and are used throughout the experiments.
- Hyper-parameter effects: On CIFAR10 with 250 labels, CTAugment’s exponential weight decay ρ does not significantly affect results.The comparison uses standard ReMixMatch settings.
- Hyper-parameter effects: Depth and threshold have significant effects on CTAugment results.These effects are evaluated alongside exponential weight decay under the standard ReMixMatch configuration.
- Default settings: The default CTAugment settings perform well and are used for all experiments.The defaults are highlighted in the hyper-parameter table and were previously shown to be robust across many datasets.