PaperlayerView exact revision on arXiv

Source-linked AI summary

FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence

Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, Colin Raffel

arXiv:2001.07685v2cs.LGcs.CVstat.ML

TL;DR

Labeling data for deep learning is costly, motivating methods that exploit unlabeled data. FixMatch combines confidence-filtered pseudo-labeling with weak-to-strong augmentation consistency, achieving state-of-the-art SSL performance, including 94.93% accuracy on CIFAR-10 with 250 labels.

  • Problem

    Labeling large datasets can require substantial human labor, motivating semi-supervised learning methods that leverage plentiful unlabeled data.

  • Method

    FixMatch combines pseudo-labeling and consistency regularization by using confident predictions on weakly augmented images as targets for strongly augmented versions.

  • Results

    94.93% accuracy on CIFAR-10 with 250 labeled examples demonstrates state-of-the-art performance across standard SSL benchmarks.

  • Takeaways & Limitations

    FixMatch shows that a simple SSL objective can achieve strong performance with very few labels, including one label per class.

  • Takeaways & Limitations

    Performance with extremely few labels is sensitive to random seeds and exhibits substantially higher variance across data folds.

Abstract

from arXiv · show

Semi-supervised learning (SSL) provides an effective means of leveraging unlabeled data to improve a model's performance. In this paper, we demonstrate the power of a simple combination of two common SSL methods: consistency regularization and pseudo-labeling. Our algorithm, FixMatch, first generates pseudo-labels using the model's predictions on weakly-augmented unlabeled images. For a given image, the pseudo-label is only retained if the model produces a high-confidence prediction. The model is then trained to predict the pseudo-label when fed a strongly-augmented version of the same image. Despite its simplicity, we show that FixMatch achieves state-of-the-art performance across a variety of standard semi-supervised learning benchmarks, including 94.93% accuracy on CIFAR-10 with 250 labels and 88.61% accuracy with 40 -- just 4 labels per class. Since FixMatch bears many similarities to existing SSL methods that achieve worse performance, we carry out an extensive ablation study to tease apart the experimental factors that are most important to FixMatch's success. We make our code available at https://github.com/google-research/fixmatch.

1 Introduction

FixMatch combines consistency regularization with pseudo-labeling in a simple SSL method that uses confident predictions on weakly augmented images to supervise strongly augmented versions. It achieves state-of-the-art benchmark performance, including 94.93% accuracy on CIFAR-10 with 250 labeled examples and 88.61% with 4 labels per class.

  • Motivation: Semi-supervised learning leverages unlabeled data to reduce the labeling requirements of deep-network training at low human-labor cost.Deep networks benefit from larger datasets, but supervised learning requires labeled data; SSL addresses this constraint by exploiting unlabeled examples.
  • Method: FixMatch combines consistency regularization and pseudo-labeling, retaining confident predictions from weakly augmented images as targets for strongly augmented versions.The weak prediction becomes a one-hot pseudo-label only when a class probability exceeds a threshold, and training uses cross-entropy on the strong augmentation.
  • Results: FixMatch achieves 94.93% accuracy on CIFAR-10 with 250 labeled examples, compared with the previous state-of-the-art of 93.73%.The comparison uses the standard experimental setting from [36].
  • Results: 88.61% accuracy is achieved on CIFAR-10 with only 4 labels per class, demonstrating performance in the extremely-scarce-labels regime.The paper also reports state-of-the-art performance across the most commonly studied SSL benchmarks.

2 FixMatch

FixMatch combines consistency regularization with confidence-thresholded pseudo-labeling, generating labels from weakly augmented images and enforcing them on strongly augmented versions. Its objective combines supervised and unlabeled cross-entropy losses, while confidence filtering can create a natural curriculum during training.

  • Core method: FixMatch combines consistency regularization and pseudo-labeling, using separate weak and strong augmentations as its central methodological innovation.The model produces pseudo-labels from weakly augmented images and applies consistency training to strongly augmented images.
  • Core method: FixMatch computes a pseudo-label from the predicted class distribution on a weakly augmented unlabeled image, then applies cross-entropy to the strongly augmented image.The pseudo-label is the arg max of the weak-view prediction, and the loss is enforced against the strong-view output.
  • Confidence filtering: FixMatch retains a pseudo-label only when its maximum predicted class probability exceeds threshold τ.The unsupervised loss is weighted by λu relative to the supervised loss, with total loss ℓs + λuℓu.
  • Confidence filtering: As training progresses, predictions become more confident and more examples satisfy max(qb) > τ, yielding a natural curriculum without additional machinery.Early in training, max(qb) is typically below τ; later, threshold-qualified pseudo-labels become more frequent.
  • Augmentation: FixMatch uses standard flip-and-shift weak augmentation and strong AutoAugment-based transformations followed by Cutout.Images are horizontally flipped with 50% probability except on SVHN and translated by up to 12.5% vertically and horizontally.

3 Related work

FixMatch builds on two established SSL lines: self-training with confidence-filtered pseudo-labels and consistency regularization across perturbations. It most closely resembles UDA and ReMixMatch, while differing in its use of pseudo-labeling.

  • Self-training and pseudo-labeling: Self-training uses model predictions to create artificial labels for unlabeled data and has been applied across domains including NLP, object detection, image classification, and domain adaptation.Pseudo-labeling is a specific variant that converts predictions into hard labels, often retaining examples only above a confidence threshold.
  • Consistency regularization: Consistency regularization trains models to produce stable predictions under perturbations such as augmentation, stochastic regularization, or adversarial changes.Earlier methods also used exponential moving averages or previous checkpoints when producing artificial labels.
  • Closest related methods: FixMatch most closely resembles UDA and ReMixMatch, which use weak augmentations to generate artificial labels and strong augmentations to enforce consistency.Neither method uses pseudo-labeling; instead, both sharpen artificial labels to encourage high-confidence predictions.
  • Algorithm comparison: FixMatch combines two existing techniques, motivating a comparison of SSL algorithms by their label-generating augmentation, model prediction, and artificial-label post-processing.The paper presents this comparison in Table 1 and provides a more thorough discussion of the constituent approaches later.

4 Experiments

FixMatch is evaluated across standard SSL benchmarks, label regimes, augmentation strategies, and a larger-scale ImageNet setting under controlled training protocols. It achieves state-of-the-art performance broadly, while extreme label scarcity exposes substantial sensitivity to labeled-example quality and dataset variance.

  • Benchmark comparisons: FixMatch substantially outperforms MixMatch, ReMixMatch, and UDA with 40 and 250 labels while using a simpler method.Baselines were reimplemented in the same codebase, using the same architecture, optimizer, learning-rate schedule, preprocessing, and training protocol.
  • Benchmark comparisons: FixMatch achieves state-of-the-art results on all evaluated datasets except CIFAR-100, where ReMixMatch performs slightly better.Distribution Alignment is identified as the most important ReMixMatch component, and combining it with FixMatch reaches a 40.14% error rate.
  • Augmentation and variance: 3.35% variance across five folds occurs for CIFAR-10 with 4 labels per class, versus 0.33% with 25 labels per class.FixMatch performance with CTAugment and RandAugment is generally similar, except in the particularly high-variance 4-label setting.
  • ImageNet: 28.54 ± 0.52% top-1 error is achieved on ImageNet with 10% labeled training data, 2.68% better than UDA.The corresponding top-5 error rate is 10.87 ± 0.28%, using ResNet-50 and RandAugment.
  • Extreme label scarcity: 64.28% median test accuracy is reached across four randomly selected one-example-per-class datasets, with results ranging from 48.58% to 85.32%.Four models trained on the first dataset reach between 61% and 67% accuracy, while the second dataset yields between 68% and 75%.
  • Extreme label scarcity: 78% median accuracy is achieved using the most prototypical examples, compared with 65% for middle-distribution examples, while outlier-only training fails to converge and reaches 10% accuracy.These results support the hypothesis that labeled-example quality drives variability in the one-example-per-class setting.

5 Ablation Study

The ablation study examines why FixMatch performs well using a single 250-label CIFAR-10 split with CTAugment, where default FixMatch achieves 4.84% error. Results identify confidence thresholding, strong augmentation, and the correct weak–strong augmentation paths as important factors.

  • Experimental setup: Default FixMatch achieves 4.84% error on the 250-label CIFAR-10 split used for the ablation study.The experiments focus on this split and report results using CTAugment.
  • Confidence threshold: A confidence threshold of 0.95 gives the lowest error rate, while small thresholds reduce accuracy by more than 1.5%.Increasing the threshold to 0.97 or 0.99 does not substantially hurt performance; the threshold trades pseudo-label quality against quantity.
  • Strong augmentation: Both Cutout and CTAugment are required for best performance, since removing either causes a significant increase in error rate.Cutout is applied after strong augmentation in both RandAugment and CTAugment.
  • Weak–strong augmentation paths: Replacing weak augmentation with strong augmentation for label guessing causes the model to diverge early in training.The ablation varies weak and strong augmentations across pseudo-label generation and prediction paths.
  • Weak–strong augmentation paths: Using weak rather than strong augmentation for the training prediction peaks at 45% accuracy before collapsing to 12%.The result is unstable and progressively collapses, suggesting that strong data augmentation is important.

6 Conclusion

FixMatch is presented as a simple SSL algorithm achieving state-of-the-art results across many datasets, while its analysis identifies weight decay and optimizer choice as important design factors. The authors argue that such simple, effective SSL methods can support deployment where labels are expensive or difficult to obtain.

  • Conclusion: FixMatch simplifies SSL while achieving state-of-the-art results across many datasets.The authors position it as a way to bridge low-label semi-supervised learning with few-shot learning or clustering.
  • Conclusion: Weight decay and optimizer choice are especially important design choices in FixMatch.Their importance can prevent direct comparisons of the same technique across implementations, even when model architecture is controlled.
  • Conclusion: Simple, performant SSL algorithms could enable machine learning deployment where labels are expensive or difficult to obtain.The authors connect this potential to increasingly many practical domains.

Broader Impact

FixMatch may broaden access to machine learning through its simplicity and strong accuracy with few labels. This democratization could also make machine-learning research easier for both beneficial and harmful actors to apply.

  • Access: FixMatch’s simplicity makes machine learning available to a wider audience.The paper identifies simplicity as one way FixMatch helps democratize machine learning.
  • Access: High accuracy with only a few labels enables applications in domains where machine learning was previously infeasible.The passage cites low-label accuracy as a second mechanism for democratization.
  • Risks: The democratization of machine-learning research may make it easier for both good and bad actors to apply.The authors express hope that this increased capability will be used for good.

Funding Disclosure … B.2 Trade-off between the Quality and the Quantity of Pseudo-Labels with Confidence

FixMatch combines confidence-thresholded pseudo-labeling with weak-to-strong augmentation, while experiments show that threshold choice trades pseudo-label quantity against quality and confirmation bias. The work was funded solely by Google and reports implementation details, shared hyperparameters, and ablations on CIFAR-10.

  • Funding Disclosure: Google is the sole source of funding for this work.
  • A Algorithm: FixMatch computes labeled and unlabeled cross-entropy losses, using weakly augmented predictions to form confidence-filtered pseudo-labels for strongly augmented inputs.The total objective is the labeled loss plus λu times the unlabeled loss.
  • B.1 Hyperparameters: The study uses almost identical FixMatch hyperparameters across CIFAR-10, CIFAR-100, SVHN, and STL-10, with architecture and weight-decay adjustments for dataset size and input dimensions.CIFAR-100 uses WRN-28-8 with doubled weight decay, while STL-10 uses WRN-37-2.
  • B Comprehensive Experimental Results: The experiments evaluate thresholding using test accuracy, pseudo-label impurity, and mask rate on a single 250-label CIFAR-10 split.Impurity measures the error rate among unlabeled examples above the threshold, while mask rate measures examples masked out.
  • B.2 Trade-off between the Quality and the Quantity of Pseudo-Labels with Confidence: Small confidence thresholds include most unlabeled examples in the loss, but incorrect pseudo-labels introduce noise and significantly impede learning through confirmation bias.
  • B.2 Trade-off between the Quality and the Quantity of Pseudo-Labels with Confidence: High confidence thresholds admit fewer, ostensibly higher-quality unlabeled examples and reduce confirmation bias under strong augmentation, producing lower test-set error rates.The result reflects a trade-off between pseudo-label quality and quantity.

B.3 Ablation Study on Optimizer … B.8 Comparison to Supervised Baselines

FixMatch performance depends substantially on optimization choices, unlabeled-data usage, and weight decay, while carefully selected labeled examples can yield over 80% accuracy. Supervised-only comparisons further isolate the contribution of unlabeled data.

  • B.3 Ablation Study on Optimizer: Momentum β affects performance: overly large β prevents convergence, whereas small β values remain effective.The study reports that optimizer and hyperparameter choices can strongly affect semi-supervised learning performance.
  • B.3 Ablation Study on Optimizer: 0.53%: Adam’s best error rate is only 0.53% larger than momentum SGD’s, but its error increases by more than 8% at learning rate 0.002.Adam was more sensitive to learning-rate changes than momentum SGD.
  • B.4 Ablation Study on Learning Rate Schedule: 0.86%: using no learning-rate decay degrades accuracy by 0.86%, while linear decay performs nearly as well as cosine decay.The proper decay rate remains important for cosine learning-rate decay.
  • B.5 Ratio of Labeled to Unlabeled Data in Minibatch: Increasing the unlabeled-data ratio μ significantly decreases error rates, and scaling learning rate η with batch size is especially effective when μ is small.The observed benefit of more unlabeled data is consistent with UDA; linear learning-rate scaling follows large-batch supervised training practice.
  • B.6 Weight Decay: Ten percentage points or more: choosing weight decay one order of magnitude above or below its optimum can cost at least ten percentage points in low-label regimes.A value of 0.0005 is reported as a good default for WRN-28-2 across datasets.
  • B.7 Labeled Data for Barely Supervised Learning: Ordering labeled images from most to least prototypical reveals over 80% accuracy when FixMatch trains on the best examples.Each complete labeled dataset contains 10 images from 10 CIFAR-10 classes, and two models are averaged per dataset.
  • B.8 Comparison to Supervised Baselines: Models trained only on labeled data with strong augmentations provide supervised baselines for assessing FixMatch’s effectiveness from using unlabeled data.The comparisons are presented in tables 9 and 10.

C Implementation Details for Section 4.3 · D Extensions of FixMatch · D.1 Augmentation Anchoring and Distribution Alignment

ImageNet FixMatch uses distributed pre-activation ResNet50 training with specified optimization and batching choices. Extensions add Augmentation Anchoring and Distribution Alignment, reducing error on CIFAR-10 and CIFAR-100 under limited-label settings.

  • C Implementation Details for Section 4.3: ImageNet experiments use a distributed pre-activation ResNet50 trained on a 32-core TPU across five random labeled-data folds.The batch contains 1024 labeled and 5120 unlabeled examples per step.
  • C Implementation Details for Section 4.3: Training runs for 300 epochs of unlabeled examples, with five-epoch warmup to learning rate 0.4 and decays at epochs 60, 120, 160, and 200.The optimizer is Nesterov Momentum with momentum 0.9, EMA decay is 0.999, and λu = 10.
  • C Implementation Details for Section 4.3: One unlabeled epoch covers all 1.2 million ImageNet training examples and corresponds to 10 passes through the labeled set in the 10% ImageNet task.
  • D.1 Augmentation Anchoring and Distribution Alignment: FixMatch incorporates Augmentation Anchoring by applying M strong augmentations to each unlabeled example for consistency regularization.The strong augmentation is stochastic and produces M distinct strongly augmented examples.
  • D.1 Augmentation Anchoring and Distribution Alignment: Distribution Alignment aligns weakly augmented predictions with the labeled-data marginal class distribution and the unlabeled-data running average of model predictions.The technique encourages predictions to match the labeled set’s class distribution.
  • D.1 Augmentation Anchoring and Distribution Alignment: 4.81% error results from adding Augmentation Anchoring to FixMatch (CTA), down from 5.07% on CIFAR-10 with 250 labels, using M = 4 and µ = 4.The result is averaged over five different folds.
  • D.1 Augmentation Anchoring and Distribution Alignment: 9.47% error results from adding Distribution Alignment to FixMatch (CTA), down from 11.38% on CIFAR-10 with 40 labels.On CIFAR-100 with 400 labels, error falls from 49.95% to 40.14%, below ReMixMatch’s 44.28%.

D.2 Datatype-Agnostic Data Augmentation

This section examines whether FixMatch can use datatype-agnostic augmentation instead of vision-specific strong augmentation. It evaluates MixUp and Virtual Adversarial Training on CIFAR-10 under a 250-label protocol.

  • Motivation: Strong augmentation is central to FixMatch, motivating augmentation strategies that extend beyond vision-specific methods.The paper notes domain-specific alternatives such as back-translation for text and SpecAugment for speech.
  • Augmentation schemes: MixUp and Virtual Adversarial Training replace RandAugment or CTAugment as datatype-agnostic augmentation schemes for image classification.For MixUp, the experiment mixes random input pairs without mixing labels, using α = 9; VAT uses τ = 0.5.
  • Evaluation: The methods are evaluated on CIFAR-10 with 250 labeled examples, reporting mean and standard deviation over 5 folds.The accompanying table reports error rates, and all models use the same codebase.

E List of Data Transformations

The section describes RandAugment’s fixed- or randomly sampled-magnitude strategy and CTAugment’s online learning of transformation magnitudes. It also documents the transformation operations and parameter discretization used by both strategies.

  • RandAugment: RandAugment randomly selects transformations per mini-batch sample and can use a randomly sampled magnitude from a predefined range at each training step.The original method instead uses one fixed global magnitude controlling all distortion severities, optimized as a validation-set hyperparameter.
  • CTAugment: CTAugment learns transformation magnitudes online by dividing their possible values into bins and assigning each bin an initially equal weight.Examples are augmented with pipelines of two transformations sampled uniformly at random.
  • Transformation lists: Both strategies use the same sets of image transformations, whose complete operation lists are provided in Tables 12 and 13.Table 13 discretizes listed parameters into 17 equal bins, except Rescale’s M parameter, which has six interpolation options.
Loading 2001.07685v2…