PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Data Augmentation for Consistency Training

Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, Quoc V. Le

arXiv:1904.12848v6cs.LGcs.AIcs.CLcs.CVstat.ML

TL;DR

Semi-supervised consistency training depends on how unlabeled examples are noised, but the role of noise quality has received limited attention. UDA replaces simple noise with advanced, task-appropriate augmentations and reports improvements across language and vision tasks, including low-label and high-data settings.

  • Problem

    Consistency training uses unlabeled data but typically relies on simple noise, leaving the role of higher-quality noising operations insufficiently addressed.

  • Method

    UDA replaces traditional noise injection with advanced supervised-learning augmentations, including back-translation for text and task-appropriate vision augmentations.

  • Results

    UDA improves performance across six language and three vision tasks, including 5.43 error on CIFAR-10 with 250 labeled examples and gains on ImageNet.

  • Takeaways & Limitations

    Advanced data augmentation can serve as a superior noise source for consistency-based semi-supervised learning and can combine with BERT and high-data training.

  • Takeaways & Limitations

    On five-category sentiment classification, the task remains more difficult than binary classification, suggesting room for further improvement.

Abstract

from arXiv · show

Semi-supervised learning lately has shown much promise in improving deep learning models when labeled data is scarce. Common among recent approaches is the use of consistency training on a large amount of unlabeled data to constrain model predictions to be invariant to input noise. In this work, we present a new perspective on how to effectively noise unlabeled examples and argue that the quality of noising, specifically those produced by advanced data augmentation methods, plays a crucial role in semi-supervised learning. By substituting simple noising operations with advanced data augmentation methods such as RandAugment and back-translation, our method brings substantial improvements across six language and three vision tasks under the same consistency training framework. On the IMDb text classification dataset, with only 20 labeled examples, our method achieves an error rate of 4.20, outperforming the state-of-the-art model trained on 25,000 labeled examples. On a standard semi-supervised learning benchmark, CIFAR-10, our method outperforms all previous approaches and achieves an error rate of 5.43 with only 250 examples. Our method also combines well with transfer learning, e.g., when finetuning from BERT, and yields improvements in high-data regime, such as ImageNet, whether when there is only 10% labeled data or when a full labeled set with 1.3M extra unlabeled examples is used. Code is available at https://github.com/google-research/uda.

1 Introduction

UDA argues that the quality of noise in consistency training is central to semi-supervised learning, replacing simple noise with advanced data augmentation. Across language and vision tasks, this produces strong results with limited labeled data and remains useful in high-data settings.

  • Motivation: Consistency training regularizes predictions to remain invariant to small noise in inputs or hidden states, helping semi-supervised learning exploit unlabeled data.Typical noise sources include additive Gaussian, dropout, and adversarial noise.
  • Contribution: UDA substitutes traditional noise injection with advanced data augmentation methods that perform well in supervised learning.The paper reports a strong correlation between augmentation performance in supervised learning and consistency training.
  • Results: UDA achieves significant improvements across six text classification tasks and outperforms existing semi-supervised methods on vision benchmarks.The reported vision benchmark error rates are 5.43 on CIFAR-10 and 2.72 on SVHN with 250 labeled examples.
  • Results: On IMDb, UDA with 20 labeled examples outperforms a state-of-the-art model trained on 1250x more labeled data.The introduction states this comparison without reporting the corresponding IMDb error rate.
  • Results: On ImageNet, UDA improves top-1 accuracy from 58.84 to 68.78 with 10% of the labeled set and from 78.43 to 79.05 with full labels plus 1.3M unlabeled examples.These results indicate benefits in both lower-data and high-data regimes.

2 Unsupervised Data Augmentation (UDA)

UDA extends consistency training by using task-appropriate, high-quality augmentations to create valid and diverse noise for unlabeled examples. Its objective combines supervised learning with an unlabeled consistency loss, using methods such as RandAugment and back-translation.

  • Background: Data augmentation creates novel realistic-looking examples while preserving the original example’s ground-truth label.The augmentation distribution q(x̂ | x) is valid when sampled examples share the original label.
  • Core idea: UDA studies how the form and quality of input noise influence consistency-training performance, rather than treating noise injection as interchangeable.Prior approaches commonly use Gaussian noise or simple input augmentations.
  • Training objective: UDA combines supervised cross entropy with an unlabeled consistency loss weighted by λ, comparing predictions on original and augmented examples.The two losses are computed on labeled and unlabeled minibatches and then summed.
  • Why advanced augmentation: Advanced augmentations provide valid, diverse noise and task-targeted inductive biases beyond the local changes produced by simple Gaussian noise.The paper links these properties to realistic examples, larger input modifications, and augmentation choices suited to different tasks.
  • Vision augmentation: For vision, UDA uses RandAugment, which uniformly samples image transformations without search or labeled data.The method is inspired by AutoAugment but removes its search procedure.
  • Text augmentation: For text classification, back-translation translates a sentence from language A to B and back to A to generate diverse paraphrases while preserving semantics.The paper uses back-translation to paraphrase training data for text classification tasks.
  • Text augmentation: UDA also replaces low-TF-IDF words while retaining high-TF-IDF words to preserve informative keywords for topic classification.This method addresses the limited control over retained words in back-translation.

3 Theoretical Analysis

UDA’s theoretical analysis models augmentation as a graph whose connectivity determines how many labeled examples are needed for label propagation and target error rates.

  • Graph construction: Under UDA, augmentation defines a graph whose nodes are samples and whose edges connect samples with positive augmentation probability.The analysis assumes in-domain, label-preserving, and reversible augmentation.
  • Label propagation: A single labeled example in each connected component can propagate its label throughout that component through unsupervised augmentation.Supervised augmentation reaches only directly connected neighbors, whereas unsupervised augmentation traverses the entire sub-graph.
  • Augmentation quality: Better augmentation reduces the number of components by increasing graph connectivity and giving each node more neighbors.The number of components lower-bounds the labeled examples required to learn a perfect classifier.
  • Empirical illustration: Figure 3 compares prediction outcomes across settings using labeled, indeterminate unlabeled, and correctly inferred unlabeled nodes, with accuracy shown for each setting.The figure’s graph-based comparison illustrates the effect of different algorithms and augmentation settings.
  • Formal result: O(k/ϵ) labeled examples can guarantee an error rate of O(ϵ), where k is the number of graph components.The theorem links the required labeled-data quantity directly to the graph’s component count.

4 Experiments

UDA is evaluated across language and vision benchmarks, label regimes, architectures, and data scales. Across these settings, it generally improves over semi-supervised and supervised baselines, while revealing remaining difficulty for five-category sentiment classification.

  • UDA is evaluated on six text-classification datasets, CIFAR-10, SVHN, and ImageNet, including scalability tests with labeled and unlabeled data.
  • 4.1 Correlation between Supervised and Semi-supervised Performances: UDA tests whether supervised augmentation effectiveness transfers to semi-supervised learning by comparing augmentation methods on Yelp-5 and CIFAR-10.
  • 4.1 Correlation between Supervised and Semi-supervised Performances: Stronger augmentations correlate with stronger semi-supervised results on both Yelp-5 and CIFAR-10.
  • 4.2 Algorithm Comparison on Vision Semi-supervised Learning Benchmarks: UDA consistently outperforms VAT and MixMatch across different labeled-data sizes on CIFAR-10 and SVHN.
  • 4.2 Algorithm Comparison on Vision Semi-supervised Learning Benchmarks: With 4,000 labeled examples, UDA matches fully supervised reference performance on CIFAR-10 and SVHN using substantially larger labeled sets.
  • 4.3 Evaluation on Text Classification Datasets: On text classification, UDA with 20 labeled examples outperforms the fully supervised IMDb state-of-the-art model, while BERT initialization reduces IMDb error from 6.50 to 4.20.
  • 4.4 Scalability Test on the ImageNet Dataset: UDA significantly improves over supervised baselines on ImageNet with either 10% labeled data or all labeled data plus 1.3M filtered unlabeled images.

5 Related Work

Related consistency-training methods differ in how they define noise and where they enforce consistency. UDA instead emphasizes state-of-the-art augmentation rather than weak augmentation or mixup layered on simple transformations.

  • ICT and MixMatch combine mixup with simple augmentations such as flipping and cropping, whereas UDA emphasizes state-of-the-art data augmentations.
  • Existing consistency-training methods use Gaussian, dropout, adversarial, masking, or parameter-based perturbations to enforce consistency.

6 Conclusion

UDA connects data augmentation with semi-supervised learning by using advanced supervised augmentations as realistic consistency-training noise. It improves vision results, complements representation learning in text, and approaches fully supervised performance with far more labels.

  • UDA uses state-of-the-art supervised data augmentation to generate diverse and realistic noise, then enforces prediction consistency under that noise.
  • In text, UDA combines with representation learning such as BERT, while in vision it outperforms prior work and nearly matches fully supervised models trained on ten times more labeled data.

Broader Impact

The paper argues that limited labeled data can still support strong performance, while introducing techniques that regulate training signals as training progresses. It describes schedule choices for this regulation.

  • Broader Impact: Limited-budget groups may benefit from strong performance with limited labeled data on standard benchmarks.The paper states that its method does not leverage data biases and includes IMDb, CIFAR-10, SVHN, and ImageNet.
  • A.1 Training Signal Annealing for Low-data Regime: Training Signal Annealing gradually releases labeled-example training signals to address overfitting when unlabeled data greatly exceeds labeled data.Examples whose predicted probability for the correct category exceeds the threshold are removed from the loss.
  • A.1 Training Signal Annealing for Low-data Regime: The threshold ηt increases from 1/K to 1, acting as a ceiling against over-training on easy labeled examples.
  • A.1 Training Signal Annealing for Low-data Regime: The exp-schedule is suited to settings prone to overfitting because it releases most supervised signal near the end of training.The paper contrasts this with settings where overfitting is less likely and more gradual release is appropriate.

A.2 Extended Augmentation Strategies for Different Tasks

The augmentation strategies balance diversity against label-preserving validity and use task-specific mechanisms to avoid changing the ground-truth label. Several implementation choices remain untuned.

  • Trade-off Between Diversity and Validity for Data Augmentation: Text augmentation tunes sampling temperature to balance diverse paraphrases against validity, with temperatures 0.7, 0.8, or 0.9 performing best.Temperature 0 yields identical greedy paraphrases, whereas temperature 1 produces very diverse but barely readable paraphrases.
  • RandAugment Details: RandAugment samples transformations, magnitudes, and fixed probabilities for each operation from a defined set of 15 transformations.Each operation uses probability 0.5 and samples a magnitude in [1, 10).
  • RandAugment Details: The implementation did not tune RandAugment’s magnitude range or probability, so tuning these hyperparameters might improve accuracy.
  • TF-IDF based word replacing Details: TF-IDF-based word replacement retains keywords while replacing less informative words with alternatives.Replacement probability is designed to decrease as a word’s TF-IDF score increases.
  • TF-IDF based word replacing Details: Replacement words are sampled using corpus-level scores that reduce the probability of selecting keywords, helping preserve sentence labels.

B.1 Ablation Studies

Ablations show that UDA benefits from abundant unlabeled data, diverse RandAugment transformations, and Training Signal Annealing, especially when labeled data are scarce.

  • Ablation Studies for Unlabeled Data Size: Reducing unlabeled examples worsens performance more than reducing labeled examples by the same ratio.With the same labeled-example count, abundant unsupervised data is reported as more important than additional labeled data.
  • Ablations Studies on RandAugment: UDA’s CIFAR-10 error rate decreases as the number of possible RandAugment transformations increases.The figure interprets this trend as evidence for the importance of a rich augmentation set.
  • Ablation Studies for TSA: The exp-schedule performs best on Yelp-5 by postponing release of the supervised training signal until the end of training.

B.2 More Results on CIFAR-10, SVHN and Text Classification Datasets

Additional results show UDA across CIFAR-10, SVHN, and text classification settings, including comparisons with fully supervised baselines and performance across label-set sizes.

  • CIFAR-10: 4.23 and 5.36 error rates are reported for fully supervised CIFAR-10 training with 50,000 examples, with and without RandAugment.
  • CIFAR-10: The authors reimplemented MixMatch and obtained comparable but slightly better results than the original reported comparison, making the UDA comparison more competitive.For example, the reimplementation reports 7.00 ± 0.59 and 7.39 ± 0.11 error rates with 4,000 and 2,000 examples.
  • SVHN: 2.28 and 2.84 error rates are reported for fully supervised SVHN training with 73,257 examples, with and without RandAugment.
  • Text Classification: UDA produces consistent improvements across all labeled-data sizes on IMDb and Yelp-2.The accompanying results examine accuracy as the number of labeled examples varies.

C Proof for Theoretical Analysis

The theoretical analysis defines the probability of failing to infer a test label under UDA and derives an error guarantee from labeled examples. The surrounding discussion situates UDA among consistency training, data augmentation, and related semi-supervised approaches.

  • Theoretical analysis: Theorem 1 defines Pr(A) as the probability that UDA cannot infer a new test example’s label from m labeled examples.The theorem provides the expression for this failure probability.
  • Theoretical analysis: O(k/ϵ) labeled examples guarantee an error rate of O(ϵ) under UDA.
  • Proof: The proof bounds the failure probability through a convex optimization problem, its Lagrangian dual, and KKT conditions yielding equal probabilities across indices.The derivation introduces an optimization over the probabilities and uses stationarity to obtain Pi = Pj for i ≠ j.
  • Related work: UDA relates to semi-supervised methods using graph propagation, latent-variable models, teacher ensembles, and data augmentation as explicit regularization.The related-work discussion also connects consistency training with representation learning and robustness.

E.2 Semi-supervised learning benchmarks CIFAR-10 and SVHN

The CIFAR-10 and SVHN benchmark experiments use distinct architectures, large unlabeled batches, and augmentation-focused training configurations. The setup also includes separate ImageNet settings for limited and full labeled data.

  • CIFAR-10 and SVHN: Wide-ResNet-28-2 training uses 500K steps, batch sizes of 64 labeled and 448 unlabeled examples, and a confidence threshold β of 0.8.The configuration also applies EMA with decay 0.9999 and temperature τ = 0.4.
  • CIFAR-10 and SVHN: For CIFAR-10, 100 augmented examples are generated for each unlabeled example before training.The passage notes that online augmentation is always better or as good as using dumped augmented examples.
  • CIFAR-10 and SVHN: UDA trains Shake-Shake for 300k steps and PyramidNet+ShakeDrop for 700k steps with larger unsupervised than supervised batch sizes.Both configurations use a learning rate of 0.03 and cosine learning-rate decay.
  • ImageNet settings: The 10% labeled ImageNet setting uses batch sizes of 512 labeled and 15,360 unlabeled examples, λ = 20, and confidence masking.
  • ImageNet settings: The full ImageNet setting uses batch sizes of 8,192 labeled and 16,384 unlabeled examples, λ = 1, and entropy minimization.
Loading 1904.12848v6…