PaperlayerView exact revision on arXiv

Source-linked AI summary

Unlabeled Data Improves Adversarial Robustness

Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, John C. Duchi

arXiv:1905.13736v4stat.MLcs.CVcs.LG

TL;DR

Adversarially robust classification can suffer a sample-complexity gap relative to standard classification, motivating the use of unlabeled data. The paper analyzes self-training theoretically and robust self-training empirically, finding improved heuristic and certified robustness on CIFAR-10 and SVHN. These results support semisupervised learning as a way to obtain robustness with fewer labels, subject to limitations from irrelevant data and a loss-formulation discrepancy.

  • Problem

    The paper addresses whether unlabeled data can reduce the sample burden for adversarially robust classification.

  • Method

    It proves in a Gaussian model that self-training uses unlabeled inputs to bridge the robust-versus-standard sample complexity gap and applies robust self-training empirically.

  • Results

    Robust self-training improves heuristic and certified robustness on CIFAR-10 and raises SVHN robust accuracy by 4–10% using unlabeled data.

  • Takeaways & Limitations

    The findings support semisupervised learning as a way to improve adversarial robustness using substantially less labeled data.

  • Takeaways & Limitations

    Irrelevant unlabeled data can significantly hinder self-training, and the robust-loss regularization convention differs from the cited formulation in one implementation.

Abstract

from arXiv · show

We demonstrate, theoretically and empirically, that adversarial robustness can significantly benefit from semisupervised learning. Theoretically, we revisit the simple Gaussian model of Schmidt et al. that shows a sample complexity gap between standard and robust classification. We prove that unlabeled data bridges this gap: a simple semisupervised learning procedure (self-training) achieves high robust accuracy using the same number of labels required for achieving high standard accuracy. Empirically, we augment CIFAR-10 with 500K unlabeled images sourced from 80 Million Tiny Images and use robust self-training to outperform state-of-the-art robust accuracies by over 5 points in (i) $\ell_\infty$ robustness against several strong attacks via adversarial training and (ii) certified $\ell_2$ and $\ell_\infty$ robustness via randomized smoothing. On SVHN, adding the dataset's own extra training set with the labels removed provides gains of 4 to 10 points, within 1 point of the gain from using the extra labels.

1 Introduction

The paper studies adversarial robustness through semisupervised learning, arguing that unlabeled inputs can support stability and self-training. Theory and experiments show robust self-training improves robustness across benchmarks and attack or certification settings.

  • Motivation: Sample complexity may explain why adversarial robustness has seen smaller gains than standard accuracy despite many proposed defenses.Prior work found that robust classification can require substantially more samples than standard classification in a simple model.
  • Motivation: Unlabeled inputs can support robustness because stability around naturally occurring inputs does not inherently require labels.Classifiers trained on labeled data can generate pseudo-labels for unlabeled inputs, motivating self-training.
  • Theoretical motivation: In the Gaussian model, self-training achieves high robust accuracy with n0 labels and O(n0 · ϵ^2p d/n0) unlabeled examples.The result places the increased sample requirement on unlabeled data rather than labels.
  • Approach: Robust self-training combines standard training for pseudo-label generation with robust training on the resulting pseudo-labeled data.The experiments use TRADES for heuristic ℓ∞ robustness and stability training with randomized smoothing for certified ℓ2 robustness.
  • Empirical results: 500K unlabeled CIFAR-10 images yielded 7% higher heuristic ℓ∞ robustness, 10% higher certified ℓ2 robustness than prior state-of-the-art, and matched certified ℓ∞ state-of-the-art.The corresponding standard accuracy for certified ℓ∞ robustness improved by over 16%.
  • Empirical results: SVHN robust accuracies increased by 4–10% after adding 531K unlabeled examples, while using true labels improved them by at most 1% further.This indicates that most of the benefit from the extra data came from inputs rather than labels.

2 Setup

The setup defines semisupervised classification from labeled examples and additional unlabeled inputs, then evaluates both standard and adversarially robust error. Self-training extends a supervised learner by generating pseudo-labels and retraining on combined data.

  • Semisupervised classification: Semisupervised classification maps inputs x ∈ X ⊆ R^d to labels y ∈ Y using labeled and unlabeled training examples.The learner is a classifier fθ parameterized by θ ∈ Θ.
  • Error metrics: Standard quality is measured by the classifier’s error probability.
  • Error metrics: Robust evaluation considers perturbations within an ℓp norm ball of radius ϵ around each input.The paper studies p = 2 and p = ∞ and defines corresponding robust error and certified accuracy.
  • Self-training: Self-training first fits an intermediate model on labeled data, uses it to assign pseudo-labels to unlabeled inputs, and then retrains on the combined dataset.The same supervised algorithm A is applied in both stages.

3 Theoretical results

The theoretical analysis revisits a high-dimensional Gaussian model with a gap between standard and robust sample complexity. It shows that self-training uses unlabeled data to achieve high robust accuracy with far fewer labels, while irrelevant data increases data requirements.

  • Theoretical setting: The Gaussian model is the paper’s formal setting for demonstrating an information-theoretic sample complexity gap between standard and robust classification.The labeled-data-only setting can achieve nontrivial standard accuracy while robust accuracy remains impossible with the same data scale.
  • Supervised learning: In the high-dimensional regime, robust learning requires roughly n0 · ϵ^2p d/n0 examples, versus n0 examples for nontrivial standard accuracy.The gap grows as d increases.
  • Semisupervised learning: Self-training generates pseudo-labels from an intermediate classifier trained on labeled data and applies the learning rule to the unlabeled examples.The final classifier aggregates labeled and pseudo-labeled information.
  • Semisupervised learning: With n ≥ n0 labeled examples and additional unlabeled data, the self-training classifier achieves high robust accuracy under the theorem’s high-dimensional condition.The theorem states the required regime as ϵ^2p d/n0 ≥ r̃.
  • Sample complexity comparison: Compared with fully supervised learning, self-training needs only a constant-factor increase in total inputs and roughly ϵ^2p d/n0 fewer labels.The analysis attributes convergence to the unlabeled-data average becoming less noisy as the unlabeled sample grows.
  • Irrelevant unlabeled data: When only a fraction α of unlabeled data is relevant, the required number of relevant examples grows by a factor of 1/α.Irrelevant data can significantly hinder self-training but does not prevent high robust accuracy for fixed α.

4 Semi-supervised learning of robust neural networks

Robust self-training adapts self-training to neural-network robustness by separating pseudo-label generation from robust final training. The paper instantiates this framework with supervised losses, adversarial training, and randomized smoothing.

  • Robust self-training: Robust self-training adapts supervised robust training methods to use additional unlabeled data.
  • Robust self-training: The procedure takes labeled and unlabeled data together with standard loss, robust loss, and an unlabeled-data weight.
  • Robust self-training: The intermediate network generates pseudo-labels for unlabeled inputs before the final robust-training stage.
  • Robust self-training: Standard training is used first for high-quality pseudo-labels, while robust training is used second for the final classifier.The two stages therefore have different goals.
  • Neural-network instantiation: Both stages use supervised learning with neural networks that output class-probability distributions.The classifier predicts the label with the highest conditional probability.
  • Robust objectives: The robust loss combines a stability-promoting regularization term with a parameter β that balances robustness and accuracy.The regularizer keeps predictions stable within the perturbation ball.
  • Robust objectives: Adversarial training approximates the robust-loss maximization for ℓ∞ perturbations using projected gradient methods.The paper also considers stability training with larger Gaussian noise for certified ℓ2 robustness and uses a smoothed classifier at test time.

5 Experiments

The experiments evaluate robust self-training on CIFAR-10 and SVHN using heuristic adversarial training and certified stability training. Across both benchmarks, unlabeled inputs provide substantial robustness gains, while pseudo-labels incur little loss relative to true labels.

  • CIFAR-10 certified defenses: 3–5%: RSTstab(50K+500K) improves certified CIFAR-10 ℓ2 accuracy over the fully supervised Baselinestab(50K).It also achieves a 10% improvement over single-model state-of-the-art certified ℓ∞ robustness and matches the cascade approach, while improving standard accuracy by 16%.
  • Alternatives and ablations: 3–6%: Robust self-training achieves higher accuracy than adversarial- and stability-flavored VAT adaptations.VAT provides only limited benefit over fully supervised robust training.
  • Alternatives and ablations: Most gains from SVHN extra data come from unlabeled inputs across adversarial and certified training settings.Figure 2 compares no extra data, unlabeled extra data, and labeled extra data under ℓ∞ attack and certified ℓ2 evaluation.

6 Related work

The paper relates robust self-training to semisupervised, domain-adaptation, robust-classification, and sample-complexity research. Concurrent studies also use unlabeled data for robustness, but this work emphasizes certified robustness, irrelevant-data sensitivity, and additional empirical analyses.

  • Semisupervised learning: Semisupervised learning commonly exploits consistency under perturbations or during training, whereas this paper studies self-training for adversarial robustness.The related literature has often focused on improving standard accuracy.
  • Self-training for domain adaptation: Domain-adaptation self-training targets unlabeled data from a different target distribution than the labeled source distribution.These approaches approximate target class-conditional distributions and learn matching feature transformations.
  • Robust classification: Prior robust-classification work includes heuristic adversarial training and certified methods, with certified approaches often trading off empirical robustness.Sample-complexity studies suggest statistical barriers to robust learning.
  • Semisupervised robustness: Concurrent work uses PCA, soft-label distributionally robust optimization, or robust self-training to leverage unlabeled data for adversarial robustness.The cited PCA and soft-label studies did not augment CIFAR-10 with new unlabeled data or improve state-of-the-art adversarial robustness.
  • Comparison with concurrent work: This paper complements concurrent work by studying certified robustness, irrelevant unlabeled data, data augmentation, and unlabeled-data quantity.Uesato et al. is identified as the closest related work, including Gaussian-model analysis and CIFAR-10 experiments with Tiny Images.

7 Conclusion

The paper concludes that unlabeled data closes a sample-complexity gap in a stylized model and that robust self-training consistently improves robustness on two image-classification benchmarks. It identifies broader theoretical and practical questions for future work.

  • Conclusion: Unlabeled data closes a sample-complexity gap in a stylized model, while robust self-training is consistently beneficial on two image-classification benchmarks.These are the paper’s stated theoretical and empirical conclusions.
  • Future directions: The authors ask whether sufficient unlabeled data universally resolves gaps between standard and adversarially robust learning.They also raise questions about better ways to leverage unlabeled data and alternative notions of robustness.
  • Reproducibility: The released code, data, and experiments support reproducibility through GitHub and CodaLab.The paper provides public links for both resources and experiments.

Supplementary Material

The supplementary material develops the Gaussian-model analysis of standard and robust error, proves concentration and sample-complexity results for self-training, and supplies the estimator’s two-stage construction. It also records assumptions and proof steps supporting the main theoretical claims.

  • Assumptions and extensions: The analysis allows class imbalance, with the upper bounds unchanged and the lower bound modified according to the smaller class proportion.This extension changes the lower-bound expression but not the upper-bound claims.
  • Error probabilities: The Gaussian model defines standard and robust error for linear classifiers under ℓ∞ perturbations, with the strongest perturbation shifting each coordinate by −ϵsign(yθ).The robust error is analyzed through the classifier’s margin after this coordinate-wise perturbation.
  • Model behavior: In the high-dimensional regime d/n0 ≫ 1, the optimal classifier has essentially perfect standard and robust accuracy, and enough unlabeled data lets self-training approach it.The required unlabeled-data scale is approximately d/n0, without prior knowledge of µ.
  • Semisupervised estimator: The self-training estimator uses labeled data to construct an intermediate classifier, pseudo-labels unlabeled examples, then applies the learning rule to those pseudo-labeled examples.The final estimator receives a high-probability performance bound analogous to the fully supervised estimator.
  • Theoretical guarantee: Theorem 2 gives a sample-complexity upper bound showing high robust accuracy with n ≥ n0 labeled examples and additional unlabeled data in the specified high-dimensional regime.Lemma 2 supplies the concentration result used for this bound.

A.5 Performance in the presence of irrelevant data

With irrelevant unlabeled data, the paper analyzes when self-training can still achieve robust accuracy and when insufficient relevant data makes robustness impossible. The results refine the sample-complexity boundary by separating relevant from irrelevant unlabeled examples.

  • Model with irrelevant data: The model includes an α-fraction of relevant unlabeled data, with the remaining examples treated as irrelevant.The irrelevant examples may contain no signal component, while related variants allow a weak independent signal.
  • Theoretical questions: The analysis asks whether sufficiently many unlabeled examples enable robustness and whether the resulting upper bound is tight.It specifically studies both achievability and failure with too few relevant unlabeled examples.
  • Achievability conditions: Self-training can achieve robust accuracy under conditions involving the dimension, perturbation magnitude, labeled-sample count, and fraction of relevant unlabeled data.The formal guarantee is stated through parameter conditions in Theorem 3 and Lemma 3.
  • Proof strategy: The proof establishes bounds on the final estimator’s norm and density before applying them to the robust error probability.Lemmas 3 and 4 provide the key intermediate controls used to prove Theorem 3.
  • Proof strategy: The analysis uses pseudo-label indicators, decomposes noise into components relative to the intermediate estimator, and derives high-probability norm bounds.These steps support the final classifier analysis and its robust-error guarantee.
  • Empirical context: Without unlabeled data, adversarial-training traces overfit: robust accuracy declines while standard accuracy continues improving; RST with 500K unlabeled images avoids this pattern.An early-stopped supervised model reaches 55.5% robust accuracy against PGTRADES, 54.1% against PGOurs, and 84.5% standard accuracy.

B.5 Comparison between stability and noise training

The paper compares stability training with noise training while varying architecture and hyperparameters, finding that stability training and larger capacity improve certified robustness. The unlabeled-data pipeline uses filtered 80M-TI images selected by a classifier, but the resulting set remains imperfectly relevant.

  • Experimental scope: The experiments comparing supervised randomized-smoothing methods use only CIFAR-10’s labeled training set and no unlabeled data.This isolates architecture, hyperparameter, and objective effects from the contribution of additional unlabeled data.
  • Training comparison: Using the authors’ training configuration increases certified accuracy by 2–3% across most perturbation radii relative to the model of [9].This comparison changes the training configuration while retaining the ResNet-110 architecture and noise-training objective.
  • Architecture and radius trade-offs: Using Wide ResNet 28-10 further improves performance by roughly 2%, while noise training performs better below radius 0.25 and worse at larger radii.The paper attributes baseline differences a priori to architecture, training configuration, and training objective, then tests these factors separately.
  • Unlabeled-data sourcing: The selected unlabeled set contains 50,000 images per CIFAR-10 class, is approximately class balanced, and is 10 times the original training-set size.Images are selected from 80M-TI by highest classifier confidence after near-duplicate removal.
  • Unlabeled-data limitations: The sourced unlabeled data are not entirely relevant: many images selected as frogs or deer are not actually frogs or deer.The authors retain this imperfection to simulate irrelevant data and avoid using more advanced selection methods for a fair comparison.

C.1 Alternative semisupervised training method

The paper compares robust self-training with alternative semisupervised objectives and data augmentation, finding that robust self-training generally provides the strongest robustness gains.

  • Increasing the entropy weight steadily degrades accuracy against PGOurs in the alternative adversarial-training approach.
  • Robust self-training significantly outperforms the alternative approach in both heuristic adversarial and certified stability training.The authors suggest that locking in accurate pseudo-labels may provide a more direct training signal.
  • VAT-like consistency regularization produces only minor gains over a no-unlabeled-data baseline and significantly underperforms robust self-training.
  • Cutout improves adversarial accuracy by 4%, while AutoAugment offers no improvement over the baseline and remains below robust self-training.
  • Data augmentation provides performance essentially identical to robust self-training in the reported certified-robustness comparison.

C.4 Effect of unlabeled data amount

The experiments examine how unlabeled-data quantity affects robust self-training and find gains with more data but possible diminishing returns under the tested configuration.

  • The larger-data experiment was not extended because relevant class-balanced images, training convergence, hyperparameter changes, model capacity, and computation complicate interpretation.
  • Accuracy generally grows with unlabeled-data size, aside from one errant 40K-data adversarial-training point.The authors attribute that drop to an unsuitable training configuration for so few unlabeled examples.
  • 100K unlabeled examples achieve about half the gain obtained with 500K unlabeled examples.
  • Under the tested computation budget, architecture, and training configuration, the benefit of unlabeled data seems likely to plateau around 500K examples.The authors caution that training was not carefully tuned at each data amount, so they cannot describe the trend with confidence.

D.2 Pseudo-label generation and standard self-training.

SVHN experiments use pseudo-labels generated from the labeled core training set, then evaluate standard self-training and certified robustness across perturbation settings. The results show substantial robustness gains from unlabeled data, while comparisons remain constrained by SVHN’s unsettled attack benchmarks.

  • Pseudo-label generation: 96.6% test accuracy was achieved by the Wide ResNet 16-8 model used to generate SVHN pseudo-labels.Pseudo-labels were generated using standard training on the core 73K-image SVHN training set.
  • Standard self-training: 97.1% accuracy was obtained with standard self-training, versus 98.2% when all SVHN labels were used.The pseudo-labels corrupted 1.6% of the extra labels, and self-training still improved over the core-data-only model.
  • Evaluation and attack details: SVHN lacks an agreed benchmark perturbation radius for heuristic ℓ∞ defenses, and prior heuristic defenses have not withstood significant scrutiny.The authors report SVHN results at ϵ = 4/255 and note that attack accuracies appear to saturate after 100 steps there, unlike at ϵ = 8/255.
  • Certified robustness: 59.8% certified accuracy was achieved at an ℓ2 radius containing an ℓ∞ ball of radius 0.01, less than 3% below Gowal et al.The authors identify randomized-smoothing stability training as a viable alternative to interval bound propagation, with tuning of σ and β potentially improving results.

E Comparison to Uesato et al. [48]

The paper compares its semisupervised robustness results with Uesato et al., whose work reaches similar overall conclusions using related theoretical and empirical settings. Differences in algorithms, models, data selection, and training hyperparameters prevent a clear attribution of performance differences to the robust loss.

  • Comparison scope: Uesato et al. independently studied semisupervised adversarial learning theoretically and empirically on CIFAR-10, Tiny Images, and SVHN, reaching conclusions similar to this work.The comparison covers both the Gaussian model and experiments using additional unlabeled data.
  • Algorithmic differences: The strongest Uesato et al. algorithm, UAT++, can be understood as an instance of the paper’s meta-algorithm with a different choice of robust loss.The paper contrasts this choice with its own loss construction, including not maximizing over L_standard.
  • CIFAR-10 comparison: Within 1% robust accuracy, the two works’ best PG_TRADES models differ across architectures, while this work obtains about 3% higher standard accuracy.Uesato et al.’s WRN-106-8 is 1.1% higher in robust accuracy, whereas their WRN-34-8 is 1% lower than the corresponding model here.
  • Limitations of the comparison: Similar final accuracies cannot identify which robust loss performs better because the studies also differ in model size, hyperparameters, batch sizes, and unlabeled-image selection.The data-selection procedures differ in keyword filtering, classifier predictions, and removal of regions around the CIFAR-10 test set.
  • Complementary findings: Uesato et al. report strong improvements in low-labeled-data regimes, tolerance to inaccurate pseudo-labels, and a MultiTargeted attack that lowers state-of-the-art robust accuracies by 3–8%.Contributions unique to this work include theoretical and experimental study of irrelevant data and certified-robustness improvements via randomized smoothing.
Loading 1905.13736v4…