Source-linked AI summary
Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, Dawn Song
TL;DR
The paper asks whether self-supervision remains useful when labeled datasets and standard supervised training are available, given its limited clean-accuracy gains. It combines supervised learning with self-supervised objectives and finds improved robustness across several conditions plus stronger near-distribution out-of-distribution detection, sometimes exceeding fully supervised methods.
Problem
Existing self-supervised approaches lag behind fully supervised training on standard accuracy, and their effects on robustness and uncertainty are not the usual focus despite imperfect real-world inputs.
Method
The paper supplements supervised cross-entropy with an auxiliary rotation loss, using pretraining followed by fine-tuning with the combined objective.
Results
Self-supervision improves robustness to adversarial examples, label corruption, and common input corruptions, while outperforming fully supervised methods for difficult out-of-distribution detection in CIFAR and ImageNet.
Takeaways & Limitations
Robustness and uncertainty estimation should serve as additional evaluation axes for self-supervised learning beyond clean accuracy and annotation efficiency.
Takeaways & Limitations
On CIFAR-100, auxiliary rotation predictions increase error by 2.3% at 0% corruption strength, although they improve performance past 20% corruption strength.
Abstract
from arXiv · showhide
Self-supervision provides effective representations for downstream tasks without requiring labels. However, existing approaches lag behind fully supervised training and are often not thought beneficial beyond obviating or reducing the need for annotations. We find that self-supervision can benefit robustness in a variety of ways, including robustness to adversarial examples, label corruption, and common input corruptions. Additionally, self-supervision greatly benefits out-of-distribution detection on difficult, near-distribution outliers, so much so that it exceeds the performance of fully supervised methods. These results demonstrate the promise of self-supervision for improving robustness and uncertainty estimation and establish these tasks as new axes of evaluation for future self-supervised learning research.
1 Introduction
Although self-supervised methods often lag supervised training on clean accuracy, combining them with full supervision improves robustness and uncertainty estimation. These benefits include multiple corruptions and difficult out-of-distribution detection, even when clean accuracy remains unchanged.
- Self-supervised methods often lag behind fully supervised training on standard accuracy metrics and add little clean-accuracy benefit on fully labeled datasets.This motivates testing whether large labeled datasets make self-supervision needless.
- Self-supervision improves robustness to adversarial examples, label corruptions, and common input corruptions while clean accuracy stays constant.The evaluated common corruptions include fog, snow, and blur.
- Self-supervised learning greatly improves out-of-distribution detection for difficult, near-distribution examples and surpasses fully supervised methods on CIFAR-10 and ImageNet.
- Combining self-supervision with full supervision provides strong regularization that improves robustness and uncertainty estimation without requiring larger models or additional data.The methods can also be used with task-specific methods for additive effect with no additional assumptions.
2 Related Work
Related work covers self-supervised pretext tasks, robustness to imperfect training and testing conditions, and out-of-distribution detection. The paper distinguishes its focus by examining how self-supervision affects robustness and uncertainty, rather than only representation learning without labels.
- Self-supervised learning: Prior self-supervised methods learn representations through pretext tasks including patch position, surrogate classes, rotation prediction, colorization, clustering, and mutual-information maximization.
- Self-supervised learning: Figure 1 motivates rotation prediction because determining whether a zebra is flipped requires modeling shape, whereas texture alone may suffice for ideal-condition classification.
- Robustness: Robustness research evaluates resistance to common image corruptions, corrupted labels, and adversarial examples under imperfect training or testing conditions.
- Out-of-distribution detection: Out-of-distribution detection includes methods based on traditional one-class models and supervised deep representations, including maximum softmax probability.
3 Robustness
Auxiliary rotation self-supervision improves robustness to adversarial perturbations, common input corruptions, and label corruption while leaving clean accuracy largely unchanged. It complements existing defenses and can outperform increased trusted-data access in some label-noise settings.
- Adversarial perturbations: 5.6% absolute improvement over classical PGD training is achieved for both 20-step and 100-step adversaries.The method also withstands up to 11% larger perturbations without a performance drop.
- Adversarial perturbations: Auxiliary rotation prediction combines with PGD and TRADES, producing a 1.22% robust-accuracy boost over standard TRADES.It provides a 7.79% boost over standard PGD training and is not degraded by SPSA attacks.
- Common input corruptions: 76.9% average accuracy on corrupted images exceeds normal training’s 72.3%, while clean accuracy remains similar at 95.5% versus 94.7%.Gains occur across corruption types, including glass blur, Gaussian noise, contrast, shot noise, and frost.
- Label corruptions: 5.6% lower average error on CIFAR-10 and 5.2% lower average error on CIFAR-100 are obtained without loss correction.These correspond to 26% and 11% relative improvements over normal training, respectively.
- Label corruptions: 10.5% average error with GLC and auxiliary rotations beats both 14.6% with GLC using 5% trusted data and 11.6% with doubled trusted data.Auxiliary rotation prediction is complementary to GLC and produces softer degradation as corruption strength increases.
- Label corruptions: At 0% corruption strength on CIFAR-100, auxiliary rotations raise error by 2.3%, while improving performance at every tested strength above 20%.The clean-data versus robustness trade-off depends on the application, and longer fine-tuning can eliminate the initial gap while risking label-noise overfitting.
4 Out-of-Distribution Detection
Self-supervised auxiliary tasks improve out-of-distribution detection, including difficult near-distribution outliers, without requiring additional data. The approach combines geometric-transformation prediction with supervised representations and can outperform fully supervised baselines.
- Self-supervised rotation prediction improves OOD detection when the in-distribution data contain multiple classes or a single class.
- 4.1 Multi-Class Out-of-Distribution Detection: The multi-class detector combines supervised classification with auxiliary rotation prediction and adds a rotation score to a KL-divergence detector.
- 4.1 Multi-Class Out-of-Distribution Detection: 4.8% average AUROC improvement is reported across Gaussian, Rademacher, Blobs, Textures, SVHN, Places365, LSUN, and CIFAR-100 anomalies.
- 4.2.1 CIFAR-10: In the one-class CIFAR-10 setting, OOD examples come from the remaining classes, creating a difficult near-distribution detection challenge.
- 4.2.1 CIFAR-10: The self-supervised detector uses transformed inputs and three softmax heads to predict rotations and vertical and horizontal translations.
- 4.2.1 CIFAR-10: On CIFAR-10 and ImageNet, self-supervised methods surpass fully supervised OOD models; combining self-supervision with Outlier Exposure nearly solves the CIFAR-10 task.
5 Conclusion
The paper applies self-supervised learning to improve model robustness and uncertainty beyond purely supervised approaches. Supplementing supervised methods with an auxiliary rotation loss produced consistent robustness gains across the studied settings.
- The approach targets improved robustness and uncertainty beyond purely supervised deep-learning methods.
- Large improvements were observed against adversarial examples, label corruption, and common input corruptions.
- Consistent gains across all studied robustness types came from supplementing supervised methods with an auxiliary rotation loss.
A Self-Supervised Learning for Multi-Class OOD Detection
The paper evaluates multi-class out-of-distribution detection using CIFAR-10 as the in-distribution dataset and compares a maximum-softmax baseline with auxiliary rotation prediction. Rotation prediction improves detection across numerous anomaly types and supports multi-class detectors.
- Table 5 reports multi-class out-of-distribution detection results for the maximum softmax probability baseline and the rotation method.All results are percentages averaged across five runs.
- Auxiliary rotation prediction produces large improvements across numerous anomaly types.
- Rotation prediction improves performance in all reported cases and can augment detectors based on multi-class representations.
- The evaluation uses CIFAR-10 as the in-distribution dataset and includes Gaussian, Rademacher, Blobs, Textures, and SVHN out-of-distribution data.Gaussian and Rademacher images are procedurally sampled, while Blobs and Textures provide generated or image-based anomalies.
B ImageNet OOD Dataset
The ImageNet out-of-distribution dataset uses 30 classes selected to avoid obvious semantic overlap. It contains 1,300 training images and 100 test images per class, with expanded test sets collected from validation and additional images.
- The dataset contains 30 ImageNet classes, including acorn, airliner, ambulance, banjo, dragonfly, and volcano.
- The classes were selected to avoid obvious overlap, unlike semantically related pairs such as bee and honeycomb.
- Each class has 1,300 training images and 100 test images.The test set combines ImageNet’s 50 validation images with 50 additionally collected images per class.
- The expanded test dataset is available for download from the paper’s GitHub repository.
C Additional Ablations
Additional ablations show that the auxiliary rotation loss, rather than attacking the rotation branch, is central to the robustness result. Rotation prediction also outperforms rotation data augmentation for corruption robustness.
- Attacking only the classification branch performs similarly to attacking both branches, indicating that the rotation loss itself is the crucial component.
- 72.3% average corruption accuracy falls to 63.7% with rotation data augmentation, whereas auxiliary rotation prediction raises it to 76.9%.