Source-linked AI summary
Contrastive Learning with Adversarial Examples
Chih-Hui Ho, Nuno Vasconcelos
TL;DR
Self-supervised contrastive learning lacks systematic selection of challenging negative pairs within batches. The paper introduces adversarial examples and CLAE to generate challenging positive and hard negative pairs while accounting for all batch images. CLAE improves several contrastive-learning baselines across datasets.
Problem
Prior contrastive-learning approaches do not necessarily optimize on hard negative pairs or account for relationships among images within a batch.
Method
CLAE uses adversarial augmentations and adversarial training to generate challenging positive pairs and hard negative pairs for self-supervised contrastive learning.
Results
CLAE improves the performance of several contrastive-learning baselines across different datasets.
Takeaways & Limitations
Adversarial examples can be leveraged to improve contrastive self-supervised learning by producing more challenging training pairs.
Takeaways & Limitations
The formulation is ill-defined because the relevant quantity is the difference between two transformations rather than their absolute values.
Abstract
from arXiv · showhide
Contrastive learning (CL) is a popular technique for self-supervised learning (SSL) of visual representations. It uses pairs of augmentations of unlabeled training examples to define a classification task for pretext learning of a deep embedding. Despite extensive works in augmentation procedures, prior works do not address the selection of challenging negative pairs, as images within a sampled batch are treated independently. This paper addresses the problem, by introducing a new family of adversarial examples for constrastive learning and using these examples to define a new adversarial training algorithm for SSL, denoted as CLAE. When compared to standard CL, the use of adversarial examples creates more challenging positive pairs and adversarial training produces harder negative pairs by accounting for all images in a batch during the optimization. CLAE is compatible with many CL methods in the literature. Experiments show that it improves the performance of several existing CL baselines on multiple datasets.
1 Introduction
Contrastive learning uses augmented positive pairs for self-supervised representation learning, but prior approaches give less attention to challenging negatives. The paper introduces adversarial examples and CLAE to address both pair-generation challenges.
- Contrastive learning: Contrastive learning generates paired augmentations of unlabeled instances and trains an encoder with a contrastive loss.Pairs from the same instance are positive pairs whose embeddings are encouraged to be close.
- Pair construction: Positive-pair construction has received substantial attention through transformations such as rotation, color jittering, and scaling.Different augmentation approaches have different effects on self-supervised performance.
- Negative-pair selection: Prior self-supervised contrastive methods generally do not systematically select difficult negative pairs within a batch.They treat instances independently, unlike metric-learning approaches that construct batches with challenging negatives.
- Adversarial augmentation: The paper frames pair generation as finding augmentation sets that maximize contrastive optimization cost and uses adversarial examples to synthesize them.The goal is better self-supervised representations rather than robustness to adversarial attacks.
- Contributions: CLAE produces adversarial pairs accounting for both positive and negative pairs in a batch, and experiments show gains across several contrastive baselines and datasets.The paper presents adversarial augmentation as a way to improve self-supervised learning performance.
2 Related work
Related work has established contrastive learning and adversarial examples separately, while leaving systematic hard-negative selection in unlabeled contrastive learning unresolved. CLAE connects these areas by defining adversarial augmentations that account for relationships among all batch images.
- Contrastive learning: Contrastive learning is used in self-supervised learning to learn invariant image representations from augmented image pairs.The approach minimizes a contrastive loss over feature vectors extracted from augmentations.
- Contrastive learning: Most self-supervised contrastive methods do not mine hard negatives or relate image instances within a batch systematically.Prior work mentions negative-pair selection but does not provide a systematic algorithm.
- Hard-negative mining: Metric-learning hard-negative mining becomes increasingly difficult as the number of possible positive and negative pairs grows with dataset size.The related-work discussion notes cubic growth for triplet-loss pair combinations.
- Novelty: The paper proposes adversarial augmentations that maximize contrastive loss while considering how all unlabeled batch images relate to one another.This differs from metric-learning hard-negative mining because self-supervised learning provides no class labels.
- Adversarial examples: Adversarial examples have traditionally been used to induce network errors and, through adversarial training, increase robustness in supervised settings.The paper draws on this literature while pursuing a different self-supervised objective.
- Novelty: CLAE targets better contrastive embeddings by generating challenging positive pairs and effective hard negatives, not by robustifying representations against attacks.Its pair-based adversaries require a definition distinct from conventional single-example adversarial methods.
3 Leveraging adversarial examples for improved contrastive learning
The paper introduces adversarial augmentations and CLAE to create challenging positive and negative pairs for contrastive learning, while accounting for relationships among all batch instances. The procedure combines adversarially generated augmentations with contrastive training and can be generalized across many CL methods.
- Adversarial augmentation: CLAE introduces adversarial examples for contrastive learning and uses them in a training scheme designed to improve representation learning.The approach generates adversarial augmentations and trains with contrastive loss.
- Contrastive learning: Contrastive learning uses two transformed views of each unlabeled image to learn an invariant representation.Transformations are sampled from a transformation set T to produce augmentation pairs.
- Adversarial augmentation: The adversarial augmentation objective selects perturbations that maximize contrastive risk after fixing one transformation, resolving the ambiguity between absolute and relative transformations.The original optimization is ill-defined because the difference between transformations matters more than their absolute values.
- Adversarial augmentation: Unlike standard input attacks, the contrastive adversarial optimization perturbs classifier weights whose effects appear across all batch instances.This makes the optimization account for the batch simultaneously and seek challenging positive and negative pairs.
- Adversarial training: Adversarial augmentations are generated by backpropagating contrastive-loss gradients to the input, then combining standard and adversarial contrastive terms for training.The implementation uses untargeted attacks with FGSM, while the overall loss balances clean and adversarial components through α.
- Adversarial training: The procedure can be applied to most contrastive-learning methods and adapts by replacing the underlying contrastive objective while retaining the adversarial-training framework.The paper explicitly discusses generalization to methods such as UEL and SimCLR.
4 Experiments
Experiments evaluate adversarial contrastive learning across datasets, baselines, perturbations, architectures, training settings, and transfer tasks. CLAE consistently improves contrastive-learning performance, can reduce dependence on batch size and training duration, and generalizes across downstream datasets.
- Setup: Experiments use CIFAR10, CIFAR100, and tinyImagenet with Plain, UEL, and SimCLR baselines, evaluated through downstream classification.Unless otherwise noted, a ResNet18 encoder is trained with Algorithm 1, α = 1, standard PyTorch augmentation, and adversarial batch-normalization momentum 0.011.
- Influence of adversarial examples: Adversarial perturbations produce larger contrastive and downstream cross-entropy losses than random perturbations of equal magnitude.This comparison is reported on CIFAR10 for the contrastive pretext task and downstream classification.
- Influence of adversarial examples: 20%: adversarial examples degrade downstream classification accuracy in SSL, a weaker effect than previously reported for supervised learning.Batch-normalization statistics also differ less between clean and adversarial examples than in the supervised setting.
- CLAE performance: Adversarial training improves all evaluated CL algorithms on all datasets, with gains consistent across downstream classifiers.The baseline is recovered when ϵ = 0, while the best performance is reported using ϵ = 0.03 in the supplied passage.
- Ablation study: 54%: adversarial training with ϵ = 0.03 and batch size 64 outperforms the baseline with batch size 256 at 53.79%.Larger batch sizes improve both methods, but adversarial training makes large batch sizes less critical; embedding dimension has little apparent effect.
- Ablation study: 5%: larger-network gains reach 5% for adversarial training, compared with at most 1% for the baseline; ResNet18 with adversarial training beats the ResNet101 baseline.The authors attribute the stronger architecture effect to larger networks benefiting from more challenging adversarial examples.
- Ablation study: α = 0.8 gives peak accuracy while performance remains fairly stable for α > 0.2, and adversarial training consistently beats the ϵ = 0 baseline across attack strengths.The reported stability includes α = 2 and varying ϵ, with batch normalization proposed as a possible explanation for the latter.
- Ablation study: 300 epochs: adversarial augmentation approaches the baseline at 400 epochs, and at 400 epochs it outperforms the baseline at 500 epochs.Adversarial training also beats the baseline consistently with longer pretext training; ϵ = 0.07 benefits more from longer training than smaller perturbations.
5 Conclusion
CLAE addresses the lack of hard-negative optimization in contrastive SSL by using adversarial examples to generate more challenging positive and negative pairs. The proposed adversarial augmentations improve several CL baselines.
- CLAE generates more challenging positive and hard negative pairs on-the-fly by leveraging adversarial examples.
- Adversarial training with the proposed augmentations improves the performance of several contrastive-learning baselines.
6 Broader Impact
The work advances self-supervised learning without labels and broadens deep learning's applicability where dataset annotations are difficult to obtain. It also suggests that optimization choices can reduce the importance of larger models, batches, and longer training.
- The work advances self-supervised learning solutions where no labels are provided.
- The approach may support applications where dataset annotations are difficult to obtain.
- The experiments suggest that larger networks, larger batch sizes, and longer training epochs are less critical when optimizing the representation.