Source-linked AI summary
AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations from Self-Trained Negative Adversaries
Qianjiang Hu, Xiao Wang, Wei Hu, Guo-Jun Qi
TL;DR
Contrastive learning needs hard negatives, but existing queues update only part of their contents while minibatch-only methods discard past negatives. AdCo jointly alternates representation learning with adversarial-negative updates, achieving strong ImageNet accuracy and efficient pretraining.
Problem
Existing contrastive methods either update only a small portion of queued negatives or discard past-minibatch negatives, limiting representation tracking or requiring larger minibatches.
Method
AdCo alternately updates a representation network and trainable negative adversaries, maximizing adversarial contrastive loss so negatives track positive queries.
Results
75.7% top-1 accuracy is achieved over 800 ImageNet pretraining epochs, exceeding BYOL at 74.3% and SWAV at 75.3%.
Takeaways & Limitations
AdCo updates negative examples as a whole and retains past information, supporting efficient pretraining with fewer epochs.
Takeaways & Limitations
An alternative adversarial loss performed worse than AdCo, and its theoretical implications require further study.
Abstract
from arXiv · showhide
Contrastive learning relies on constructing a collection of negative examples that are sufficiently hard to discriminate against positive queries when their representations are self-trained. Existing contrastive learning methods either maintain a queue of negative samples over minibatches while only a small portion of them are updated in an iteration, or only use the other examples from the current minibatch as negatives. They could not closely track the change of the learned representation over iterations by updating the entire queue as a whole, or discard the useful information from the past minibatches. Alternatively, we present to directly learn a set of negative adversaries playing against the self-trained representation. Two players, the representation network and negative adversaries, are alternately updated to obtain the most challenging negative examples against which the representation of positive queries will be trained to discriminate. We further show that the negative adversaries are updated towards a weighted combination of positive queries by maximizing the adversarial contrastive loss, thereby allowing them to closely track the change of representations over time. Experiment results demonstrate the proposed Adversarial Contrastive (AdCo) model not only achieves superior performances (a top-1 accuracy of 73.2\% over 200 epochs and 75.7\% over 800 epochs with linear evaluation on ImageNet), but also can be pre-trained more efficiently with fewer epochs.
1. Introduction
AdCo addresses limitations of queue-based and minibatch-only negatives by jointly training representation networks with adversarial negatives. It reports stronger ImageNet accuracy and efficient pretraining with fewer epochs.
- Queue-based methods update only a small portion of negatives, limiting their ability to track rapidly changing representations.
- Minibatch-only methods discard past negatives and often require larger minibatches, increasing memory and computation.
- AdCo alternately updates a representation network and negative adversaries, with the adversaries maximizing contrastive loss against positive queries.
- 44.4% top-1 accuracy is achieved after 10 epochs of pretraining.
- AdCo outperforms MoCHi by 4.1% in top-1 accuracy over 800 epochs and is reported as 5% higher than MoCo v2 over the same training duration.
- 75.7% top-1 accuracy is achieved over 800 epochs, compared with BYOL at 74.3% and SWAV at 75.3%.The reported accuracy uses linear evaluation on ImageNet and comparable or smaller GPU time.
2. Related Works
Related work frames contrastive learning as inter-instance discrimination using negative examples, alongside transformation-prediction approaches that learn equivariant features without labels.
- Contrastive learning: Contrastive learning distinguishes positive queries from negative examples to learn unsupervised representations.
- Contrastive learning: Instance discrimination has used minibatches, memory banks, and dynamic queues with momentum updates to organize negative examples.
- Contrastive learning: Negative examples challenge self-trained representations and support progressively more discriminative features.
- Transformation prediction: Transformation prediction instead learns representations equivariant to transformations of images or 3D point clouds.
3. The Proposed Approach
AdCo trains a representation network against directly optimized negative adversaries in a minimax framework. The adversaries track difficult queries through weighted updates, while convergence is empirical rather than theoretically guaranteed.
- Preliminaries: Contrastive learning minimizes a loss that distinguishes augmented positive pairs from a set of negative representations.
- Motivation: Existing negative-construction methods either update a queue incrementally or use only current-minibatch negatives, losing timely updates or past information.
- Motivation: AdCo updates the entire negative set adversarially, allowing it to track representation changes while retaining information from past minibatches.
- Adversarial training: The method jointly trains a backbone network and negative adversaries by minimizing over network parameters and maximizing over negative samples.
- Adversarial training: Alternating gradient descent and ascent updates the representation network and negative adversaries, with positive learning rates for both players.
- Adversarial training: The alternating procedure works empirically, although no theory guarantees convergence to the minimax saddle point.
- Negative-adversary updates: Each negative sample moves toward a probability-weighted combination of current queries, emphasizing queries that are difficult to distinguish.
4. Further Discussions
The paper reviews an alternative adversarial loss, derives how AdCo adapts negative adversaries toward low-density queries, and reports why the original AdCo loss remains preferable in experiments.
- Alternative adversarial loss: The section presents an alternative adversarial loss that replaces p(n_k|q_i) with p(q_i|n_k) as the weighting factor.This choice makes the derivative a conditional expectation of query representations.
- Alternative adversarial loss: The resulting loss provides an alternative explanation of how negative samples can be trained through conditional expectations.The paper states that this alternative may motivate future forms of adversarial loss and merits further theoretical study.
- Experimental choice: The authors retain the AdCo loss in experiments because preliminary experiments found the alternative loss performed worse than AdCo.They attribute this result to the alternative loss not directly opposing the contrastive loss used to train the embedding network.
- Adapting to Low-Density Queries: For low-density queries, normalized query representations have larger values, so the derivatives push negative adversaries closer to under-represented queries.This adaptation helps the negative adversaries cover queries that existing negative examples represent poorly.
5. Experiments
Experiments evaluate AdCo on ImageNet linear classification, transfer learning, and object detection, using ResNet-50 pretraining and comparisons with existing methods. AdCo delivers strong accuracy, faster effective pretraining, and competitive computational cost.
- Training Details: AdCo uses ResNet-50 with a 128-D normalized feature vector, batch size 256, and 65,536 negative adversaries alternately updated with the backbone.The negative-adversary count matches MoCo's queue length for direct comparison.
- Linear Classification on ImageNet: AdCo achieves the best compared ImageNet performance after 200 pretraining epochs while keeping GPU time per epoch comparable to leading contrastive methods.Updating negative adversaries adds negligible overhead relative to backbone updates.
- Linear Classification on ImageNet: 75.7% top-1 accuracy is achieved after 800 epochs with linear evaluation on ImageNet, while AdCo also surpasses MoCo v2 and MoCHi without multi-crop augmentation.Without multi-crop augmentation, the reported comparisons are 72.8% versus 71.1% for MoCo v2 and 72.8% versus 68.7% for MoCHi.
- Transfer Learning and Object Detection: AdCo improves downstream transfer results, including 93.1% VOC07 classification accuracy versus 88.9% for SWAV and a 3.3%–3.6% COCO APS improvement.The evaluation covers cross-dataset classification and object detection.
- Analysis and Visualization of Results: With 10 pretraining epochs, AdCo exceeds MoCo v2 by more than 5% top-1 accuracy and generally reaches comparable accuracy 30–50 epochs earlier.Both methods use K = 65,536 negative samples in this comparison.
- Analysis and Visualization of Results: t-SNE visualizations show fewer negative outliers for AdCo than MoCo v2, indicating closer tracking of positive representations across iterations.MoCo v2 outliers form isolated clusters that contribute less to contrastive training.
6. Conclusions
AdCo learns challenging negative adversaries that update collectively to track changing representations and make positive-query discrimination harder. Experiments across multiple downstream tasks support its performance and pretraining efficiency.
- AdCo learns challenging negative adversaries that criticize and improve deep-network representations.
- The adversaries maximize adversarial contrastive loss, updating negative examples as a whole to track changing representations.
- Each negative adversary is pushed toward a weighted combination of positive queries.
- Experiments on multiple downstream tasks demonstrate superior performance and efficient representation pretraining.
Appendix for “AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations from Self-Trained Negative Adversaries”
The appendix examines factors affecting AdCo and reports gains from symmetric loss, robustness to fewer negative adversaries, and comparable computing costs against BYOL.
- Symmetrizing AdCo's contrastive loss increases top-1 accuracy by 1.5 ∼2.0% over its asymmetric counterpart.
- Reducing negative samples from 65,536 to 8,192 leaves AdCo's top-1 accuracy unaffected.
- AdCo and its negative samples have computing costs comparable to the compared methods.
A. Symmetric Loss
Symmetric loss improves learned representations in several self-supervised models, and applying it to AdCo yields a larger gain with single-crop augmentations than with multi-crop augmentations.
- Symmetric loss improves downstream representation performance in SimCLR and BYOL.
- 1.5 ∼2.0%: AdCo's performance improves when its contrastive loss is symmetrized with other factors fixed.
- 73.2% to 73.6%: with multi-crop augmentations over 200 epochs, symmetric loss only marginally improves top-1 accuracy.
B. Numbers of Negative Adversaries
AdCo's top-1 accuracy is nearly insensitive to reducing the number of negative samples, supporting effective pretraining with substantially fewer adversaries.
- 65,536 to 8,192: AdCo's top-1 accuracy is almost unaffected when negative samples are reduced to one-eighth.
- 0.2 ∼0.4: top-1 accuracy varies only slightly as negative samples decrease to one-eighth of the original size.
- AdCo's efficient adversarial training may make fewer negative adversaries sufficiently representative of the learned embedding space.
C. Do We Still Need Negative Samples?
The comparison asks whether contrastive learning remains useful when BYOL removes explicit negative samples. AdCo matches BYOL accuracy with a smaller batch size and uses less GPU time, while both methods include extra model components.
- AdCo and BYOL outperform the compared models when symmetric loss is applied.
- BYOL removes explicit negative samples but depends on an extra prediction MLP for competitive results.
- AdCo similarly adds trainable negative samples as an additional neural layer, allowing their parameters to be trained end-to-end.
- AdCo achieves the same top-1 accuracy as BYOL with a smaller batch size, while BYOL needs almost 20% more GPU time.The comparison uses 200 epochs of ImageNet pretraining with single-crop augmentation.