PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive Learning with Stronger Augmentations

Xiao Wang, Guo-Jun Qi

arXiv:2104.07713v2cs.CVcs.AIcs.LG

TL;DR

Existing contrastive methods rely on identity-preserving augmentations, while stronger transformations can expose novel patterns but make retrieval difficult. CLSA uses distributional divergence between weakly and strongly augmented views to supervise strong-query retrieval, achieving strong ImageNet and downstream results, including 76.2% top-1 accuracy on ImageNet.

  • Problem

    Contrastive learning’s carefully designed identity-preserving augmentations limit exploration of novel patterns from stronger transformations, whose distorted views are difficult to retrieve.

  • Method

    CLSA uses distributional divergence between weakly and strongly augmented images over a representation bank to supervise retrieval of strongly augmented queries.

  • Results

    76.2% top-1 accuracy on ImageNet with a standard ResNet-50 nearly matches the 76.5% top-1 accuracy of the fully supervised model, alongside gains on downstream datasets.

  • Takeaways & Limitations

    Strongly augmented images can improve representations for weak and strong views simultaneously and boost concurrent contrastive learning methods.

  • Takeaways & Limitations

    The coefficient and temperature of the contrastive and distributional-divergence losses remain open for further exploration.

Abstract

from arXiv · show

Representation learning has significantly been developed with the advance of contrastive learning methods. Most of those methods have benefited from various data augmentations that are carefully designated to maintain their identities so that the images transformed from the same instance can still be retrieved. However, those carefully designed transformations limited us to further explore the novel patterns exposed by other transformations. Meanwhile, as found in our experiments, the strong augmentations distorted the images' structures, resulting in difficult retrieval. Thus, we propose a general framework called Contrastive Learning with Stronger Augmentations~(CLSA) to complement current contrastive learning approaches. Here, the distribution divergence between the weakly and strongly augmented images over the representation bank is adopted to supervise the retrieval of strongly augmented queries from a pool of instances. Experiments on the ImageNet dataset and downstream datasets showed the information from the strongly augmented images can significantly boost the performance. For example, CLSA achieves top-1 accuracy of 76.2% on ImageNet with a standard ResNet-50 architecture with a single-layer classifier fine-tuned, which is almost the same level as 76.5% of supervised results. The code and pre-trained models are available in https://github.com/maple-research-lab/CLSA.

1 INTRODUCTION

CLSA extends contrastive learning with stronger augmentations by using distributional divergence between weak and strong views, improving representations and retrieval without requiring their embeddings to match.

  • Motivation: Contrastive methods typically use carefully designed augmentations that preserve instance identities for reliable retrieval, limiting exploration of patterns from stronger transformations.Stronger augmentations can distort image structure and make retrieval difficult.
  • Method: CLSA introduces a stronger augmentation formed by random combinations of 14 augmentation types.The listed operations include geometric, color, contrast, and sharpness transformations.
  • Method: CLSA minimizes distribution divergence between weakly and strongly augmented images over a representation bank to supervise strong-query retrieval.This avoids assuming that strong-view and weak-view embeddings should be identical while leveraging information from strong views.
  • Method: CLSA is independent of the contrastive loss and can integrate with methods including MoCo, SimCLR, and BYOL.The framework is designed as a complement to concurrent contrastive approaches.
  • Results: 76.2% top-1 accuracy on ImageNet with ResNet-50 nearly matches the 76.5% top-1 accuracy of the fully supervised model.The paper also reports 93.6% top-1 accuracy on VOC07 and 24.4% APS for small-object detection on COCO.
  • Results: CLSA improves representations for weakly and strongly augmented images simultaneously, while ablation shows naive strong augmentation in contrastive learning degrades performance.The framework’s distributional loss is separately examined through ablation.

2 RELATED WORK

Related work spans self-supervised representation-learning paradigms and the evolution of data augmentation, from manually designed transformations to stronger or automatically combined policies.

  • Self-Supervised Learning: Self-supervised learning seeks robust, general representations without labels through approaches including contrastive learning, clustering, consistency learning, and pretext tasks.These methods use different supervisory signals derived from images or relationships between views.
  • Instance Discrimination and Contrastive Learning: Contrastive and instance-discrimination methods pull positive pairs together and push negative pairs apart, making high-quality pair construction important.Clustering methods can also be viewed as extensions of contrastive learning because cluster centers function like negative prototypes.
  • Generative Methods: Generative methods learn representations with autoencoders or adversarial learning, often focusing on pixel-wise information and latent semantic relationships.BiGAN is cited as an example that captures relationships between latent semantic representations and input images.
  • Consistency Representation Learning: Consistency representation learning enforces agreement between positive pairs and can train an encoder without negative examples, as in BYOL.BYOL uses a Siamese architecture with a predictor in the query branch.
  • Data Augmentation: Early data augmentation used manually designed transformations such as cropping, mirroring, color shifting, and whitening, which required time and expertise.Later work investigated automatically finding useful augmentation combinations.
  • Data Augmentation: Semi-supervised methods also use mixed views, affine and projection transformations, and RandAugment-based distortion to exploit limited labeled data.MixMatch, EnAET, and FixMatch illustrate these directions.
  • Data Augmentation: Self-supervised methods progressively expanded augmentation pipelines with color jittering, Gaussian blur, stronger combinations, multi-crop, and CutMix.InfoMin studied augmentation combinations, while SwAV used additional smaller crops and BoWNet incorporated CutMix.

3 CLSA: CONTRASTIVE LEARNING WITH STRONGER AUGMENTATIONS

CLSA addresses the difficulty of using stronger augmentations in contrastive learning, which can distort image structures and make same-instance retrieval unreliable. It instead aligns weakly and strongly augmented queries through distributional divergence over a representation bank, preserving novel strong-augmentation patterns without forcing identical embeddings.

  • Motivation: Stronger transformations can distort image structures and perceptual patterns, making augmented views difficult to treat as the same instance in contrastive training.
  • CLSA Framework: CLSA introduces Distributional Divergence Minimization between weakly and strongly augmented images over a representation bank.
  • CLSA Framework: The method avoids directly forcing a strongly augmented embedding to match its positive target while using weak-query distributions to expose novel strong-augmentation patterns.
  • CLSA Framework: CLSA learns weakly and strongly augmented patterns simultaneously through an overall optimization combining contrastive learning and distributional divergence minimization.

4 EXPERIMENTS

Experiments show CLSA improves representation quality under stronger augmentations, matching or exceeding contrastive baselines across ImageNet, downstream tasks, runtime comparisons, and framework extensions. The DDM loss is central to exploiting distorted strong views without performance degradation.

  • Linear Classification on ImageNet: CLSA compares two ImageNet linear-evaluation settings: CLSA uses one 96×96 stronger augmentation, whereas CLSA* sums DDM losses across five crop sizes.
  • Linear Classification on ImageNet: 73.3% top-1 accuracy from CLSA* surpasses SWAV’s 72.7% with multi-crop training over 200 epochs.
  • Transfer Learning Results on Downstream Tasks: 93.6% top-1 accuracy on VOC07 exceeds the previous 88.9% record, while COCO small-object APS rises to 24.4% from 20.8%.
  • Ablation Study: Replacing DDM with contrastive loss reduces top-1 accuracy to 68.0% from 69.4%, supporting distributional supervision over a one-hot approximation.
  • Ablation Study: 68.3% accuracy in 52.5 hours beats MoCo v2’s 67.5% in 53 hours, indicating faster convergence at comparable training time.
  • Ablation Study: CLSA improves strongly augmented-image KNN accuracy by 8.2% at 200 epochs and 9.9% at 800 epochs relative to MoCo v2.
  • Ablation Study: Extending CLSA to MoCo v2 and BYOL yields improvements of +1.9% and +0.8%, respectively, supporting compatibility with other contrastive approaches.

5 CONCLUSION

The paper presents CLSA as a distributional-divergence framework for learning from strongly augmented images. It reports strong benchmark and downstream performance, while identifying compatibility with other contrastive methods as an avenue for further improvement.

  • CLSA uses distributional divergence to learn information from strongly augmented images and outperforms reported supervised and self-supervised comparisons across datasets.
  • The DDM loss is independent of concurrent contrastive-learning methods, allowing CLSA to combine with methods such as MoCo, SimCLR, AdCo, and BYOL.
Loading 2104.07713v2…