PaperlayerView exact revision on arXiv

Source-linked AI summary

ReSSL: Relational Self-Supervised Learning with Weak Augmentation

Mingkai Zheng, Shan You, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang, Chang Xu

arXiv:2107.09282v2cs.CVcs.LG

TL;DR

Many self-supervised methods emphasize matching augmented views of the same instance, leaving relationships among different instances less attended. ReSSL models those relationships with sharpened similarity distributions and weak augmentations, and reports stronger performance and training efficiency than prior methods.

  • Problem

    Many self-supervised methods focus on instance-level matching, while relationships between different instances receive less attention.

    Source: §1 · p. 2

  • Method

    ReSSL aligns pairwise-similarity relationships across augmentations using a sharpened target distribution, weak augmentations, and a momentum strategy.

    Source: §1 · p. 2

  • Results

    69.9% on ImageNet linear evaluation after 200 epochs is 2.4% higher than MoCoV2, and ReSSL is reported to achieve state-of-the-art performance under the same training cost.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The large-scale experiments demonstrate ReSSL's efficiency and state-of-the-art performance for unsupervised representation learning.

    Source: §6 · p. 10

Abstract

from arXiv · show

Self-supervised Learning (SSL) including the mainstream contrastive learning has achieved great success in learning visual representations without data annotations. However, most of methods mainly focus on the instance level information (\ie, the different augmented images of the same instance should have the same feature or cluster into the same class), but there is a lack of attention on the relationships between different instances. In this paper, we introduced a novel SSL paradigm, which we term as relational self-supervised learning (ReSSL) framework that learns representations by modeling the relationship between different instances. Specifically, our proposed method employs sharpened distribution of pairwise similarities among different instances as \textit{relation} metric, which is thus utilized to match the feature embeddings of different augmentations. Moreover, to boost the performance, we argue that weak augmentations matter to represent a more reliable relation, and leverage momentum strategy for practical efficiency. Experimental results show that our proposed ReSSL significantly outperforms the previous state-of-the-art algorithms in terms of both performance and training efficiency. Code is available at \url{https://github.com/KyleZheng1997/ReSSL}.

1 Introduction

Existing self-supervised methods often focus on matching views of the same instance, while instance discrimination can separate similar images and clustering can impose noisy equipartition constraints. ReSSL instead aligns relationships across instances and uses weak augmentation and a sharpened target distribution; the paper reports strong benchmark results and training efficiency.

1 Introduction

  • Instance discrimination can push similar images apart, while deep clustering may introduce noise through its equipartition assumption.The paper notes that similar images should instead be pulled closer.Source: §1 · pp. 1–2
  • ReSSL maintains pairwise-similarity relationships across augmentations rather than explicitly pushing different instances apart.It uses a similarity distribution as a metric to align different views of the same images.Source: §1 · p. 2
  • 69.9% on ImageNet linear evaluation after 200 epochs is 2.4% higher than MoCoV2; with Multi-Crop, ReSSL reaches 74.7% Top-1 accuracy, 1.4% higher than CLSA-Multi.The paper also describes ReSSL as achieving state-of-the-art performance under the same training cost.Source: §1 · p. 2
  • Weak augmentations and a sharpened target distribution are proposed to provide a more stable, higher-quality similarity relation.The paper also describes a momentum strategy as a way to improve practical efficiency.Source: §1 · p. 2

2 Related Works

Related work spans pretext-task SSL, instance discrimination and deep clustering, as well as approaches that learn without explicit negative pairs. The section contrasts their mechanisms and notes trade-offs involving representation generality, memory, and clustering assumptions.

Self-Supervised Learning

  • Earlier pretext tasks include colorization, jigsaw solving, super-resolution, inpainting, patch-offset prediction, rotation prediction, and image reconstruction, but are described as lacking representation generality.The cited examples include,,,,,,.Source: §2 · p. 2

Instance Discrimination

  • Instance discrimination treats views of one image as positives and views from different images as negatives; SimCLR [7] [8] highlights augmentation, projection heads, and large batches.MoCo [24] [9] uses a slowly progressing key encoder and memory buffer to reduce reliance on large batches.Source: §2 · pp. 2–3

Deep Clustering

  • Deep clustering groups similar samples, with SeLa using equipartition constraints and SwAV extending the idea to scalable online clustering.PCL combines instance discrimination and unsupervised clustering, reporting better downstream-task performance than MoCoV2 despite the same linear classification accuracy.Source: §2 · p. 3

Contrastive Learning Without Negatives

  • BYOL predicts a target-network representation without negatives, while SimSiam reports learning meaningful representations without negatives, large batches, or momentum encoders.BYOL adds a predictor to the online encoder to avoid model collapse.Source: §2 · p. 3

3 Methodology

The methodology moves from standard view-based self-supervised objectives to ReSSL, which aligns relationships across augmented views. Weak teacher augmentation and momentum targets support this relational approach.

3.1 Preliminaries on Self-supervised Learning

  • Standard SSL forms two views of unlabeled samples and maps them through an encoder and projection head; contrastive methods use NCE, while BYOL and SimSiam add a predictor.The section also notes stop-gradient and momentum teachers as techniques used to avoid collapse.Source: §3.1 · p. 3

3.2 Relational Self-Supervised Learning

  • A momentum-updated teacher supplies the student’s target, while a FIFO embedding buffer approximates large-batch relationships and stabilizes that target.The teacher parameters are updated as an exponential moving average of the student’s parameters.Source: Figure 1 · p. 4, §3.2 · pp. 4–5
  • ReSSL uses pairwise similarities to preserve relationships across views, relaxing the requirements to push different instances apart or make same-instance features identical.The method instead requires augmented views to share similar, not necessarily identical, features.Source: §3.2 · p. 4
  • Two-view embeddings yield relationship distributions that ReSSL aligns by minimizing KL divergence, implemented as cross-entropy because the teacher distribution is the target.The target distribution is sharpened with a smaller temperature, emphasizing the most similar features.Source: §3.2 · pp. 4–5
  • The teacher uses weak augmentation because standard contrastive augmentation can add disturbances and mislead the student, while reducing target quality and stability.The stated purpose is to improve the quality and stability of the target distribution.Source: §3.2 · p. 5

4 Empirical Study

ReSSL outperforms prior methods on four benchmarks, while the experiments examine temperature sharpening, weak teacher augmentation, relation-buffer size, and learned representations.

4 Empirical Study

  • Experiments compare SimCLR, BYOL, SimSiam, and MoCoV2 on CIFAR-10/100, STL-10, and Tiny ImageNet using 200-epoch pretraining and linear evaluation.For STL-10, pretraining uses labeled and unlabeled images, while linear evaluation uses only labeled images.Source: §4 · pp. 5–6
  • ReSSL outperforms previous methods on all four benchmarks, while most earlier methods require twice the back-propagation and incur higher training cost.Table 1 compares SSL algorithms on small and medium datasets.Source: §4 · p. 6, Table 1 · p. 6

4.1 A Properly Sharpened Relation is A Better Target

  • Across datasets, τt performs best at 0.04 or 0.05, and τs=0.1 consistently outperforms τs=0.2; the target should be sharper than the student distribution.The authors report that τt should be neither so small that the target becomes one-hot nor so large that it becomes too flat.Source: §4.1 · p. 6, Table 2 · p. 7

4.2 Weak Augmentation Makes Better Relation

  • Weak teacher augmentation boosts performance across datasets; the authors attribute this to more accurate similarity guidance from smaller teacher-view disturbances.The teacher uses random resized crops and random horizontal flips; Figure 2 compares nearest neighbors under weak and typical contrastive augmentation.Source: §4.2 · pp. 6–7, Figure 2 · p. 7, Table 3 · p. 7

4.3 Dimension of the Relation

  • Increasing the memory buffer from 256 to 32768 can improve performance, but gains become marginal once the buffer is large enough.The experiment fixes τs=0.1 and τt=0.04 while varying buffer size.Source: §4.3 · pp. 7–8, Table 4 · p. 8

4.4 Visualization of Learned Representations

  • On CIFAR-10, t-SNE visualizations show better class separation for representations learned with ReSSL's relational consistency loss than with MoCov2's contrastive loss.Figure 3 marks classes by color.Source: §4.4 · p. 8, Figure 3 · p. 8

5 Results on Large-scale Datasets

On ImageNet, ReSSL outperforms prior methods in linear evaluation, across multi-crop and smaller-backbone settings, and in VOC transfer. The authors also report competitive training speed alongside improved accuracy.

Linear Evaluation

  • ImageNet experiments use τt = 0.04 and τs = 0.1, selected as the best setting from Table 2.The setup also specifies a 130k memory buffer and a two-layer projection head with 4096 hidden and 512 output dimensions.Source: §5 · p. 8
  • ReSSL significantly outperforms all 1× backprop methods and beats CLSA, SimCLR, and SwAV among 2× methods at lower training cost.The comparison is under linear evaluation on ImageNet with a ResNet-50 backbone after 200 epochs.Source: Table 5 · p. 8, §5 · p. 9

Working with Multi-Crop Strategy

  • With the multi-crop strategy, ReSSL is reported significantly better than previous state-of-the-art methods.The four-crop setting uses resolutions 224, 160, 128, and 96; the five-crop setting adds 192, as in AdCo.Source: §5 · p. 9, Table 6 · p. 9

Working with Smaller Architecture

  • On ResNet-18, ReSSL performs better than SEED without requiring a larger pretrained teacher network.The comparison follows the ResNet-50 training recipe.Source: §5 · p. 9, Table 7 · p. 9

Training Cost

  • ReSSL is 3% slower than MoCo v2 but yields 2.4% improvements; its 4-crop training speed is on par with 2× backprop methods.The authors also report significantly better performance than all state-of-the-art methods for ReSSL (4 crops).Source: §5 · p. 9, Table 8 · p. 9

Transfer Learning

  • ReSSL consistently outperforms MoCo v2 and PCL v2 across labeled-example counts K on PASCAL VOC2007 transfer.The reported mAP averages five runs except for K=full.Source: §5 · p. 9, Table 9 · p. 10

6 Conclusion

ReSSL maintains relational consistency between instances across augmentations, relaxing common contrastive-learning constraints. Experiments on large-scale datasets report efficient training and state-of-the-art performance for unsupervised representation learning.

Conclusion

  • ReSSL maintains relational consistency between instances under different augmentations.It relaxes the constraints that different instances must be pushed apart and augmented views must share exactly the same feature.Source: §6 · p. 10
  • An extensive empirical study examines the effect of each component in ReSSL.Source: §6 · p. 10
  • Experiments on large-scale datasets demonstrate efficient training and state-of-the-art performance for unsupervised representation learning.Source: §6 · p. 10

A Implementation Details on Small and Medium Dataset

The small- and medium-dataset implementation uses different optimizer settings for SimCLR and BYOL than for SimSiam, MoCo, and ReSSL.

Implementation Details

B More Experiments on Temperature

Additional temperature experiments find collapse as τt approaches τs and identify a narrow optimal τt range.

More Experiments on Temperature

C More Experiments on Weak Augmentation

The Tiny ImageNet experiment compares teacher-model augmentations and finds that mildly weak augmentation performs best, with random resized crops and random flip achieving the highest performance.

C More Experiments on Weak Augmentation

  • No teacher augmentation tends to degrade performance, while overly strong augmentation can make the target distribution inaccurate through added noise.The authors suggest that no augmentation may leave too little feature gap between views; mildly weak augmentation is preferred.Source: C More Experiments on Weak Augmentation · p. 13
  • The experiment fixes τt = 0.04 and evaluates teacher-augmentation choices by linear evaluation on Tiny ImageNet.Tested choices include random resized crops, random flip, color jitter, grayscale, Gaussian blur, and combinations.Source: C More Experiments on Weak Augmentation · p. 13
  • Random resized crops with random flip achieve the highest linear-evaluation performance among the tested teacher augmentations on Tiny ImageNet.Table 11 reports the teacher-augmentation comparison.Source: Table 11 · p. 13, C More Experiments on Weak Augmentation · p. 13

Related papers

Loading 2107.09282v2…