PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring the Equivalence of Siamese Self-Supervised Learning via A Unified Gradient Framework

Chenxin Tao, Honghui Wang, Xizhou Zhu, Jiahua Dong, Shiji Song, Gao Huang, Jifeng Dai

arXiv:2112.05141v3cs.CV

TL;DR

Self-supervised methods use different objectives and architectures, making their relationships and performance comparisons difficult to understand. This paper uses gradient analysis to unify three method families and compares them under a fair experimental setting. The methods show similar performance, momentum encoders are identified as key to performance, and UniGrad is competitive across reported tasks.

  • Problem

    Different self-supervised methods use distinct loss functions, networks, and training tricks, leaving their relationships and performance differences difficult to explain and compare.

    Source: §1 · p. 2

  • Method

    The paper analyzes gradients to unify contrastive, asymmetric-network, and feature-decorrelation methods, then proposes UniGrad, a gradient formula without a memory bank or asymmetric network.

    Source: §1 · p. 2

  • Results

    The methods achieve close performance under fair comparison, with momentum encoder use identified as the key factor; UniGrad is reported effective in linear evaluation and downstream tasks.

    Source: §1 · p. 2, §6 · pp. 8–9

  • Takeaways & Limitations

    The shared gradient structure suggests these methods work through a similar mechanism despite organizing their loss functions differently.

    Source: §1 · p. 2, §6 · p. 8

    Performance comparison uses only linear evaluation, leaving transfer-learning comparisons for object detection and semantic segmentation to future work.

    Source: §6 · p. 9

Abstract

from arXiv · show

Self-supervised learning has shown its great potential to extract powerful visual representations without human annotations. Various works are proposed to deal with self-supervised learning from different perspectives: (1) contrastive learning methods (e.g., MoCo, SimCLR) utilize both positive and negative samples to guide the training direction; (2) asymmetric network methods (e.g., BYOL, SimSiam) get rid of negative samples via the introduction of a predictor network and the stop-gradient operation; (3) feature decorrelation methods (e.g., Barlow Twins, VICReg) instead aim to reduce the redundancy between feature dimensions. These methods appear to be quite different in the designed loss functions from various motivations. The final accuracy numbers also vary, where different networks and tricks are utilized in different works. In this work, we demonstrate that these methods can be unified into the same form. Instead of comparing their loss functions, we derive a unified formula through gradient analysis. Furthermore, we conduct fair and detailed experiments to compare their performances. It turns out that there is little gap between these methods, and the use of momentum encoder is the key factor to boost performance. From this unified framework, we propose UniGrad, a simple but effective gradient form for self-supervised learning. It does not require a memory bank or a predictor network, but can still achieve state-of-the-art performance and easily adopt other training strategies. Extensive experiments on linear evaluation and many downstream tasks also show its effectiveness. Code is released at https://github.com/fundamentalvision/UniGrad.

1. Introduction

The introduction asks how apparently different siamese self-supervised learning methods relate and what drives performance differences. It presents gradient analysis as a unifying explanation, reports similar performance across methods with momentum encoders identified as key, and introduces UniGrad.

1. Introduction

  • Contrastive methods use positive and negative views, asymmetric methods use a predictor and stop-gradient, and feature-decorrelation methods reduce redundancy between dimensions.The introduction notes that differing networks and training tricks make the methods difficult to compare fairly.Source: §1 · p. 2
  • The introduction frames the open questions as how the methods relate, whether their mechanisms connect, and what factors cause performance differences.The methods appear distinct in their loss functions and are evaluated with different networks and tricks.Source: §1 · p. 2
  • Different gradient formulas yield close performance, while the use of a momentum encoder is identified as the key factor affecting performance.The paper also reports that UniGrad is competitive across linear evaluation, semi-supervised learning, and downstream vision tasks.Source: §1 · p. 2
  • Gradient analysis unifies contrastive, asymmetric-network, and feature-decorrelation methods as similar forms with positive and negative gradients balanced by a scalar.The positive term represents another augmented view of the same image; the negative term is a weighted combination of representations from different images.Source: §1 · p. 2
  • UniGrad is a concise gradient formula that maximizes positive-sample similarity and expects negative-sample similarity to be zero.It requires neither a memory bank nor an asymmetric network, and can adopt CutMix [32] and multi-crop [3] [4].Source: §1 · p. 2

2. Related Work

Related work describes contrastive, asymmetric-network, and feature-decorrelation approaches to self-supervised learning. The paper connects their gradient structures, interpreting the predictor in asymmetric methods as encoding previous-sample information that serves as a negative gradient.

2. Related Work

  • Contrastive methods pull positive samples together and push negative samples apart, producing alignment and uniformity on the representation hypersphere [29].MoCos [6] [8] [16] use a memory bank and momentum encoder for consistent negatives, while SimCLR [5] uses strong augmentations and a nonlinear projection head.Source: §2 · p. 2
  • Asymmetric methods use positive pairs and asymmetric architecture to avoid collapse; BYOL adds a predictor and momentum target, while SimSiam [7] stops target-branch gradients.The paper notes that the mechanism by which asymmetric networks avoid collapse remains poorly understood [27].Source: §2 · pp. 2–3
  • The paper connects asymmetric networks to contrastive learning by showing that the predictor encodes previous-sample information in its weights, serving as a negative gradient during back-propagation.This yields a gradient structure similar to contrastive learning.Source: §2 · p. 3
  • Feature-decorrelation methods reduce redundancy, and the paper shows their gradient formulas can be transformed into positive- and negative-sample terms like those in contrastive learning.Examples include Barlow Twins [33], which drives cross-correlation toward the identity, and VICReg [1], which uses a variance-invariance-covariance principle.Source: §2 · p. 3

3. A Unified Framework for SSL

The framework compares siamese self-supervised methods through their gradients rather than their loss functions, expressing them in terms of positive and negative gradients and a balance factor. It connects contrastive, asymmetric-network, and feature-decorrelation methods through this shared structure.

3. A Unified Framework for SSL

  • A siamese framework uses online and target branches, with the target branch serving as the online branch’s training target; methods differ in target-branch type and sample-set construction.The target can share weights, stop gradients, or use a momentum encoder; the sample set may be the current batch, a memory bank, or all previous samples.Source: §3 · p. 3

3.1. Contrastive Learning Methods

  • Contrastive gradients combine a positive term that pulls same-image views together and a weighted negative term that pushes other samples apart.The target-branch type and construction of the contrastive sample set distinguish methods in this family.Source: §3.1 · p. 4

3.2. Asymmetric Network Methods

  • Asymmetric-network gradients also contain positive and negative components, with the predictor’s optimization supplying the negative component through encoded feature-correlation information.The predictor’s eigenspace may align with the feature-correlation matrix, whose encoded information contributes to the back-propagation direction.Source: §3.2 · p. 4

3.3. Feature Decorrelation Methods

  • Feature-decorrelation gradients share the same positive-and-negative structure, with the two components arising from diagonal and off-diagonal correlation-matrix elements.The balance factor weights the negative gradient; methods differ in the negative coefficient’s subscript.Source: §3.3 · p. 5

4. Key Factors in SSL

Under a unified training and evaluation setting, the paper compares gradient forms and target-branch types. The methods have minor performance gaps, while momentum encoders improve results across method families.

4.1. Gradient Form

  • <0.5% points: different methods have only a minor performance gap, and the construction of sample set V is not vital for self-supervised learning.The comparison uses unified gradient forms with a shared positive gradient and searched balance factor; methods differ in their negative gradient.Source: Table 2 · p. 6, §4.1 · pp. 6–7
  • MoCo and SimCLR show nearly no final-result difference, and using batch negatives instead of memory-bank negatives makes only a minor difference.The paper notes that, with proper training settings, larger numbers of negative samples may not be necessary for good performance.Source: §4.1 · p. 6
  • Simplifications preserve performance: the asymmetric-network form uses an identity positive-gradient matrix and constant balance factor, while reported Barlow Twins and VICReg simplifications show no accuracy decrease or similar results.The compared changes include replacing Barlow Twins’ positive-gradient matrix with identity and batch normalization with ℓ2 normalization, and simplifying VICReg’s negative gradient.Source: §4.1 · p. 6

4.2. Target Branch Type

  • ~2% points: momentum-encoder targets improve all method families over stop-gradient, and applying momentum only to the positive gradient is enough.The authors suggest a consistent, slowly updating positive goal may be important for self-supervised learning.Source: Table 3 · p. 7, §4.2 · p. 7

5. A Concise Gradient Form for SSL

UniGrad presents a concise gradient form for self-supervised learning and examines its behavior with common augmentations and longer training. The section compares its learning dynamics and performance with other methods.

5.1. UniGrad

  • UniGrad’s gradient combines attraction to the other augmented view with a correlation-matrix term; its objective maximizes positive-view cosine similarity and pushes negative-view similarity toward zero.The objective’s second term is intended to avoid representational collapse.Source: §5.1 · p. 7
  • Unlike InfoNCE in MoCo and SimCLR, UniGrad drives negative-sample similarity toward zero rather than making it lower than positive-sample similarity.A correlation matrix lets UniGrad encode infinitely many negative samples with less memory cost than a memory bank.Source: §5.1 · p. 7
  • UniGrad learns meaningful representations without the predictor used by BYOL and SimSiam, avoiding predictor-related optimization tricks and design influence.It also removes the need for SVD decomposition used by DirectPred.Source: §5.1 · p. 7
  • UniGrad can decorrelate feature channels similarly to Barlow Twins and VICReg without directly optimizing covariance or cross-correlation matrices.It uses ℓ2 normalization rather than batch normalization or extra per-channel variance restrictions.Source: §5.1 · p. 7
  • Across methods, Figure 2 shows no significant difference in positive/negative similarity, k-NN accuracy, or feature decorrelation trajectories.The principal-component-ratio metric counts how many PCA eigenvalues are needed for their cumulative sum to exceed 90%.Source: §5.1 · p. 7, Figure 2 · p. 8

5.2. Application on Data Augmentations

  • CutMix and multi-crop improve performance by 0.9% and 1.4%, respectively, and combining them yields a 2.0% improvement.CutMix mixes positive gradients from normal images according to the mixup ratio; multi-crop uses 2 global and 6 local views, with F calculated from global views only. Multi-crop has relatively heavy training cost, so variants can be chosen according to available compute.Source: §5.2 · p. 8, Table 4 · p. 8
  • With longer training, UniGrad with CutMix surpasses methods without multi-crop; adding multi-crop yields performance comparable to current state-of-the-art methods, with competitive downstream results.The reported transfer tasks include semi-supervised ImageNet and object detection on PASCAL VOC and COCO.Source: §5.2 · p. 8, Table 5 · p. 8

6. Conclusion

The paper unifies three self-supervised learning method families through gradient analysis and reports similar performance, with the momentum encoder identified as a key performance factor. UniGrad is effective in linear evaluation and downstream tasks, while the paper limits its performance comparison to linear evaluation.

6. Conclusion

  • The methods share a similar gradient form and deliver similar performance, with the momentum encoder identified as the key factor affecting performance.The shared form comprises positive and negative gradients and a balance factor.Source: §6 · p. 8
  • UniGrad is a simple gradient form that showed effectiveness in linear evaluation and downstream tasks.Source: §6 · p. 9
  • The performance comparison uses only linear evaluation, and the authors leave transfer-learning comparisons for future work.They note that methods may affect downstream tasks such as object detection and semantic segmentation differently.Source: §6 · p. 9
  • The work may inherit SSL-related societal risks, including energy use and pollution from large-scale training and bias in learned representations.Source: §6 · p. 9

A.1. Contrastive Learning Methods

The appendix derives gradients for MoCo and SimCLR in terms of positive-view attraction and weighted negative-sample contributions. It also describes a simplified SimCLR gradient and reports that this simplification does not harm performance.

A.1. Contrastive Learning Methods

  • MoCo’s per-sample gradient combines a positive-view term with a weighted sum of memory-bank representations.The negative weights are computed over the memory bank, and the gradient is scaled by the temperature and batch size.Source: §A.1 · p. 11
  • SimCLR backpropagates through its target branch, so the online representation receives gradients from other-view and negative-sample losses.Source: §A.1 · p. 11
  • The simplified SimCLR gradient has a MoCo-like structure, and the authors report that this simplification does not harm performance.The simplification is described after removing the second gradient term; the implementation uses a weighted combination of positive and negative terms.Source: §A.1 · p. 11

A.2. Asymmtric Network Methods

The appendix analyzes asymmetric-network gradients and their simplifications. It reports that removing a predictor-gradient term does not reduce performance and that the dynamic balance factor can be replaced by a constant because it tends to remain stable.

A.2. Asymmtric Network Methods

  • DirectPred’s gradient includes a transformed positive-view term and additional terms involving the feature-correlation matrix.The authors report that removing the F^1/2 term does not cause a performance drop.Source: §A.2 · pp. 11–12
  • The dynamic balance factor λ tends to remain stable, so the authors say it can be replaced with a constant scalar.Source: §A.2 · p. 12
  • The asymmetric-network pseudocode forms a moving average of the correlation matrix and uses it in the negative term.The pseudocode also constructs the predictor transformation from the correlation matrix’s eigendecomposition.Source: §A.2 · p. 12

A.3. Feature Decorrelation Methods

Feature-decorrelation methods include Barlow Twins, which drives the cross-correlation matrix toward identity, and VICReg, whose loss combines invariance, covariance, and variance terms. The appendix also provides UniGrad pseudocode with positive and negative terms.

A.3. Feature Decorrelation Methods

  • Barlow Twins drives the cross-correlation matrix toward identity, with its loss expressed using a diagonal weighting matrix.Source: §A.3 · pp. 12–13
  • VICReg combines invariance, covariance, and variance loss terms, with the variance term penalizing standard deviations below a threshold.Source: §A.3 · p. 13

A.4. Pseudocode of UniGrad

  • UniGrad’s pseudocode combines a detached positive-view term with a correlation-matrix-based negative term, weighted by λ.Source: §A.4 · p. 14

B. Implementation Details

The paper uses shared hyperparameters for different loss functions in 100-epoch pretraining. It also specifies a separate 800-epoch setup and a frozen-feature linear evaluation protocol.

B. Implementation Details

  • The 100-epoch pretraining setup follows [7], while the 800-epoch setup follows and uses a larger batch size for faster training.When a momentum encoder is used, its exponential moving average parameter increases from 0.996 to 1 with a cosine schedule.Source: B. Implementation Details · p. 14
  • For 100-epoch pretraining, the setup uses SGD, batch size 1024, ResNet50, and a three-layer projector, with hyperparameters shared across loss functions.The learning rate follows linear scaling from a 0.05 base rate, with cosine decay and five warmup epochs; weight decay is 1.0 × 10−4 and momentum is 0.9.Source: B. Implementation Details · p. 14, Table 7 · p. 14
  • For 800-epoch pretraining, the setup uses LARS, batch size 4096, and a three-layer projector with hidden and output dimensions of 8192.It uses a 0.3 base learning rate for weights and 0.05 for biases and batch-normalization parameters, with cosine decay after ten warmup epochs.Source: B. Implementation Details · p. 14
  • Linear evaluation trains a supervised linear classifier on frozen pretrained features for 90 epochs using LARS and batch size 4096.The protocol uses zero weight decay, momentum 0.9, base learning rate 0.02, and cosine decay.Source: B. Implementation Details · p. 14

C. Additional Results

Additional experiments show that projector depth and width affect linear-evaluation performance, while UniGrad obtains competitive semi-supervised and transfer results. The section also records the access and licensing terms for the evaluated datasets.

C.1. Projector Structure

  • Increasing projector depth from one to three layers greatly improves linear-evaluation accuracy, but further depth increases yield saturating improvement.The comparison uses UniGrad with 100-epoch pretraining; Table 9 reports projector-depth effects.Source: §C.1 · p. 14, Table 9 · p. 15
  • Increasing projector width steadily improves performance, with no apparent saturation up to dimension 16384.The study changes hidden and output dimensions together; Table 10 reports projector-width effects.Source: §C.1 · p. 14, Table 10 · p. 15

C.2. Semi-supervised Learning

  • UniGrad obtains results comparable with [28] and reports 5% and 1% improvements over other methods on ImageNet’s 1% and 10% subsets, respectively.The pretrained model is fine-tuned on the two training-set subsets; Table 11 reports the semi-supervised comparison.Source: §C.2 · p. 14, Table 11 · p. 15

C.3. Transfer Learning

  • UniGrad delivers competitive transfer performance against other self-supervised methods and surpasses the supervised baseline on PASCAL VOC and COCO tasks.The evaluation covers object detection on VOC and object detection and instance segmentation on COCO; Table 8 identifies the benchmarks and model configurations.Source: §C.3 · p. 15, Table 8 · p. 15

D. Licenses of Assets

  • The listed assets have distinct access or licensing terms: ImageNet follows its terms of access, VOC images are subject to Flickr terms, and COCO annotations use CC BY 4.0.COCO images are also subject to Flickr terms of use.Source: D. Licenses of Assets · p. 15

Related papers

Loading 2112.05141v3…