PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Simple Siamese Representation Learning

Xinlei Chen, Kaiming He

arXiv:2011.10566v1cs.CVcs.LG

TL;DR

Preventing collapse is a central challenge in Siamese representation learning, where constant outputs are an undesired solution. SimSiam tests whether a simple Siamese network can learn useful representations without negative pairs, large batches, or momentum encoders, and achieves competitive ImageNet and downstream-task results.

  • Problem

    Siamese representation learning must avoid trivial constant-output collapse, motivating strategies such as negative pairs and clustering.

  • Method

    SimSiam directly maximizes similarity between two views using shared weights, neither negative pairs nor a momentum encoder, and relies critically on stop-gradient.

  • Results

    SimSiam achieves competitive ImageNet and downstream-task performance, with the highest accuracy among compared methods after 100-epoch pre-training and better results than SimCLR in all cases.

  • Takeaways & Limitations

    The results suggest that Siamese architecture itself may be a core reason related methods work and can provide an inductive bias for modeling invariance.

  • Takeaways & Limitations

    The proposed optimization hypothesis does not explain why collapse is prevented, so SimSiam’s non-collapsing behavior remains an empirical observation.

Abstract

from arXiv · show

Siamese networks have become a common structure in various recent models for unsupervised visual representation learning. These models maximize the similarity between two augmentations of one image, subject to certain conditions for avoiding collapsing solutions. In this paper, we report surprising empirical results that simple Siamese networks can learn meaningful representations even using none of the following: (i) negative sample pairs, (ii) large batches, (iii) momentum encoders. Our experiments show that collapsing solutions do exist for the loss and structure, but a stop-gradient operation plays an essential role in preventing collapsing. We provide a hypothesis on the implication of stop-gradient, and further show proof-of-concept experiments verifying it. Our "SimSiam" method achieves competitive results on ImageNet and downstream tasks. We hope this simple baseline will motivate people to rethink the roles of Siamese architectures for unsupervised representation learning. Code will be made available.

1. Introduction

SimSiam shows that a simple Siamese network can learn useful visual representations without negative pairs, large batches, or a momentum encoder. Its stop-gradient operation is critical for avoiding collapse, motivating an alternating-optimization hypothesis and highlighting Siamese weight sharing as a useful invariance-inducing bias.

  • SimSiam: SimSiam maximizes similarity between two augmented views of one image without negative pairs, large-batch training, or a momentum encoder.The architecture applies a prediction MLP on one branch and stop-gradient on the other.
  • Relation to prior methods: SimSiam relates to BYOL, SimCLR, and SwAV by removing one core component from each method.It is described as BYOL without the momentum encoder, SimCLR without negative pairs, and SwAV without online clustering.
  • Collapse prevention: Stop-gradient is critical because collapsing solutions exist for the underlying Siamese loss and structure.The paper reports that removing this operation permits collapse, while its presence prevents such solutions.
  • Optimization hypothesis: The importance of stop-gradient motivates a hypothesis that SimSiam implicitly alternates optimization over two sets of variables.Proof-of-concept experiments are provided to verify this hypothesis.
  • Inductive bias: Weight-sharing Siamese networks can provide an inductive bias for modeling invariance between different observations of the same concept.The paper compares this role with convolutional weight sharing for translation invariance.

2. Related Work

Prior self-supervised Siamese methods commonly use negative samples, large batches, memory banks, queues, or momentum encoders, while clustering methods alternate between assignments and representation learning. BYOL predicts one view from another using a momentum-encoder branch, whereas this paper challenges the necessity of that encoder and identifies stop-gradient as critical.

  • Contrastive learning: Contrastive learning attracts positive pairs and repels negative pairs, with Siamese networks providing simple and effective implementations for self-supervised representation learning.
  • Contrastive learning: Contrastive methods often depend on many negative samples supplied by memory banks, queues, or large batches; MoCo uses a momentum encoder, whereas SimCLR uses in-batch negatives.
  • Clustering: Clustering-based methods alternate between clustering representations and predicting cluster assignments, with SwAV computing assignments from one view and predicting them from another.
  • Clustering: Clustering methods can treat cluster centers as negative prototypes and require memory banks, large batches, or queues to provide enough samples.
  • BYOL: BYOL predicts one view’s output from another using a Siamese network with a momentum-encoder branch, but the paper finds stop-gradient critical and questions the encoder’s necessity.

3. Method

SimSiam processes two augmented views with a shared encoder and prediction head, optimizing a symmetrized negative-cosine-similarity loss. Stop-gradient is essential to the method, which uses standard SGD-based pre-training and evaluates frozen representations with linear classification on ImageNet.

  • Architecture: The architecture applies a shared encoder f to two randomly augmented views and uses a prediction MLP h to match one view’s output to the other.The encoder combines a backbone such as ResNet with a projection MLP; p1 = h(f(x1)) and z2 = f(x2).
  • Stop-gradient: Stop-gradient treats each target z as constant in its corresponding term, so each encoder branch receives gradients from its prediction term but not from that target.The implemented loss is 1/2 D(p1, stopgrad(z2)) + 1/2 D(p2, stopgrad(z1)).
  • Baseline settings: The default pre-training setup uses SGD with base lr = 0.05, weight decay 0.0001, momentum 0.9, batch size 512, synchronized BN, and a cosine-decay learning-rate schedule.The method does not require a large-batch optimizer such as LARS, and the default backbone is ResNet-50.
  • Experimental setup: Representations are pre-trained without labels on ImageNet’s 1000-class training set and evaluated by training a supervised linear classifier on frozen representations before validation testing.This is the stated common protocol for assessing the quality of pre-trained representations.

4. Empirical Study

The empirical study finds that stop-gradient is central to preventing collapse in SimSiam, while predictor design, batch size, batch normalization, similarity function, and symmetrization mainly affect optimization or accuracy. SimSiam remains effective across broad settings, achieving 67.7% linear accuracy with stop-gradient versus 0.1% without it.

  • Stop-gradient: Without stop-gradient, training reaches the minimum loss of −1 and the normalized outputs collapse to a constant vector.With stop-gradient, outputs remain scattered on the unit hypersphere and the kNN accuracy improves steadily.
  • Stop-gradient: 67.7% linear evaluation accuracy is achieved with stop-gradient, whereas removing only stop-gradient reduces accuracy to 0.1%.The 0.1% result is chance-level on ImageNet, although chance-level accuracy alone does not establish collapse.
  • Predictor MLP: The predictor h must be trainable and adaptive: removing it fails, fixing it at random initialization fails to converge, while constant learning rate can outperform the baseline.The predictor should adapt to the latest representations rather than being forced to converge before representation learning is sufficiently advanced.
  • Batch size: 0.8% and 2.0% accuracy drops occur at batch sizes 128 and 64, while results from 256 to 2048 differ only at random-variation levels.The method works across batch sizes from 64 to 4096; the lower 4096 result is attributed to standard SGD rather than collapse prevention.
  • Batch normalization: 34.6% accuracy without head BN improves to 67.4% with hidden-layer BN, 68.1% with output BN, and 68.2% when output-BN affine parameters are disabled.Adding BN to the prediction-MLP output causes unstable, oscillating training rather than collapse; BN helps optimization but not collapse prevention.
  • Other design choices: Cross-entropy can converge without collapse, and symmetrization boosts accuracy but is not related to collapse prevention.The study concludes that batch size, batch normalization, similarity function, and symmetrization may affect accuracy, whereas stop-gradient plays the essential role in preventing collapse.

5. Hypothesis

The paper hypothesizes that SimSiam implements an EM-like alternating optimization over network parameters and per-image representation variables, with stop-gradient arising naturally from this formulation. Proof-of-concept experiments support the formulation and relate the predictor to approximating augmentation expectations, while leaving non-collapse unexplained.

  • Hypothesis: SimSiam is hypothesized to implement an Expectation-Maximization-like algorithm with two variable sets and two underlying sub-problems.The extra variable set is presented as the consequence of introducing stop-gradient.
  • Hypothesis: The additional variables ηx represent per-image representations, while θ denotes encoder parameters analogous to clustering centers and ηx to k-means assignment vectors.The joint optimization over θ and η is described as analogous to k-means clustering.
  • Alternating optimization: Stop-gradient follows naturally when SGD updates θ while the previous ηt−1 remains constant in the corresponding sub-problem.The gradient does not back-propagate to ηt−1 during optimization of θ.
  • One-step alternation: One-step alternation with a single sampled augmentation and one SGD update approximates the SimSiam algorithm and produces its Siamese architecture with stop-gradient.The random augmentation supplies another view, while the one-step SGD reduction approaches SimSiam.
  • Proof-of-concept experiments: All multi-step variants work well, and the 10-/100-step variants outperform SimSiam at the cost of extra precomputation, supporting alternating optimization as a valid formulation.The 1-step variant is equivalent to SimSiam, which is therefore a special case of the formulation.
  • Expectation over augmentations: 55.0% accuracy is achieved without predictor h when a moving average approximates multiple augmentation views, whereas removing h without that moving average fails completely.This experiment supports linking predictor h to approximation of the augmentation expectation ET[·].
  • Limitations: The hypothesis does not explain why collapse is prevented; non-collapsing behavior remains an empirical observation.The authors suggest initialization and the alternating trajectory may make convergence to constant ηx difficult, but identify this as an open question.

6. Comparisons

SimSiam achieves competitive ImageNet and transfer-learning results despite using neither negative samples nor a momentum encoder. Comparisons with SimCLR, SwAV, and BYOL clarify how its simplified Siamese design relates to their additional components.

  • ImageNet: SimSiam uses batch size 256 without negative samples or a momentum encoder, yet achieves the highest accuracy after 100 pre-training epochs and outperforms SimCLR throughout.Training longer produces a smaller gain for SimSiam.
  • Transfer Learning: SimSiam representations transfer beyond ImageNet and remain competitive across VOC detection, COCO detection, and COCO instance segmentation.An optimal recipe with lr=0.5 and wd=1e-5 improves results across all transfer tasks while maintaining similar ImageNet accuracy.
  • Transfer Learning: All compared methods can surpass or match supervised ImageNet pre-training across transfer tasks, suggesting Siamese structure as a common factor in their general success.The comparison emphasizes shared Siamese structure despite substantial design differences.
  • Relation to SimCLR: SimSiam can be viewed as SimCLR without negatives, while adding a predictor and stop-gradient to SimCLR is neither necessary nor helpful in the reported ablation.The paper attributes these extra SimCLR components to a different underlying optimization problem rather than contrastive learning itself.
  • Relation to SwAV: SimSiam is conceptually SwAV without online clustering; removing stop-gradient from SwAV causes divergence, whereas adding a predictor does not help.SwAV’s Sinkhorn-Knopp transform derives from balanced online clustering that can prevent collapse, but SimSiam does not use it.
  • Relation to BYOL: SimSiam can be viewed as BYOL without the momentum encoder, which may improve accuracy but is not necessary to prevent collapsing.The paper proposes that the relevant η sub-problem may instead be solved with other optimizers, potentially yielding temporally smoother updates.

7. Conclusion

The study finds that a simple Siamese design remains competitive, suggesting that the Siamese architecture itself may be central to recent methods’ effectiveness. It emphasizes Siamese networks as natural tools for modeling invariance in representation learning.

  • 7. Conclusion: A minimalist Siamese method achieves competitive performance, suggesting the Siamese shape may drive recent methods’ effectiveness.The study argues that this competitiveness warrants attention to the architecture’s fundamental role.
  • 7. Conclusion: Siamese networks provide a natural and effective structure for modeling invariance in representation learning.Modeling invariance is identified as a central focus of representation learning.
  • 7. Conclusion: The study calls attention to the fundamental role of Siamese networks in representation learning.The authors hope their findings will attract broader community attention to this role.

A. Implementation Details

The implementation follows established unsupervised pre-training practices, with specified geometric, color, and blurring augmentations. It also details initialization, weight decay, and frozen-feature linear evaluation settings.

  • Unsupervised pre-training: Unsupervised pre-training follows practices established by prior work.
  • Data augmentation: Data augmentation combines RandomResizedCrop, RandomHorizontalFlip, ColorJitter, RandomGrayscale, and Gaussian blurring with specified ranges and probabilities.RandomResizedCrop uses scale [0.2, 1.0]; ColorJitter strengths are {0.4, 0.4, 0.4, 0.1} with probability 0.8; RandomGrayscale uses probability 0.2; blurring uses Gaussian std [0.1, 2.0].
  • Initialization: Convolutional and fully connected layers use default PyTorch initialization, while substantially different fully connected initialization may prevent convergence.The last batch-normalization layer in every residual block has scale parameters initialized to 0.
  • Weight decay: Weight decay is 0.0001 for all parameter layers, including batch-normalization scales and biases, unlike implementations that exclude them.
  • Linear evaluation: Linear evaluation trains a supervised classifier on frozen ResNet pool5 features using cosine-decayed learning rate, 90 epochs, batch size 4096, and LARS.The setup uses base lr = 0.02, weight decay = 0, and momentum = 0.9; an SGD alternative with batch size 256 gives ∼1% lower accuracy.

B. Additional Ablations on ImageNet

On ImageNet, SimSiam benefits from increasing output dimension until saturation at d = 2048, while its bottleneck prediction MLP improves robustness and training stability. The authors recommend keeping the hidden dimension at one quarter of the output dimension.

  • Output dimension: SimSiam benefits from larger output dimensions and saturates at d = 2048, unlike existing methods that saturate at 256 or 512.The comparison is reported for existing methods [17] [8] [15].
  • Prediction MLP bottleneck: The prediction MLP uses a hidden dimension equal to 1/4 of the output dimension, and this bottleneck structure is more robust.Setting hidden and output dimensions equal can make training less stable or cause failure in some explored variants.
  • Prediction MLP bottleneck: The bottleneck may behave like an auto-encoder that forces the predictor to digest information, so the authors recommend using it.This is presented as a hypothesis explaining the bottleneck’s robustness.

C. Reproducing Related Methods

The comparison in Table 4 uses faithful reproductions of related methods, with small modifications improving SimCLR, MoCo v2, and SwAV. These changes add a third projection-MLP layer for SimCLR and SwAV and symmetrize MoCo v2’s loss.

  • C. Reproducing Related Methods: The authors re-implement related methods as faithfully as possible for the Table 4 comparison.The comparison is explicitly based on reproductions following each individual paper.
  • C. Reproducing Related Methods: Small modifications improve SimCLR, MoCo v2, and SwAV.SimCLR and SwAV use 3 projection-MLP layers instead of 2, while MoCo v2 uses a symmetrized rather than asymmetric loss.

D. CIFAR Experiments

On CIFAR-10, SimSiam learns reasonable representations without collapsing and performs slightly better than SimCLR under the same training setting.

  • Results: SimSiam achieves a reasonable CIFAR-10 result without collapsing, with training curves similar to SimCLR under the same setting.The comparison uses kNN classification accuracy during pre-training and linear evaluation accuracy.
  • Results: 0.7%: SimSiam is slightly better than SimCLR under the same CIFAR-10 training setting.Training uses CIFAR ResNet-18 with SGD for 800 epochs and batch size 512.
Loading 2011.10566v1…