PaperlayerView exact revision on arXiv

Source-linked AI summary

Siamese Image Modeling for Self-Supervised Vision Representation Learning

Chenxin Tao, Xizhou Zhu, Weijie Su, Gao Huang, Bin Li, Jie Zhou, Yu Qiao, Xiaogang Wang, Jifeng Dai

arXiv:2206.01204v3cs.CV

TL;DR

ID lacks spatial sensitivity, while MIM lacks good semantic alignment. SiameseIM predicts dense representations of one augmented view from another masked view, combining the two properties. It surpasses both approaches across downstream tasks, with stronger reported gains in few-shot, long-tail, and robustness-concerned settings.

  • Problem

    ID's global representations miss intra-image structure, while MIM's within-image reconstruction does not align semantically similar images well.

    Source: §1 · pp. 1–2

  • Method

    SiameseIM predicts one view's dense representations from another masked, differently augmented view, using relative positions and a dense loss.

    Source: §3.2 · p. 5, §3.3 · p. 5, §3.4 · p. 6

  • Results

    SiameseIM surpasses MIM and ID across downstream tasks, with larger reported gains on ADE20k (about 3 points) and LVIS rare classes (about 1.6 points).

    Source: §1 · p. 3, §5 · p. 8

  • Takeaways & Limitations

    The paper reports that a single dense loss can learn semantic alignment and spatial sensitivity together.

    Source: §1 · p. 3, §5 · p. 8

    Large-scale pretraining requires substantial computational resources and may consume much electricity; biases in the training data are also possible, so the method should be used with caution.

    Source: §5 · pp. 8–9

Abstract

from arXiv · show

Self-supervised learning (SSL) has delivered superior performance on a variety of downstream vision tasks. Two main-stream SSL frameworks have been proposed, i.e., Instance Discrimination (ID) and Masked Image Modeling (MIM). ID pulls together representations from different views of the same image, while avoiding feature collapse. It lacks spatial sensitivity, which requires modeling the local structure within each image. On the other hand, MIM reconstructs the original content given a masked image. It instead does not have good semantic alignment, which requires projecting semantically similar views into nearby representations. To address this dilemma, we observe that (1) semantic alignment can be achieved by matching different image views with strong augmentations; (2) spatial sensitivity can benefit from predicting dense representations with masked images. Driven by these analysis, we propose Siamese Image Modeling (SiameseIM), which predicts the dense representations of an augmented view, based on another masked view from the same image but with different augmentations. SiameseIM uses a Siamese network with two branches. The online branch encodes the first view, and predicts the second view's representation according to the relative positions between these two views. The target branch produces the target by encoding the second view. SiameseIM can surpass both ID and MIM on a wide range of downstream tasks, including ImageNet finetuning and linear probing, COCO and LVIS detection, and ADE20k semantic segmentation. The improvement is more significant in few-shot, long-tail and robustness-concerned scenarios. Code shall be released at https://github.com/fundamentalvision/Siamese-Image-Modeling.

1. Introduction

Instance Discrimination (ID) and Masked Image Modeling (MIM) each capture one representation property but leave a gap in the other. SiameseIM combines matching augmented views with dense prediction from masked views, and surpasses both approaches across downstream tasks.

1. Introduction

  • ID matches augmented views and avoids collapse, but its global representations miss intra-image structure; MIM models local structure but has weak semantic alignment and linear separability.The introduction describes ID as poorly suited to dense prediction and MIM as weak in few-shot classification.Source: §1 · pp. 1–2
  • Matching different augmented views supports semantic alignment, while predicting dense representations from masked images supports spatial sensitivity.The paper's motivation combines these two ingredients in one model.Source: §1 · p. 2, Figure 1 · p. 2
  • SiameseIM predicts dense representations of one augmented view from another masked, differently augmented view of the same image.Its online branch encodes the first view and predicts the second view's representations using relative positions; a target branch encodes the second view.Source: §1 · p. 2
  • SiameseIM surpasses both MIM and ID across downstream tasks, with larger gains on ADE20k (about 3 points) and LVIS rare classes (about 1.6 points).The reported evaluations include ImageNet fine-tuning, few-shot learning and linear probing, COCO and LVIS detection, ADE20k segmentation, and robustness benchmarks.Source: §1 · pp. 2–3, Table 1 · p. 2
  • SiameseIM uses a single dense loss to learn semantic alignment and spatial sensitivity, with stronger improvements reported in few-shot, long-tail, and robustness-concerned settings.The contribution claim describes the two properties as learned together using only a single dense loss.Source: §1 · p. 3

2. Related work

Prior ID and MIM approaches offer complementary strengths, while earlier attempts to combine global and dense objectives retain a trade-off. SiameseIM instead predicts another augmented view with dense supervision and relative-position correspondence.

2. Related work

  • ID pulls together augmented views and supports linear separability, but its dense prediction performance on Vision Transformers can lag supervised pretraining.The related-work discussion notes this limitation particularly for object detection.Source: §2 · p. 3
  • Earlier ID methods add dense supervision through several correspondence techniques, but their focus or reliance on an extra global loss leaves a trade-off between semantic alignment and spatial sensitivity.The cited techniques include Earth Mover’s Distance, feature similarity, nearest neighbors, and region-proposal or flow modules.Source: §2 · p. 3
  • SiameseIM uses relative positions to align dense correspondence between views, and reports a considerable improvement over original ID methods on a strong detection baseline.The approach is presented as using only a dense loss to learn semantic alignment and spatial sensitivity together.Source: §2 · p. 3, Figure 2 · p. 4
  • MIM reconstructs masked-image content and can excel in full fine-tuning, but its representations are less linearly separable and less data-efficient in few-shot settings.The discussion cites BEiT and MAE as surpassing other paradigms on detection given enough training epochs.Source: §2 · p. 3
  • SiameseIM predicts another augmented view rather than the original view, which the paper says strengthens semantic alignment and can surpass ID in linear probing.The paper presents this as a way for MIM to produce representations with adequate linear separability.Source: §2 · pp. 3–4

3. Method

SiameseIM predicts dense features of one augmented view from another masked view, using separate online and target branches. Its design combines different views, spatially informed prediction, and a dense loss.

3. Method

  • SiameseIM uses an online encoder-decoder to predict the second view’s dense features and a momentum target encoder to produce target features.The online backbone is retained for downstream evaluation.Source: §3 · p. 4, §3.2 · p. 5

3.1. Augmented Inputs

  • The method feeds different augmented views to the two branches, applies spatial and color augmentations, and masks the online view.The paper reports that color augmentation provides more invariance with different views, but not with the same view.Source: §3.1 · p. 4

3.2. Prediction Targets

  • The decoder predicts xb’s dense features from xa’s visible patches and mask tokens, using relative positions; an EMA target encoder encodes xb, whose positional embedding includes scale changes.Feature prediction is preferred to raw-pixel prediction for different views.Source: §3.2 · p. 5

3.3. Loss Function

  • SiameseIM uses only a dense UniGrad loss, treating each token as a sample; UniGrad uses O(D^2) memory rather than InfoNCE’s O(|N|).The paper states that a dense loss alone can train semantic alignment and spatial sensitivity.Source: §3.3 · pp. 5–6

3.4. Discussion

  • Reconstructing a differently augmented view rather than the same view can enhance semantic alignment, and strong augmentations benefit this learning process.Source: §3.4 · p. 6
  • Dense supervision improves spatial sensitivity while supporting semantic alignment; relative positions provide strict spatial alignment and dense correspondence.SiameseIM enforces similarity between different views at the dense level, unlike prior combination attempts [46] [57] that use both global and dense losses.Source: §3.4 · p. 6

4. Experiments

The experiments evaluate SiameseIM across classification, detection, segmentation, and robustness tasks, then ablate its design choices. Results and ablations examine both cross-task performance and factors affecting semantic alignment and spatial sensitivity.

4.1. Implementation Details

  • Experiments use ViT-B/16 with MoCo-v3 augmentations and, by default, BEiT blockwise masking and MAE-like loss normalization.The default loss applies LayerNorm without affine parameters to the target and no normalization to the prediction.Source: §4.1 · p. 6

4.2. Main Results

  • Across ImageNet, COCO, ADE20k, LVIS, and four robustness datasets, SiameseIM reports gains over competing methods, including 1.6 points on LVIS rare classes.It is on par with MAE on overall LVIS AP; reported robustness gains average 4.5 over MoCo-v3 and 6.1 over MAE.Source: §4.2 · pp. 6–7, Table 2 · p. 7

4.3. Ablation Study

  • Different views improve linear probing by ∼11 points, and feature targets outperform pixel targets when input and target views differ.For the same view, pixel prediction performs better than feature prediction.Source: §4.3 · p. 7
  • With different views, color augmentations add 3.5 points to linear probing, while BatchNorm gives slightly better linear-probing and dense-prediction results.MAE-like loss normalization performs better on linear probing and comparably on detection than MoCo-like normalization.Source: §4.3 · pp. 7–8
  • In the reported comparison, adopting dense loss in SiameseIM improved object detection by 2.8 points and instance segmentation by 2.3 points, supporting dense representations for dense prediction.The authors think blockwise masking continuous patches makes feature reconstruction harder and forces long-range dependency capture.Source: §4.3 · p. 8

5. Conclusion

SiameseIM combines semantic alignment and spatial sensitivity by predicting dense representations of an augmented view from another masked view of the same image. The paper reports that it outperforms ID and MIM methods across downstream tasks, while noting higher training cost and broader SSL resource and bias concerns.

5. Conclusion

  • SiameseIM predicts dense representations of an augmented view from another masked augmented view of the same image, combining semantic alignment and spatial sensitivity in one dense loss.The paper identifies ID with semantic alignment and MIM with spatial sensitivity.Source: §5 · p. 8
  • SiameseIM outperforms ID and MIM methods over a wide range of downstream tasks.Source: §5 · p. 8
  • SiameseIM training is less efficient than MAE, and the authors expect to pursue more efficient pretraining in future work.Source: §5 · p. 8
  • Large-scale SSL pretraining requires substantial computational resources and may consume considerable electricity.Source: §5 · pp. 8–9
  • SSL data may contain biases, so the authors advise using the method with caution.Source: §5 · p. 9

A.1. Pre-training

Pretraining follows MAE's setting, with strong augmentations and blockwise masking. Both branches use ViT-B/16, with separate projector and decoder stacks and an EMA target encoder.

A.1. Pre-training

  • Pretraining follows the MAE setting, with hyperparameters listed in Table 4.Source: §A.1 · p. 11, Table 4 · p. 11
  • The method uses MoCo-v3 strong augmentations and BEiT-style blockwise masking at a 60% masking ratio.Augmentations include random resized cropping, horizontal flipping, color jittering, grayscale conversion, Gaussian blurring, and solarization.Source: §A.1 · p. 11
  • Both branches use ViT-B/16; the projector and decoder stack two and four Transformer blocks, respectively, with 768-dimensional embeddings and 12 heads per block.Source: §A.1 · p. 11
  • The target branch uses an EMA momentum encoder initialized at 0.995 and scheduled to reach 1.0; target features receive LayerNorm without affine parameters, while predictions are unnormalized.Source: §A.1 · p. 11

A.2. Finetuning with 100% Data

The appendix specifies ImageNet finetuning and linear-probing protocols, including their learning-rate search choices. The corresponding hyperparameter tables are identified, but their values are not included in the supplied passages.

A.2. Finetuning with 100% Data

  • ImageNet finetuning follows MAE's setting except that the optimal learning rate is searched, with other hyperparameters listed in Table 5.Source: §A.2 · p. 11, Table 5 · p. 11

A.3. Linear Probing

  • Linear probing follows MAE's setting, searches for the optimal learning rate, and adds a BatchNorm layer without affine transformation before the final linear classifier.Source: §A.3 · p. 11

A.4. Finetuning with 1% Data

The few-shot evaluation freezes the backbone and trains an l2-regularized logistic regression classifier on extracted image representations.

A.4. Finetuning with 1% Data

  • Few-shot evaluation freezes the backbone, extracts image representations, and applies l2-regularized logistic regression using cyanure.For MAE, the reported result uses partial finetuning because it outperforms training only a linear classifier.Source: §A.4 · p. 11

A.5. COCO Detection

The appendix describes evaluation settings for COCO detection, ADE20k semantic segmentation, and LVIS detection. It points to hyperparameter tables for each task and adjusts the detection learning-rate schedule when performance saturates.

A.5. COCO Detection

  • COCO evaluation follows [33] on COCO [35], with the learning rate dropped when performance saturates; hyperparameters are listed in Table 7.Table 7 is identified as the COCO detection hyperparameter table.Source: §A.5 · p. 11, Table 7 · p. 12

A.6. Semantic Segmentation

  • ADE20k segmentation uses UperNet [52] with mmsegmentation, changing only the pretrained backbone; hyperparameters are listed in Table 8.The setup follows [4].Source: §A.6 · p. 12, Table 8 · p. 12

A.7. LVIS Detection

  • LVIS evaluation follows [33] and drops the learning rate when performance saturates; hyperparameters are listed in Table 9.Table 9 is identified as the LVIS detection hyperparameter table.Source: Table 9 · p. 12, §A.7 · p. 13

A.8. Robustness benchmarks

Robustness evaluation finetunes on original ImageNet and tests multiple validation sets without further finetuning. The appendix also records dataset access and content notes for ImageNet and COCO 2017.

A.8. Robustness benchmarks

  • The model is finetuned on original ImageNet, then tested on different validation sets without further finetuning.The setting is specified in Table 5.Source: §A.8 · p. 13

B. Attribution of Assets

  • ImageNet access is governed by its terms of access, while COCO 2017 is publicly available under CC BY 4.0; the authors report no known personally identifiable information or offensive content.Source: B. Attribution of Assets · p. 13

Related papers

Loading 2206.01204v3…