PaperlayerView exact revision on arXiv

Source-linked AI summary

What Do Self-Supervised Vision Transformers Learn?

Namuk Park, Wonjae Kim, Byeongho Heo, Taekyung Kim, Sangdoo Yun

arXiv:2305.00729v1cs.CVcs.AIcs.LG

TL;DR

CL and MIM have different downstream strengths, but what each teaches Vision Transformers and why they differ is not well understood. The paper compares their attention, representations, and important layers, and finds contrasting patterns across these properties and tasks. It also reports that a simple combination can outperform either method in fine-tuning and linear probing.

  • Problem

    CL and MIM differ in downstream performance, but little is known about what they teach Vision Transformers and why their performance differs.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    The paper compares CL- and MIM-trained ViTs through analyses of attention behavior, representation properties, and layer importance, and tests a combined objective.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    CL favors global, low-frequency, shape-related representations and later layers, while MIM favors token-level, high-frequency, texture-related information and early layers; their downstream strengths differ by task and model regime.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A linear combination of CL and MIM objectives can outperform either method in fine-tuning and linear probing accuracy.

    Source: §1 · p. 2

Abstract

from arXiv · show

We present a comparative study on how and why contrastive learning (CL) and masked image modeling (MIM) differ in their representations and in their performance of downstream tasks. In particular, we demonstrate that self-supervised Vision Transformers (ViTs) have the following properties: (1) CL trains self-attentions to capture longer-range global patterns than MIM, such as the shape of an object, especially in the later layers of the ViT architecture. This CL property helps ViTs linearly separate images in their representation spaces. However, it also makes the self-attentions collapse into homogeneity for all query tokens and heads. Such homogeneity of self-attention reduces the diversity of representations, worsening scalability and dense prediction performance. (2) CL utilizes the low-frequency signals of the representations, but MIM utilizes high-frequencies. Since low- and high-frequency information respectively represent shapes and textures, CL is more shape-oriented and MIM more texture-oriented. (3) CL plays a crucial role in the later layers, while MIM mainly focuses on the early layers. Upon these analyses, we find that CL and MIM can complement each other and observe that even the simplest harmonization can help leverage the advantages of both methods. The code is available at https://github.com/naver-ai/cl-vs-mim.

1 INTRODUCTION

Contrastive learning (CL) uses global image-level projections, while masked image modeling (MIM) reconstructs masked patch semantics. The paper asks what these methods teach Vision Transformers and finds that their representations and task strengths differ.

1 INTRODUCTION

  • CL leads linear probing and small-model classification, while MIM leads fine-tuning, large-model scaling, and dense prediction.Figure 1 summarizes the task and model-regime comparisons.Source: Figure 1 · p. 2, §1 · p. 2
  • CL captures global relationships and object shapes, but homogeneous later-layer attention maps may reduce local interaction diversity; MIM captures more local relationships.The paper links CL’s global patterns to recognizing and distinguishing images, while noting that the attention collapse can impair preservation of local information.Source: §1 · pp. 1–2
  • CL primarily uses low-frequency signals and is shape-oriented, whereas MIM uses high-frequency signals and is texture-oriented.The paper describes CL as collecting object-shape information across tokens and MIM as preserving and amplifying token-level information.Source: §1 · p. 2
  • Later layers play a key role for CL, while early layers play a key role for MIM.The paper relates early layers to low-level texture and later layers to global shape information.Source: §1 · p. 2
  • A linear combination of CL and MIM objectives can outperform either method alone in fine-tuning and linear probing accuracy.The paper presents this simple combination as a way to leverage advantages of both methods.Source: §1 · p. 2

2 HOW DO SELF-ATTENTIONS BEHAVE?

CL self-attentions capture global relationships and object shapes, especially in later layers, while MIM focuses on local relationships. CL’s attention maps become homogeneous across query tokens, reducing representational diversity and helping explain differences in downstream performance.

2 HOW DO SELF-ATTENTIONS BEHAVE?

  • CL attention distances and effective receptive fields are more global than MIM’s, especially in later layers, capturing shape information that can help distinguish objects.MIM mainly captures local relationships and may have difficulty recognizing whole objects and their shapes.Source: §2 · p. 3, Figure 3 · p. 3
  • CL attention maps become nearly identical across query tokens and heads, whereas MIM maps are more faithful to their queries.CL has significantly lower normalized mutual information in later layers; low mutual information indicates maps are less dependent on query tokens.Source: §2 · pp. 3–4
  • In later layers, CL representations are more similar than MIM’s, and increasing CL model size, head count, or depth does little to improve diversity.Increasing depth adds redundant modules, while increasing heads hardly improves CL’s representational diversity.Source: §2 · p. 4, Figure 5 · p. 4
  • CL’s shape-focused representations align with its linear-probing advantage, while MIM performs better on large-model fine-tuning and dense prediction.The paper attributes CL’s dense-prediction weakness to token features that are homogeneous across spatial coordinates.Source: §2 · pp. 2, 4
  • Restricting self-attention to local receptive fields lowers linear-probing accuracy but raises fine-tuning accuracy.The authors report this as consistent with their observations about CL and MIM.Source: §2 · p. 5

3 HOW ARE REPRESENTATIONS TRANSFORMED?

CL transforms representations toward image-level separation, while MIM changes individual tokens; their representation spaces also emphasize different spatial frequencies. These differences align with CL’s shape bias and MIM’s texture bias.

3 HOW ARE REPRESENTATIONS TRANSFORMED?

  • CL’s homogeneous self-attentions translate tokens in unison, yet move image-representation centers apart, making images linearly separable while distinguishing tokens less.The nearly constant additions do not increase token distances or representation volume.Source: §3 · p. 5
  • MIM assigns different self-attentions to individual tokens, altering their distances and the volume of a single image’s token representations.Source: §3 · p. 5
  • Token spectra hardly grow or can decrease with CL but increase with MIM, whereas CL produces larger image-level representation volume than MIM.The study uses singular-value spectra to measure representation-distribution volume.Source: §3 · pp. 5–6, Figure 7 · p. 6
  • CL uses lower-frequency spatial information associated with global structures and shapes, while MIM uses higher frequencies associated with narrow structures and fine textures.Source: §3 · p. 6, Figure 9 · p. 7
  • MIM’s later layers reduce high frequencies despite attending to local areas, which the authors conjecture reflects an implicit encoder-decoder structure.They contrast this with CL, where the last layer has linearly separable information.Source: §3 · p. 7, Figure 9 · p. 7
  • Stylized ImageNet and high-frequency-noise tests support a shape bias for CL and a texture bias for MIM, with CL more robust to high-frequency noise.The authors also report that this distinction persists after fine-tuning.Source: §3 · p. 7, Figure 8 · p. 7

4 WHICH COMPONENTS PLAY AN IMPORTANT ROLE?

Intermediate-layer probing shows that MIM leads early and CL leads late, while MIM’s accuracy declines in its later layers. The section relates this pattern to decoder behavior and contrasts SimMIM’s implicit decoder with MAE’s explicit decoder.

4 WHICH COMPONENTS PLAY AN IMPORTANT ROLE?

  • MIM has higher linear-probing accuracy early, but CL outperforms MIM late; MIM’s accuracy also falls toward the end, and its highest accuracy remains below CL’s.Source: §4 · p. 8, Figure 11 · p. 8
  • The authors attribute MIM’s late-layer decline to shallow-head methods such as SimMIM using backbone layers as an implicit decoder; a deep self-attention decoder such as MAE may help linear probing.Source: §4 · p. 8
  • MAE’s explicit decoder has higher mutual information than SimMIM’s later encoder layers and prioritizes low frequencies, while later MAE encoder layers preserve high frequencies.The decoder reconstructs masked tokens based on neighborhood tokens; the reported analysis uses a masking ratio of zero.Source: §4 · p. 8, Figure 10 · p. 8

5 ARE THE TWO METHODS COMPLEMENTARY TO EACH OTHER?

The paper tests complementarity by linearly combining CL and MIM losses. The hybrid outperforms either method in linear probing and fine-tuning, with early-layer behavior resembling MIM and later-layer behavior resembling CL.

5 ARE THE TWO METHODS COMPLEMENTARY TO EACH OTHER?

  • A linear combination of CL and MIM losses outperforms vanilla CL and MIM in both ImageNet linear probing and fine-tuning.For the analyzed hybrid with λ = 0.2, early layers show MIM-like query-dependent attention and high-frequency use, while later layers show CL-like homogeneous attention and low-frequency use.Source: §5 · p. 9

6 CONCLUSION

The conclusion synthesizes opposing CL and MIM properties and reports that a combined model can outperform either method. It also identifies alternative combinations, other ViT architectures, and strengthening each method’s characteristic properties as future directions.

6 CONCLUSION

  • The study finds opposing CL and MIM properties in image-versus-token information, low-versus-high frequencies, and the importance of later-versus-early layers; a combined model can outperform both.Source: §6 · p. 9
  • Future directions include applying CL in later layers and MIM in early layers, extending analyses to multi-stage ViTs, and enhancing shape or texture learning in the respective methods.Source: §6 · p. 9

A SETUP

  • Experiments use ViT-B and some ViT-S/L models; ViT-L uses one-quarter the batch size of other models due to memory limits.The appendix lists training settings in Table A.1 and reports experiments on NVIDIA A100 GPUs.Source: A SETUP · p. 13, Table A.1 · p. 13

B RELATED WORK

The related work contrasts image-level contrastive learning with token-level masked image modeling and reviews evidence that their properties can complement each other. It also situates the paper's analysis among prior contrastive and masking approaches.

B RELATED WORK

  • CL compares global projections of two random views; negative samples help prevent collapse but can cause dimensional collapse and require large batches or memory queues.The analysis focuses on MoCo v3, which includes global projection, random views, and negative samples.Source: B RELATED WORK · p. 13
  • Prior work links feature distillation to improved CL fine-tuning through diversified attention ranges, while this paper compares CL and MIM rather than analyzing distillation schemes.Source: B RELATED WORK · p. 13
  • Across the compared methods, CL has longer attention distance and MIM higher normalized mutual information; MoCo's later-layer attention maps are homogeneous across model sizes, unlike SimMIM's local attention.Source: Figure C.1 · p. 14, Figure C.2 · p. 14
  • MIM reconstructs masked tokens, with SimMIM and MAE presented as simple, effective approaches whose downstream performance has drawn attention.Source: B RELATED WORK · p. 14
  • Prior results suggest image-level and token-level self-supervision can be harmonized; SimMIM's local self-attention also aligns with earlier observations.Source: B RELATED WORK · p. 14

C OUR INSIGHTS ARE GENERALIZABLE TO VARIOUS MODELS

The appendix checks whether the reported attention patterns extend beyond the main ViT-B MoCo and SimMIM analysis. It reports consistent properties across other self-supervised methods and ViT sizes.

C OUR INSIGHTS ARE GENERALIZABLE TO VARIOUS MODELS

D LOCALITY INDUCTIVE BIAS IMPROVES FINE-TUNING ACCURACY OF CL

The section tests whether restricting CL self-attention's receptive field can inject locality. The reported trade-off is lower linear-probing accuracy but higher fine-tuning accuracy as the attention kernel shrinks.

D LOCALITY INDUCTIVE BIAS IMPROVES FINE-TUNING ACCURACY OF CL

E A CLOSER LOOK AT THE ROLE OF SELF-SUPERVISED VIT LAYERS

The closer look finds distinct layer- and token-level behaviors in MoCo and SimMIM, including an unusual early MoCo layer and outlier attention heads. Fine-tuned models retain similar patterns, with later SimMIM attention as a stated exception.

E A CLOSER LOOK AT THE ROLE OF SELF-SUPERVISED VIT LAYERS

  • Most MoCo layers capture global patterns with weak query-token correlation, while its third layer is a local, query-dependent exception.Source: E A CLOSER LOOK AT THE ROLE OF… · pp. 15–16
  • MoCo's third layer has unusually high head-wise mutual-information variance, including outlier heads near 1.0 that raise its average; SimMIM's variance stays consistent across depth.Source: E A CLOSER LOOK AT THE ROLE OF… · p. 16
  • MoCo transforms tokens together and separates representations at the image level, whereas SimMIM transforms tokens individually and separates them at the token level.Source: E A CLOSER LOOK AT THE ROLE OF… · p. 16
  • MoCo's token representations separate images and classes into compact clusters, while SimMIM separates tokens but leaves images or classes harder to distinguish linearly.The t-SNE view covers 3,528 tokens from 18 images across three classes.Source: E A CLOSER LOOK AT THE ROLE OF… · p. 16, Figure E.2 · p. 16
  • MoCo's first layer aggregates tokens into compact clusters and reduces representation volume at both token and image levels; the authors conjecture it acts like an embedding component.Source: E A CLOSER LOOK AT THE ROLE OF… · p. 16, Figure E.3 · p. 17
  • Fine-tuned models retain the pretrained attention and frequency patterns, although SimMIM's later-layer mutual information drops as supervised or fine-tuned layers capture global information.Source: Figure F.1 · p. 17

F FINE-TUNED MODELS INHERIT THE PROPERTIES OF PRE-TRAINED MODELS

Fine-tuned models retain key properties of their pre-trained counterparts, including differences in attention, frequency use, and important layers. Some differences are smaller after fine-tuning, and the layer-initialization experiment has single-run limitations.

F FINE-TUNED MODELS INHERIT THE PROPERTIES OF PRE-TRAINED MODELS

  • Fine-tuned MoCo remains relatively shape-biased and robust to high-frequency noise, while SimMIM remains texture-biased, but the gap is smaller than under linear probing.The fine-tuning comparison evaluates ImageNet models on Stylized-ImageNet and measures accuracy drops under frequency-based random noise.Source: Figure F.2 · p. 18, F FINE-TUNED MODELS INHERIT THE… · p. 18
  • Fine-tuned MoCo self-attention maps generally have lower mutual information than SimMIM's; later SimMIM layers shift toward low-frequency information, reducing the gap.The attention and Fourier analyses show the same overall pattern as in pre-trained models, but the differences are smaller after fine-tuning.Source: F FINE-TUNED MODELS INHERIT THE… · pp. 17–18
  • Initializing early SimMIM blocks significantly harms fine-tuning accuracy, whereas initializing its later blocks does not significantly harm it, highlighting early-layer importance.The experiment measures fine-tuning accuracy drops after initializing selected pre-trained blocks.Source: F FINE-TUNED MODELS INHERIT THE… · pp. 18–19, Figure F.3 · p. 18
  • The layer-initialization accuracy-drop experiment uses a single run and identical fine-tuning settings, so method-specific settings may provide further insight.At the first initialization depth in Figure F.3b, MoCo's accuracy drop is marginally higher than SimMIM's.Source: F FINE-TUNED MODELS INHERIT THE… · p. 19

G HYBRID MODELS OUTPERFORM CL AND MIM IN DOWNSTREAM TASKS

The paper tests whether CL and MIM complement each other beyond ImageNet. Its hybrid model outperforms both standalone methods on classification and semantic segmentation evaluations.

G HYBRID MODELS OUTPERFORM CL AND MIM IN DOWNSTREAM TASKS

Related papers

Loading 2305.00729v1…