PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation

Yixuan Wei, Han Hu, Zhenda Xie, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, Baining Guo

arXiv:2205.14141v3cs.CVcs.LG

TL;DR

Contrastive and image-text pre-training methods generally trail other approaches in downstream fine-tuning, despite MIM’s strong results. The paper uses feature distillation to train new representations from pre-trained ones and reports improved fine-tuning, including contrastive methods competitive with MIM.

  • Problem

    Contrastive and image-text pre-training methods are generally not superior to other methods after downstream fine-tuning, while MIM’s strong fine-tuning performance raises the question of what could make other approaches similarly successful.

    Source: §1 · p. 2

  • Method

    Feature distillation trains new representations from pre-trained models, using feature maps as targets and designs intended to support generic, effective fine-tuning.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    Contrastive methods become competitive with MIM in fine-tuning, and CLIP ViT-L reaches 89.0% top-1 accuracy on ImageNet-1K classification.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The authors argue that improving optimization friendliness through feature distillation can leave more research attention for representation generality and scalability.

    Source: §5 · p. 9

Abstract

from arXiv · show

Masked image modeling (MIM) learns representations with remarkably good fine-tuning performances, overshadowing previous prevalent pre-training approaches such as image classification, instance contrastive learning, and image-text alignment. In this paper, we show that the inferior fine-tuning performance of these pre-training approaches can be significantly improved by a simple post-processing in the form of feature distillation (FD). The feature distillation converts the old representations to new representations that have a few desirable properties just like those representations produced by MIM. These properties, which we aggregately refer to as optimization friendliness, are identified and analyzed by a set of attention- and optimization-related diagnosis tools. With these properties, the new representations show strong fine-tuning performance. Specifically, the contrastive self-supervised learning methods are made as competitive in fine-tuning as the state-of-the-art masked image modeling (MIM) algorithms. The CLIP models' fine-tuning performance is also significantly improved, with a CLIP ViT-L model reaching 89.0% top-1 accuracy on ImageNet-1K classification. On the 3-billion-parameter SwinV2-G model, the fine-tuning accuracy is improved by +1.5 mIoU / +1.1 mAP to 61.4 mIoU / 64.2 mAP on ADE20K semantic segmentation and COCO object detection, respectively, creating new records on both benchmarks. More importantly, our work provides a way for the future research to focus more effort on the generality and scalability of the learnt representations without being pre-occupied with optimization friendliness since it can be enhanced rather easily. The code will be available at https://github.com/SwinTransformer/Feature-Distillation.

1 Introduction

Contrastive and image-text pre-training perform strongly in linear or zero-shot evaluation but generally lag after downstream fine-tuning, while MIM has recently excelled. Feature distillation is proposed to narrow this gap, with diagnostic evidence linking its gains to optimization-friendly representations.

1 Introduction

  • MIM’s fine-tuning success motivates asking which ingredients could make other pre-training approaches similarly successful.Contrastive and image-text methods are generally not superior to other methods after downstream fine-tuning.Source: §1 · p. 2
  • Feature distillation makes contrastive self-supervised methods competitive with MIM in fine-tuning and raises CLIP ViT-L to 89.0% top-1 ImageNet-1K accuracy.The paper also reports improvements across several pre-training families.Source: §1 · p. 2, Table 1 · p. 2
  • For ADE20K semantic segmentation and COCO object detection, the reported results are 61.4 mIoU and 64.2 mAP, respectively, creating new records on both benchmarks.Source: §1 · p. 3
  • Attention and optimization diagnostics indicate that distillation yields more diverse attention heads and greater reliance on relative positions, properties associated with flatter fine-tuning loss landscapes and better final accuracy.The authors characterize these properties collectively as optimization friendliness and report little additional gain from distilling MIM representations, which already exhibit them.Source: §1 · p. 3
  • The authors propose that feature distillation can let representation-learning research focus more on generality and scalability without being preoccupied with optimization friendliness.They identify generality and scalability as important for broad visual tasks and making use of larger models and datasets.Source: §1 · p. 3

2 A Feature Distillation Method

Feature distillation trains a new student representation from an already pre-trained teacher to improve fine-tuning friendliness. The method uses feature-map targets and design choices for generality and effective subsequent fine-tuning.

2 A Feature Distillation Method

  • The pre-trained model acts as teacher and a newly trained model acts as student, with the goal of transferring knowledge into a representation friendlier to fine-tuning.The method is presented as generic and effective through several design choices.Source: §2 · p. 3
  • Distilling the teacher’s full feature map rather than logits or a reduced feature vector supports pre-trained models without logits and yields higher fine-tuning accuracy.Full feature maps perform best among the tested distillation targets.Source: §2 · p. 3
  • The teacher and student use the same augmentation view, and a 1 × 1 convolution aligns their feature-map dimensions.The convolution allows teacher and student outputs to have different dimensions.Source: §2 · p. 3
  • Teacher feature maps are whitened with non-parametric layer normalization without scaling or bias to address differing feature-magnitude orders across pre-trained models.Source: §2 · p. 4
  • A smooth ℓ1 loss compares student and whitened teacher features; β defaults to 2.0, and g is a 1 × 1 convolution.The student and teacher outputs are denoted s and t, respectively.Source: §2 · p. 4
  • Shared relative position bias performs best overall among the tested position-encoding configurations and is used by default.It diversifies attention distances, especially in deeper layers, which may contribute to slightly better fine-tuning accuracy.Source: §2 · p. 4
  • Asymmetric drop path works best on ViT-B with rates of 0.1–0.3 on the student and no drop path regularization on the teacher.Source: §2 · p. 4

3 Representations before and after Feature Distillation

Feature distillation changes attention patterns and optimization landscapes toward properties the paper associates with MIM representations. These diagnostics show more diverse attention after distillation and flatter landscapes for most representations.

Feature distillation diversifies attention heads

  • After distillation, attention distances become more diverse or evenly distributed, especially in deeper layers where pretrained heads previously collapsed into a narrow range.The authors interpret the earlier collapse as heads learning similar visual cues and potentially wasting model capacity.Source: §3 · p. 4, Figure 2 · p. 5, Figure 3 · p. 5

Changes on attention patterns

  • Post-distillation attention maps show more diagonal patterns, suggesting greater reliance on relative patch locations and better translational invariance.With absolute position encoding, the maps also show more relative-location patterns, suggesting this change is primarily due to feature distillation.Source: §3 · pp. 4–5, Figure 4 · p. 5

Feature distillation gets better loss / accuracy landscapes

  • Most representations have flatter loss / accuracy landscapes after feature distillation, consistent with their better fine-tuning accuracy.The landscapes are visualized before and after distillation across five randomly generated directions.Source: §3 · pp. 5–6, Figure 5 · p. 6

On masked image modeling (MIM)

  • MAE [17] already has diverse attention heads and relatively flat landscapes; distillation adds only +0.2% fine-tuning accuracy (83.8% versus 83.6%).The authors suggest feature distillation and masked image modeling may overlap in functionality.Source: §3 · p. 6, Figure 6 · p. 6

4 Experiments

Across the evaluated pretrained models, feature distillation improves fine-tuning results, with strong outcomes on ImageNet-1K, ADE20K, and COCO. Ablations favor full feature maps, teacher-feature whitening, shared relative position bias, and asymmetric drop path rates.

4.1 Experimental Settings

  • Experiments distill on 1.28M ImageNet-1K training images, using 100 epochs for ablations and 300 epochs for system-level comparisons; evaluation includes ImageNet-1K fine-tuning and linear probing, and ADE20K segmentation.The evaluated pretrained methods include DINO, EsViT, CLIP, DeiT, and MAE.Source: §4.1 · p. 6

4.2 Main Results

  • Feature distillation improves all listed pretrained models by 1.0%–2.0% on ImageNet-1K fine-tuning and 1.0–3.3 mIoU on ADE20K.CLIP ViT-L reaches 89.0% on ImageNet-1K classification, +1.9% over a CLIP-specific fine-tuning approach.Source: §4.2 · p. 7
  • 61.4 mIoU on ADE20K and 64.2 mAP on COCO with 3-billion-parameter SwinV2-G set new records, +0.6 mIoU and +0.9 mAP over prior state of the art.The comparison uses the same UPerNet / HTC++ framework and evaluation settings as the original Swin V2 paper.Source: §4.2 · p. 7

4.3 Ablations

  • Full feature maps perform best across pretrained methods; DeiT logit distillation improves only +0.1%, and logits do not apply to models without classification outputs.Table 3 ablates alternative distillation targets.Source: §4.3 · p. 7, Table 3 · p. 8
  • Teacher-feature whitening improves over original features by +0.8%, +0.2%, and +1.0% for CLIP, DINO, and DeiT, and beats ℓ2 normalization by +0.4%, +0.0%, and +1.1%, respectively.Feature normalization also makes hyperparameters less sensitive to the pretrained model.Source: §4.3 · p. 7, Table 4 · p. 8
  • Shared relative position bias performs best overall, though all tested student position-encoding configurations perform well and the authors attribute the primary gains to feature distillation.The shared configuration has the most diverse attention heads, which may contribute to its accuracy.Source: §4.3 · p. 7, Table 5 · p. 8, Figure 7 · p. 8
  • Moderately increasing student drop path helps; optimal rates are 0.1 for CLIP and 0.2 for DINO and DeiT, while teacher drop path often compromises performance.The authors therefore use asymmetric student and teacher drop path rates by default.Source: §4.3 · pp. 7–8, Table 6 · p. 8

5 Discussion in the Context of Related Works

The discussion distinguishes optimization friendliness from the generality and scalability of learned representations, and reviews how different pretraining approaches fare across fine-tuning, linear, few-shot, and zero-shot uses. The authors argue feature distillation mainly improves optimization friendliness, leaving broader representation properties as a focus for future study.

Goals and evaluations of representation learning

  • Representation learning is discussed across fine-tuning, linear evaluation, few-shot learning, and zero-shot or prompt-aided inference, with this paper focused on fine-tuning.Linear evaluation fixes the backbone and trains a linear classifier, primarily reflecting feature linear separability.Source: §5 · pp. 8–9

What are nice properties for fine-tuning?

  • The discussion separates optimization friendliness from encoding generalizable, scalable knowledge; feature distillation mainly targets the former.Optimization friendliness concerns initialization for fine-tuning, while the second property concerns useful downstream knowledge and scaling with data or model size.Source: §5 · p. 9

Existing representation learning approaches and our reflections

  • MIM excels in fine-tuning and has demonstrated model-capacity scalability, but its benefit from larger data is described as negative in cited work.The authors say instance contrastive learning underperforms on fine-tuning with Vision Transformers, while its scalability to model capacity and data size remains unclear.Source: §5 · p. 9
  • Image classification and visual-text alignment are described as scalable in model capacity and data; visual-text data availability is also characterized as favorable for data-hungry training.The authors consider these tasks together because they can be formulated with similar objectives, and suggest they may suit large-scale representation learning.Source: §5 · p. 10
  • The discussion also lists colorization, jigsaw solving, split-brain auto-encoding, rotation prediction, clustering, channel prediction, and pixel-level contrast as approaches worth re-examining.Source: §5 · p. 10

6 Conclusion

Feature distillation generally improves fine-tuning across visual pre-training models and makes contrastive self-supervised methods competitive with state-of-the-art MIM. The analysis suggests the approach mainly improves optimization friendliness.

  • Feature distillation makes contrastive self-supervised methods as competitive in fine-tuning as state-of-the-art masked image modeling.Source: §6 · p. 10
  • A CLIP pre-trained ViT-L reaches 89.0% top-1 accuracy on ImageNet-1K classification after feature distillation.Source: §6 · p. 10
  • The authors’ attention- and optimization-related analyses suggest feature distillation mainly improves the optimization friendliness of learned representations.Source: §6 · p. 10
  • The authors hope the findings help future research focus more on representation generality and scalability.Source: §6 · p. 10

A Average Attention Maps for DINO and DeiT

After feature distillation, DINO and DeiT ViT-B average attention maps show more diagonal patterns, indicating enhanced translational invariance.

B Hyperparameters for Feature Distillation

Table 7 lists hyperparameters for feature distillation on ImageNet-1K.

  • Table 7 lists hyperparameters for feature distillation on ImageNet-1K.Source: Table 7 · p. 15

C Hyperparameters for Fine-tuning

Table 8 lists hyperparameters for fine-tuning on ImageNet-1K.

  • Table 8 lists hyperparameters for fine-tuning on ImageNet-1K.Source: Table 8 · p. 15

D Representation Properties of Masked Image Modeling before and after Distillation

Average attention maps and head similarities show similar behaviors before and after feature distillation, suggesting overlap in functionality between feature distillation and masked image modeling.

D Representation Properties of Masked Image Modeling before and after Distillation

  • The average attention maps and head cosine similarities show similar before-and-after behaviors, suggesting feature distillation and masked image modeling overlap in functionality.The comparison includes MAE ViT-B representations before and after distillation.Source: D Representation Properties of Masked… · p. 16
  • Figure 10 visualizes average attention maps for layers 0–11, ordered from top-left to bottom-right, with image patches indexed from top-left to bottom-right.Source: Figure 10 · p. 16
  • Figure 11 compares average cosine similarity between attention heads per layer before and after feature distillation using MAE ViT-B.Source: Figure 11 · p. 16

E Additional Training Cost by Feature Distillation Compared to that of Pre-training

Feature distillation adds a fixed 300-epoch ImageNet-1K training stage; its overhead can be small relative to costly pre-training, with about +3% cited for CLIP.

E Additional Training Cost by Feature Distillation Compared to that of Pre-training

Related papers

Loading 2205.14141v3…