PaperlayerView exact revision on arXiv

Source-linked AI summary

MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining

Xiaoyi Dong, Jianmin Bao, Yinglin Zheng, Ting Zhang, Dongdong Chen, Hao Yang, Ming Zeng, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, Nenghai Yu

arXiv:2208.12262v2cs.CV

TL;DR

Global image-text contrastive learning can miss fine-grained image information, motivating MaskCLIP to learn local semantics alongside global alignment. MaskCLIP adds masked self-distillation to the vision branch and local mask-language supervision to the text branch, achieving stronger transfer across diverse downstream evaluations. The paper reports gains in zero-shot, retrieval, linear-probing, and finetuning settings.

  • Problem

    Global image-text alignment may focus on text-described objects while overlooking other fine-grained image information useful for downstream tasks.

  • Method

    MaskCLIP incorporates masked self-distillation for local patch representation learning and adds local mask-language supervision to the text encoder.

  • Results

    MaskCLIP shows superior transfer across zero-shot, linear-probing, and finetuning evaluations, including +6.1% average zero-shot accuracy on 20 datasets.

  • Takeaways & Limitations

    Masked self-distillation learns local semantics that complement the global semantics targeted by vision-language contrastive learning.

Abstract

from arXiv · show

This paper presents a simple yet effective framework MaskCLIP, which incorporates a newly proposed masked self-distillation into contrastive language-image pretraining. The core idea of masked self-distillation is to distill representation from a full image to the representation predicted from a masked image. Such incorporation enjoys two vital benefits. First, masked self-distillation targets local patch representation learning, which is complementary to vision-language contrastive focusing on text-related representation. Second, masked self-distillation is also consistent with vision-language contrastive from the perspective of training objective as both utilize the visual encoder for feature aligning, and thus is able to learn local semantics getting indirect supervision from the language. We provide specially designed experiments with a comprehensive analysis to validate the two benefits. Symmetrically, we also introduce the local semantic supervision into the text branch, which further improves the pretraining performance. With extensive experiments, we show that MaskCLIP, when applied to various challenging downstream tasks, achieves superior results in linear probing, finetuning, and zero-shot performance with the guidance of the language encoder. Code will be release at \url{https://github.com/LightDXY/MaskCLIP}.

1. Introduction

MaskCLIP addresses the limitation that global image-text alignment can overlook useful image details by adding masked self-distillation for local semantic representation learning. The framework also extends local semantic supervision to the text branch and reports stronger transfer across zero-shot, linear-probing, and finetuning settings.

  • Motivation: Global vision-language contrastive learning may focus on text-described objects while ignoring other fine-grained image information useful for downstream tasks.The paper motivates learning representations that capture both global semantics and local patches.
  • Method: MaskCLIP randomly masks a large portion of the input image and incorporates masked self-distillation into vision-language contrastive pretraining.The method is designed to force the encoder to focus on remaining visible patches while advancing VL contrastive learning.
  • Method: Local patch representations complement the global representations learned by vision-language contrastive learning and are intended to carry semantic meaning through indirect language supervision.The paper connects masked self-distillation to both local patch learning and semantic alignment with global representations.
  • Method: MaskCLIP adds mask language modeling to the text branch, using tokenized word pieces as prediction targets to provide local semantic supervision.A small text decoder is introduced to reduce conflicts between global image-text contrastive learning and local mask language modeling.
  • Results: +6.1% average zero-shot accuracy on 20 datasets and +17.2%, +12.8% rank@1 improvement on Flickr30K image-text retrieval are reported.The paper also reports 48.9% TOP-1 average accuracy in the Image Classification in the Wild academic track, 3.4% above the second team.
  • Results: MaskCLIP demonstrates superiority across zero-shot, linear-probing, and finetuning evaluations on tens of benchmarks.The reported evaluation spans vision and vision-language tasks, including classification, segmentation, detection, and retrieval.

2. Related Work

Prior work combines CLIP with contrastive or masked-image-modeling objectives, while MaskCLIP compares these alternatives and uses masked self-distillation as its vision-side method. The related-work discussion distinguishes global representation learning from patch-oriented generative or distillation-based learning.

  • Vision-language pretraining: Vision-language pretraining commonly combines cross-modal objectives including image-text matching, masked language modeling, masked image modeling, and contrastive loss.These objectives are often mixed into compound training objectives.
  • Vision self-supervision: Figure 1 contrasts vanilla CLIP with CLIP augmented by contrastive learning, pixel-prediction masked image modeling, or masked self-distillation.The fourth pipeline is identified as MaskCLIP.
  • Vision self-supervision: Contrastive vision self-supervision is described as learning global representations for salient objects, whereas masked image modeling emphasizes local patch representations.The paper uses this distinction to motivate its masked self-distillation approach.
  • Self-knowledge distillation: Self-knowledge distillation uses a temporal ensemble of the student model as its teacher rather than an externally pretrained teacher.The teacher is derived from the student itself.

3. MaskCLIP

MaskCLIP combines vision-language contrastive learning with masked self-distillation to learn transferable visual representations, while adding local semantic supervision to the text branch. Its visual pipeline aligns global image-text features and predicts masked visual patches from visible ones using an EMA teacher.

  • Backbone and vision-language alignment: MaskCLIP uses a ViT image encoder and Transformer text encoder to obtain visual and linguistic features for pretraining.The visual encoder produces feature tokens, while the text encoder extracts features from paired text.
  • Backbone and vision-language alignment: Global image and text representations are projected into metric embeddings and aligned with image-text contrastive loss.The loss combines image-to-text and text-to-image directions over image-text pairs in a training mini-batch.
  • Masked self-distillation: Masked self-distillation uses an EMA teacher on the full image and a student on visible patches to predict representations for masked regions.The student inserts learnable mask-token features, restores positional embeddings, and applies a decoder to predict masked-region features.
  • Masked self-distillation: An online quantizer converts teacher and predicted features into soft codeword distributions, which are matched through cross-entropy distillation.The teacher quantizer is also updated with exponential moving averages of the online quantizer.
  • Local semantic learning for text: The text branch applies masked language modeling to tokenized word pieces and adds a small decoder to separate local prediction from global contrastive prediction.The decoder is introduced to reduce output conflict between global image-text contrastive learning and local mask language modeling.
  • Joint pretraining objective: MaskCLIP combines the contrastive, visual self-distillation, and text masking losses, training all visual and text components from scratch.The weighting between vision-language contrastive loss and self-supervised learning loss is controlled by hyper-parameters λ and β.

4. Experiments

MaskCLIP is evaluated through controlled self-supervision analyses and broad vision and vision-language benchmarks. Across zero-shot, linear-probing, and finetuning settings, the method improves transfer performance while masked objectives support local patch representations.

  • Analysis: All added vision self-supervision improves CLIP, while MaskCLIP achieves the best results across the reported ImageNet-1K and Flickr30K metrics.MaskCLIP improves ImageNet-1K zero-shot, linear probing, and finetuning by +6.9%, +7.2%, and +1.3%, respectively, and improves Flickr30K retrieval by +17.2% and +12.8%.
  • Analysis: Additional-view contrastive learning uses more GPU memory and training time because CLIP+SimCLR compares two extra image views.
  • Analysis: Masked image modeling improves annotation-free zero-shot segmentation over CLIP and CLIP+SimCLR, supporting its value for local patch representations.The experiment evaluates dense prediction on ADE20K and Pascal Context without training segmentation-specific classifiers.
  • Comparison with Previous Methods: MaskCLIP combines vision-language pretraining with image mask self-distillation and reports strong ImageNet-1K classification, ADE20K segmentation, and MS-COCO detection and segmentation results.The comparison includes supervised, self-supervised, and vision-language baselines across these downstream tasks.
  • Ablations: Removing either the mask language modeling loss or distillation loss reduces performance, while shallow visual and text decoders work better.The ablations report that both objectives are necessary; deeper decoders worsen performance across the evaluated metrics.

5. Conclusion

MaskCLIP combines masked self-distillation with VL contrastive learning to learn complementary local and global semantics, while mask language modeling strengthens the text encoder. Its visual encoder shows strong transfer capability across linear probing, fine-tuning, and zero-shot evaluation.

  • MaskCLIP incorporates masked self-distillation into vision-language contrastive pretraining.
  • Masked self-distillation learns local semantics that complement the global semantics targeted by VL contrastive learning.
  • Mask language modeling enhances the text encoder and is critical for zero-shot performance.
  • The resulting visual encoder transfers strongly across linear probing, fine-tuning, and zero-shot evaluation.

A. More Experiment

Additional experiments evaluate MaskCLIP with smaller models and datasets and isolate components of its distillation loss. The results show competitive zero-shot performance and benefits from each major distillation component.

  • Comparison over small model and small dataset: MaskCLIP outperforms the combination of DeCLIP and FILIP in zero-shot ImageNet-1K classification with a ViT-B/32 model.
  • Comparison over small model and small dataset: The smaller-model experiment uses ViT-B/32, while the smaller-dataset experiment evaluates MaskCLIP on CC3M with ViT-B/16.
  • Ablation on distillation loss: Using the feature as the prediction target improves all metrics in the distillation-loss ablation.
  • Ablation on distillation loss: The EMA model produces better performance, and the MLM loss improves all vision-language tasks.

B. Experiment detail

The experiments use large-scale pretraining with random image masking and evaluate zero-shot, linear-probing, fine-tuning, and dense prediction under specified protocols. The ICinW setup additionally combines three pretraining datasets and uses longer training with a larger batch size.

  • Pre-training: MaskCLIP pretraining runs for 25 epochs with batch size 4096, 75% random masking, and EMA weight 0.999.
  • Pre-training: The ICinW academic-track experiment uses YFCC-15M, GCC3M+12M, and ImageNet-21K, excluding ImageNet-1K data.
  • Pre-training: For ICinW, training lasts 32 epochs with batch size 16384, while the remaining settings match the YFCC15M setup.
  • Zero-shot ImageNet-1K classification: Zero-shot ImageNet-1K classification averages features from seven prompt templates and compares image features with all label features.
  • Linear-probing ImageNet-1K classification: Linear probing freezes the backbone and trains a new classifier for 90 epochs with batch normalization before classification.
  • Fine-tuning ImageNet-1K classification: ImageNet-1K fine-tuning uses average-pooled final-transformer outputs, 100 training epochs, and a layer-decayed learning rate factor of 0.7.
  • Semantic segmentation and detection: Dense prediction uses 512 × 512 inputs, specified augmentations, stochastic depth 0.1, and single-scale testing.

C. More visualization results.

Additional MS-COCO visualizations examine image-text feature alignment. MaskCLIP achieves better alignment performance in most cases.

  • MaskCLIP achieves better image-text feature alignment performance in most visualized MS-COCO validation examples.

D. Societal impacts

MaskCLIP inherits CLIP’s societal impacts, including both malicious uses and positive applications. Its Internet-sourced training data may also introduce unwanted bias.

  • MaskCLIP inherits CLIP’s societal impacts, including malicious usages and positive applications.
  • The framework may suffer from unwanted data bias because its training data are roughly collected from the Internet.
  • The paper frames MaskCLIP’s societal-impact profile as an improvement of CLIP rather than a separate category of impacts.
Loading 2208.12262v2…