PaperlayerView exact revision on arXiv

Source-linked AI summary

CLIP-KD: An Empirical Study of CLIP Model Distillation

Chuanguang Yang, Zhulin An, Libo Huang, Junyu Bi, Xinqiang Yu, Han Yang, Boyu Diao, Yongjun Xu

arXiv:2307.12732v2cs.CV

TL;DR

Small CLIP models have received limited study as distillation targets, despite their relevance to resource-constrained applications. This paper tests several ways to distill them from larger CLIP teachers, finding consistent improvements and strong results from feature mimicry and interactive contrastive learning.

  • Problem

    Few previous works explore improving small CLIP models for resource-constrained applications, and TinyCLIP’s weight inheritance requires matching teacher and student architecture styles.

    Source: §1 · p. 1

  • Method

    CLIP-KD studies relation, feature, gradient, and contrastive distillation, using teacher-derived targets for mimicry and joint teacher–student contrastive learning.

    Source: §1 · p. 1

  • Results

    CLIP-KD improves student CLIP models consistently, and the ViT-B/16 student gains 20.5% zero-shot ImageNet accuracy over its baseline.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The study offers guidelines for applying CLIP-KD and identifies future CLIP compression research as an area for further attention.

    Source: §5 · p. 8

    A possible capacity gap between a large teacher and small student may make alignment difficult; the larger ViT-L/14 teacher underperformed ViT-B/16 for distillation in the reported comparison.

    Source: §4.3.4 · p. 7

Abstract

from arXiv · show

Contrastive Language-Image Pre-training (CLIP) has become a promising language-supervised visual pre-training framework. This paper aims to distill small CLIP models supervised by a large teacher CLIP model. We propose several distillation strategies, including relation, feature, gradient and contrastive paradigms, to examine the effectiveness of CLIP-Knowledge Distillation (KD). We show that a simple feature mimicry with Mean Squared Error loss works surprisingly well. Moreover, interactive contrastive learning across teacher and student encoders is also effective in performance improvement. We explain that the success of CLIP-KD can be attributed to maximizing the feature similarity between teacher and student. The unified method is applied to distill several student models trained on CC3M+12M. CLIP-KD improves student CLIP models consistently over zero-shot ImageNet classification and cross-modal retrieval benchmarks. When using ViT-L/14 pretrained on Laion-400M as the teacher, CLIP-KD achieves 57.5\% and 55.4\% zero-shot top-1 ImageNet accuracy over ViT-B/16 and ResNet-50, surpassing the original CLIP without KD by 20.5\% and 20.1\% margins, respectively. Our code is released on https://github.com/winycg/CLIP-KD.

1. Introduction

The paper studies how to improve small CLIP models through distillation from larger CLIP teachers, without requiring teacher and student architectures to match. It evaluates several distillation strategies and reports improvements across student models and benchmarks.

1. Introduction

  • The paper addresses limited exploration of small CLIP models for resource-constrained applications and TinyCLIP’s requirement for matching teacher–student architecture styles.CLIP-KD studies relation, feature, gradient, and contrastive paradigms without relying on architectural cues.Source: §1 · pp. 1–2
  • The proposed strategies use teacher-derived image-to-text relationships, image and text features, and gradients for mimicry, alongside joint teacher–student contrastive learning.The student can serve as an anchor for contrasting teacher embeddings, or student and teacher features can be aggregated for CLIP training.Source: §1 · pp. 1–2
  • The study trains on Conceptual datasets and evaluates zero-shot ImageNet classification and cross-modal retrieval on MSCOCO and Flickr.The unified method is applied to student networks with different architectures.Source: §1 · pp. 1–2
  • CLIP-KD improves student models across zero-shot classification and cross-modal retrieval, with feature mimicry using MSE performing best and interactive contrastive learning second-best.The study attributes differing distillation performance to how much teacher–student feature similarity is maximized.Source: §1 · p. 2

2. Related Works

Related work covers language-supervised visual learning, CLIP variants that alter supervision or input masking, and multimodal knowledge distillation. The paper distinguishes its focus on compressing small CLIP models from those approaches.

Language-Supervised Learning

  • CLIP uses contrastive learning over image–text pairs, while related language-supervised visual learning also includes larger noisy-text training and generative approaches.The paper’s focus is distilling small CLIP models.Source: §2 · p. 2

CLIP Variants

  • CLIP variants improve performance or efficiency through auxiliary visual self-supervision, mask self-distillation, data-efficient supervision, or image masking.The paper contrasts these approaches with CLIP compression using knowledge distillation.Source: §2 · p. 2

Multi-Modal Knowledge Distillation

  • Multimodal distillation includes aligning teacher and student attention distributions and feature maps; TinyCLIP uses affinity mimicking and weight inheritance.TinyCLIP’s weight inheritance requires the same architecture style, whereas CLIP-KD is described as adaptable to any architecture pair.Source: §2 · p. 2

3.1. A Brief Review of CLIP

CLIP maps paired images and text to normalized feature embeddings and uses contrastive learning to bring paired embeddings together while separating unpaired ones. Its loss aligns image-to-text and text-to-image distributions.

2. CLIP framework

  • CLIP maps each image and text in a dataset to feature embeddings using separate visual and text encoders, then applies L2 normalization.The image embedding is denoted v_k and the text embedding s_k.Source: §3.1 · p. 2
  • CLIP uses an InfoNCE-based contrastive loss to maximize paired image–text similarity against negative samples.In practice, negative samples are retrieved from the mini-batch.Source: §3.1 · p. 2
  • The framework uses symmetric image-to-text and text-to-image alignment losses, with dot products measuring similarity and a learnable temperature scaling the distribution.The total CLIP loss combines the two alignment directions.Source: §3.1 · p. 2

3.2. CLIP Knowledge Distillation

CLIP-KD explores ways to transfer a large CLIP teacher’s knowledge to a small student through mimicry and teacher–student interaction.

3.2. CLIP Knowledge Distillation

  • CLIP-KD examines relation, feature, gradient, and contrastive distillation strategies for transferring knowledge from a large teacher CLIP to a small student.The methods are organized around mimicry and interaction.Source: §3.2 · p. 3

3.2.1 Contrastive Relational Distillation

  • CRD has the student match the teacher’s image-to-text and text-to-image contrastive distributions using KL-divergence.These distributions convey structured relationships among feature embeddings.Source: §3.2.1 · pp. 3–4

3.2.2 Feature Distillation

  • Feature Distillation uses Mean Squared Error to align the student’s visual and text embeddings with the teacher’s.When embedding dimensions differ, a linear projection head matches the student dimensions.Source: §3.2.2 · p. 4

3.2.3 Masked Feature Distillation

  • Masked Feature Distillation applies MSE alignment to visual and text embeddings while the student receives masked images.The patch-masking procedure follows MAE.Source: §3.2.3 · p. 4

3.2.4 Gradient Distillation

  • Gradient Distillation uses MSE to align teacher and student gradients with respect to visual and text embeddings.The method aims to make the student respond more like the teacher when inputs change.Source: §3.2.4 · p. 4

3.2.5 Interactive Contrastive Learning

  • Interactive Contrastive Learning uses student embeddings as anchors and teacher embeddings as contrasting examples in both image-to-text and text-to-image directions.Minimizing its loss is connected to maximizing a lower bound on mutual information between teacher and student networks.Source: §3.2.5 · p. 4

3.2.6 Augmented Feature Distillation

  • Augmented Feature Distillation aggregates student and teacher embeddings with visual and text fusion encoders.The fusion encoder is a linear projection layer.Source: §3.2.6 · p. 5

3.2.7 Overall Loss of CLIP Distillation.

  • The overall objective jointly trains the student on the original CLIP task and selected distillation losses, scaled by a loss weight.Multiple distillation losses can be used together.Source: §3.2.7 · p. 5

4. Experiments

Experiments train on CC3M and CC12M and evaluate zero-shot classification and cross-modal retrieval; ablations compare individual and combined distillation losses.

4.1. Experimental Setup

  • The experiments use CC3M and CC12M for pretraining, with zero-shot ImageNet classification and retrieval evaluation on CC3M Val, MSCOCO, and Flickr.Classification uses top-1 accuracy; retrieval uses R@1 by default.Source: §4.1 · p. 5

4.2. Ablation Study of Distillation Losses

  • FD improves the student by 3.6% top-1 accuracy on ImageNet and by 3.7%, 1.9%, and 4.2% R@1 on CC3M, MSCOCO, and Flickr, respectively.Among individual approaches, FD performs best; MFD shows similar performance, while ICL and CRD rank second and third overall.Source: §4.2 · p. 6
  • FD+CRD+ICL performs best in 6 out of 10 cases and is the default unified method for distilling CLIP models.Adding GD or AFD to FD+ICL+CRD may not improve performance.Source: §4.2 · p. 6

4.3. Distilling CLIP Models

This section evaluates lightweight CLIP students on zero-shot retrieval and classification, including transfer from Laion-400M teachers. Distillation improves performance across the reported settings, while the teacher’s size can affect outcomes.

4.3.1 Cross-Modal Retrieval on CC3M

  • With ViT-B/16 teachers, KD adds 2.0%∼3.9% image→text R@1 and 1.1%∼3.4% text→image R@1 across five student networks; ResNet-101 teachers also improve both directions.For ResNet-101 teachers, the respective gains are 2.0%∼4.4% and 1.9%∼3.8%. The paper attributes architecture-agnostic performance to distillation from final output embeddings rather than hidden layers.Source: §4.3.1 · p. 6

4.3.2 Zero-Shot Cross-Modal Retrieval

  • On zero-shot MSCOCO and Flickr retrieval, KD improves R@1 in both image→text and text→image directions with either ViT-B/16 or ResNet-101 teachers.Reported gains span 2.1%∼4.3% and 1.0%∼3.3% on MSCOCO for ViT-B/16, and 4.9%∼8.8% and 3.2%∼6.5% on Flickr; ResNet-101 gains are also reported for both datasets and directions.Source: §4.3.2 · pp. 6–7

4.3.3 Zero-Shot ImageNet-Related Classification

  • KD improves zero-shot ImageNet top-1 accuracy across five student networks under both ViT-B/16 and ResNet-101 supervision, with accuracy gains also reported on ImageNet variants.After distillation, Swin-T even outperforms the teacher models; the paper cites its strong baseline and transferred knowledge as reasons.Source: Table 4 · p. 7, §4.3.3 · p. 7

4.3.4 Transferred from Laion-400M

  • A Laion-400M-pretrained ViT-B/16 teacher yields a 7.7% ImageNet accuracy gain and 6.8% average cross-modal retrieval improvement when distilling ViT-T/16 over its CC3M+12M-pretrained counterpart.The results indicate that CLIP-KD can transfer knowledge from a large-scale dataset to students trained on a smaller-scale dataset.Source: §4.3.4 · p. 7
  • For the ViT-T/16 student, the more capable ViT-L/14 teacher underperforms the weaker ViT-B/16 teacher, possibly because of a teacher–student capacity gap.The authors identify this as a possible open issue for future research.Source: §4.3.4 · p. 7
  • Against TinyCLIP, CLIP-KD gains 1.8% ImageNet accuracy and 1.7% average cross-modal retrieval, while TinyCLIP supports only same-style teacher–student pairs.The paper reports these gains as evidence of CLIP-KD’s stronger performance and practicability.Source: §4.3.4 · p. 7

4.4. Analysis

The analyses examine training dynamics and relate student performance to teacher–student feature similarity. Feature Distillation (FD) directly increases similarity, while FD+ICL combines feature alignment with contrastive distillation.

Training curve of CLIP-KD

  • Training losses decrease and converge; CLIP-KD has lower task loss than the baseline, while task loss often exceeds ICL loss.The teacher provides converged contrastive embeddings in ICL, helping the student optimize its feature space readily.Source: §4.4 · p. 8, Figure 2 · p. 8
  • During training, both CLIP-KD and the baseline increase positive-minus-negative pair similarity, but CLIP-KD reaches higher values.The passage interprets this as learning more discriminative features.Source: §4.4 · p. 8, Figure 2 · p. 8

Interpreting why various KD methods have different performance

  • Student accuracy tracks teacher–student feature similarity, and FD performs best by directly increasing that similarity.FD+ICL combines feature alignment with contrastive distillation, while CRD, GD, and AFD achieve more limited gains above baseline.Source: §4.4 · p. 8

5. Conclusion

The paper studies CLIP-KD through several distillation strategies and reports significant improvements for small CLIP models. It presents the study as guidance for practical applications and future CLIP compression research.

5. Conclusion

  • The study examines relation, feature, gradient, and contrastive distillation strategies for CLIP-KD.The conclusion frames the work as a comprehensive study of distillation for small CLIP models.Source: §5 · p. 8
  • The proposed distillation methods significantly improve small CLIP models.The conclusion states this broadly without specifying a benchmark or margin.Source: §5 · p. 8
  • The authors hope the study provides practical CLIP-KD guidelines and draws attention to future CLIP compression research.Source: §5 · p. 8

A. Gradient Distillation.

Gradient distillation aligns teacher and student gradients with respect to visual and text embeddings. The appendix also connects interactive contrastive learning (ICL) to a lower bound on mutual information between student and teacher embeddings.

A. Gradient Distillation.

  • The gradient-matching approach uses gradients of CLIP’s image-to-text and text-to-image contrastive losses with respect to visual and text embeddings.The contrastive distribution from a visual embedding to text embeddings is also identified in the formulation.Source: A. Gradient Distillation · p. 11
  • Gradient distillation aligns teacher and student gradients with respect to visual and text embeddings using MSE loss.The paper motivates this as helping the student behave more like the teacher when outputs change with inputs.Source: A. Gradient Distillation · p. 11

B. Theoretical Insights of Interactive Contrastive Learning

  • For ICL, the positive-pair prior is 1/(1+N) and the negative-pair prior is N/(1+N), where N=B−1.The passage describes one positive pair for every N negative pairs.Source: A. Gradient Distillation · p. 12
  • Minimizing ICL loss maximizes a lower bound on mutual information between student and teacher cross-modal embeddings.The paper describes this as reducing uncertainty about teacher embeddings from student anchor embeddings and guiding the student toward common knowledge from the teacher.Source: A. Gradient Distillation · p. 12

C. Experiments

Experiments report downstream dense-prediction gains, compare training and intermediate-feature distillation, and examine loss weights and masking. Intermediate-feature results vary with teacher–student architecture similarity.

Analysis of Training performance curves of CLIP-KD

  • CLIP-KD consistently outperforms the baseline during training on cross-modal retrieval and ImageNet classification.Source: C. Experiments · p. 12

Linear evaluation on MS-COCO object detection and instance segmentation

  • CLIP-KD improves over original CLIP without KD by average AP margins of 1.5% on object detection and 1.2% on instance segmentation.These MS-COCO evaluations use a CC3M+12M-pretrained ResNet-50 backbone with Mask-RCNN.Source: Table 11 · p. 13, Table 12 · p. 13, C. Experiments · p. 14

Analyses of hyper-parameters

Distilling intermediate features

  • Intermediate feature distillation gives ViT-T/16 a 2.17% ImageNet accuracy gain with ViT-B/16, versus 0.56% with ResNet-101.The authors attribute the difference to more similar feature extraction and student-friendlier knowledge, and say this approach may be architecture-sensitive.Source: C. Experiments · p. 13, Figure 5 · p. 14

Related papers

Loading 2307.12732v2…