TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance

1. Introduction

• Cross-modal distillation is underexplored, and CLIP's two-branch interaction and costly pretraining make distillation challenging; this is the first work exploring language-image pretraining distillation.The cited CLIP pretraining uses 400 million image-text pairs for 32 epochs and requires thousands of GPU-days.Source: §1 · pp. 1–2
• Affinity mimicking distills cosine similarities between teacher image and text embeddings so students mimic visual-linguistic feature alignment.The paper reports that affinity-space distillation is more effective than relying on image or text features alone.Source: §1 · p. 2
• Weight inheritance transfers pretrained teacher weights to students as initialization, with manual selection or learnable masks to identify useful weights across modality branches.The masks are applied independently to the vision and language branches.Source: §1 · p. 2
• Progressive inheritance passes important weights through multiple stages, keeping each student structurally closer to its predecessor teacher during transfer.The paper presents this as a way to address architecture differences that may undermine transmitted weights.Source: §1 · p. 2
• TinyCLIP-ViT reaches 61.4% ImageNet zero-shot top-1 accuracy with 63M parameters, versus OpenCLIP's 62.9% with 126M; TinyCLIP training is 1.4×–7.8× faster.The models are pretrained on the same LAION-400M dataset.Source: §1 · p. 2

2. Related Work

• Earlier CLIP compression work targets generation or video understanding, whereas this work focuses on language-image pretraining compression.The paper identifies its work as the first on CLIP compression.Source: §2 · p. 2
• Prior cross-modal distillation addressed tasks such as VQA and image captioning, while this work targets general pretraining for transfer across downstream scenarios.The paper contrasts this with extensively studied single-modality distillation in vision and language.Source: §2 · p. 2
• Unlike predominantly single-modality pruning, TinyCLIP selects weights across modalities and uses progressive stages to accelerate small-model convergence.The paper also distinguishes its approach by emphasizing inherited weights, not only the pruned architecture.Source: §2 · pp. 2–3

3.1. Distillation with Affinity Mimicking

• Affinity mimicking trains students to match teacher image-text affinities in both image-to-text and text-to-image directions, preserving similarities among negative pairs.The two directional losses align the teacher and student affinity scores using cross-entropy.Source: §3.1 · p. 3

3.2. Distillation with Weight Inheritance

• Manual inheritance selects uniform k layers for the text branch and the first k channels for the image branch, reflecting their different redundancy patterns.The text encoder shows more depth redundancy, while the image encoder shows more width redundancy.Source: §3.2 · pp. 3–4
• Automatic inheritance learns separate branch masks for attention heads and FFN neurons, plus a shared embedding-dimension mask, under a sparsity constraint.The embedding mask is shared across layers because residual connections link corresponding hidden dimensions.Source: §3.2 · p. 4

3.3. Progressive Multi-Stage Distillation

• Above 70% target sparsity, one-stage compression can sharply reduce accuracy or prevent convergence; progressive stages use modest compression, such as 25%.Each stage inherits weights and then distills the smaller model with affinity mimicking until the target sparsity is reached.Source: §3.3 · pp. 4–5

4.1. Implementation Details

• Distillation uses OpenCLIP ViT-B/32 and extends to DaViT; training uses LAION-400M in three stages (100→75%, 75→50%, 50→25%; 6, 16, 16 epochs) or YFCC-15M in two 25-epoch stages (100→50%, 50→10%).Evaluation covers zero-shot transfer, robustness, and linear-probe classification.Source: §4.1 · p. 5

4.2. Comparison with State-of-the-Art Models

• TinyCLIP ViT-8M/16 surpasses CLIP ViT-B/16 by 3.5% on ImageNet while using 11.3× fewer parameters and running 5.1× faster.The models are distilled from OpenCLIP ViT-B/16 on YFCC-15M using affinity mimicking and manual weight inheritance.Source: §4.2 · p. 6
• TinyCLIP DaViT-5M improves ImageNet accuracy by up to 5.0% and image-to-text retrieval by 5.0% on Flickr and 4.5% on MSCOCO; TinyCLIP ResNet-19M cuts parameters by 50% and is 2× faster, with a 2.7% ImageNet performance drop.The DaViT result also reports 50.0% ImageNet accuracy using 10M parameters; ResNet-30M has a slight 0.5% ImageNet accuracy decrease and ~2% retrieval improvements.Source: §4.2 · p. 6
• TinyCLIP ViT-63M/32 outperforms the manually inherited ViT-61M/32 by more than ~1% on classification and retrieval; ViT-22M/32 reaches 5,504 throughput and 53.7% ImageNet top-1 accuracy.The comparison table covers state-of-the-art methods; automatic inheritance is used for some TinyCLIP models and manual inheritance for the others.Source: §4.2 · p. 6, Table 1 · p. 6

4.3. Ablation Study

• Affinity mimicking beats contrastive loss by 2.1% in ImageNet top-1 accuracy, while automatic inheritance adds 0.9% accuracy over manual inheritance and improves retrieval by 1.4–3.1%.Cross-modal interaction without affinity mimicking is 0.2% less accurate than affinity mimicking; manual inheritance improves ImageNet accuracy by 16.2% over no inheritance.Source: §4.3 · pp. 6–7
• Text encoders show greater depth redundancy than image encoders, while automatic inheritance reduces image-encoder width more and removes text-encoder attention layers and feed-forward channels.Teacher models with similar architectures and higher performance are better choices for inheritance; a strong teacher alone does not guarantee better results.Source: §4.3 · p. 7
• Two-stage compression beats one-stage compression by 2.0% in ImageNet zero-shot accuracy; models retaining 75% and 50% of parameters match OpenCLIP accuracy with 7.8× and 1.4× speedups, respectively.The reported training reference is OpenCLIP ViT-B/32 trained from scratch on LAION-400M for 32 epochs, taking around 287 GPU days.Source: §4.3 · p. 8

4.4. Transfer Learning Results

• On 23 datasets, TinyCLIP wins on 7 zero-shot and 9 linear-probe classification benchmarks; on five out-of-distribution robustness datasets, ViT-63M/32 beats OpenCLIP ViT-B/32 and wins three against CLIP ViT-B/32.TinyCLIP ViT-45M/32 uses 50% fewer parameters yet wins on IN-R and IN-Sketch against CLIP ViT-B/32.Source: §4.4 · p. 8, Table 5 · p. 8, Table 7 · p. 8

• TinyCLIP uses affinity mimicking and weight inheritance to distill large-scale vision-language pre-trained models.Source: §5 · p. 8
• Experiments and ablation studies show TinyCLIP can reduce model size while maintaining competitive performance.Source: §5 · p. 8
• Future work will explore ways to further improve cross-modal distillation efficiency on extremely small models.Source: §5 · p. 8