ComKD-CLIP: Comprehensive Knowledge Distillation for Contrastive Language-Image Pre-traning Model

• CLIP’s practical applicability is limited because large models cannot fit resource-constrained environments and small models often miss application performance requirements.Source: Introduction · p. 1
• ComKD-CLIP achieves the best performance on 8 of 11 diverse recognition datasets in comparisons with some state-of-the-art methods, using teacher-model knowledge distillation.The method was evaluated on 11 datasets.Source: Introduction · p. 2
• IFAlign aligns student image features with teacher image features so the student learns how the teacher extracts image representations during feature fusion.The module is designed to transfer image-feature extraction knowledge rather than only final fusion outputs.Source: Introduction · p. 2
• EduAttention models cross-relationships between teacher text features and student image features so the student learns the teacher’s text-image integration strategy.This mechanism targets knowledge embedded in the fusion process.Source: Introduction · p. 2
• The framework refines knowledge distilled by IFAlign and EduAttention using the teacher’s text-image feature fusion results.This refinement is intended to help the student accurately absorb the teacher’s knowledge.Source: Introduction · p. 2

Contrastive Language-Image Pre-training (CLIP)

• CLIP advances multimodal learning through contrastive pre-training, self-supervision, masking, attention, specialized encoders, and pruning.These methods target stronger representations, efficient attention, multi-task learning, multi-label classification, or teacher-knowledge utilization.Source: Contrastive Language-Image… · p. 2

Knowledge Distillation (KD)

• Knowledge distillation transfers knowledge from large teacher models to small students to seek comparable performance under resource constraints.CLIP-oriented approaches include layer-wise alignment, prompt-based distillation, affinity mimicking, and weight inheritance.Source: Knowledge Distillation (KD) · p. 2
• Existing CLIP distillation methods mainly use feature-fusion results, overlooking knowledge embedded within the fusion process.ComKD-CLIP addresses this gap by distilling fusion-process knowledge and refining it with fusion results.Source: Knowledge Distillation (KD) · p. 3

Preliminaries

• CLIP normalizes image and text encoder outputs into a joint embedding space, with text features serving as the classification weight matrix.Class probabilities are obtained from image-text logits scaled by the temperature parameter τ.Source: Preliminaries · p. 3

Pipline

• ComKD-CLIP trains a large prompted teacher, then trains a smaller student that reuses the teacher’s text features and requires learning only its image encoder branch.The teacher is pretrained on labeled domain data, while the student processes unlabeled domain data with a lightweight image encoder.Source: Pipline · p. 3

ComKD-CLIP

• IFAlign aligns the student’s and teacher’s image-feature means and variances with an L1 loss, using a learnable projector for dimension adjustment.This alignment is intended to transfer the teacher’s knowledge of extracting salient image features.Source: ComKD-CLIP · p. 4
• EduAttention uses attention to model the cross-relationship between student image features and teacher text features, transferring the teacher’s text-image integration strategy.The cross-relationship is computed through fully connected layers and a hyperparameter C.Source: ComKD-CLIP · pp. 4–5
• The framework combines aligned image features with attention-derived information through a learnable parameter α, initialized at zero and gradually assigned more weight.The resulting fused representation integrates knowledge from IFAlign and EduAttention.Source: ComKD-CLIP · p. 5
• KL-divergence refinement minimizes the discrepancy between teacher and student logits, and the final loss combines this distribution loss with the student alignment loss.The logits are computed from corresponding image and text features using the CLIP classification formulation, with τ controlling probability smoothness.Source: Preliminaries · p. 3, ComKD-CLIP · p. 5

Settings

• The experiments span 11 recognition datasets covering generic objects, fine-grained categories, scenes, actions, textures, and satellite imagery.Domain generalization further evaluates ImageNet-trained models on ImageNetA, ImageNet-R, ImageNet-Sketch, and ImageNetV2.Source: Settings · p. 5
• Base and novel class accuracies and their Harmonic Mean are averaged over three runs using a ViT-L/14 teacher and ViT-B/16 student.Training uses unlabeled data, SGD, and prompt depth 9 with prompt length 4 for both vision and language.Source: Settings · p. 5, Base-to-novel Generalization · p. 5

Domain Generalization Experiments

• Domain-generalization experiments train on ImageNet and evaluate robustness on datasets with domain shifts.This setting examines adaptability to different and unexpected environments.Source: Settings · p. 5, Domain Generalization Experiments · p. 5

Base-to-novel Generalization

• ComKD-CLIP shows highly competitive results against recent state-of-the-art methods across all 11 recognition datasets.The comparison covers CLIP, CoOp, CoCoOp, MaPLe, PromptSRC, and PromptKD.Source: Base-to-novel Generalization · p. 5, Table 1 · p. 6

Cross-dataset Evaluation

• 0.74% average improvement over previous methods is achieved across the cross-dataset benchmark, with ComKD-CLIP outperforming some state-of-the-art methods on 8 of 10 datasets.Evaluation uses unlabeled target-domain training data without dataset-specific fine-tuning.Source: Cross-dataset Evaluation · p. 5

The effectiveness of IFAlign & EduAttention

• The combined IFAlign and EduAttention design enables the student to absorb teacher knowledge for image-feature extraction and text-image feature integration.The ablation is conducted on SUN397.Source: Table 4 · p. 7
• The cross-dataset benchmark uses unsupervised aligned distillation with unlabeled domain data in a transductive setting, while domain generalization evaluates ImageNet-trained models under domain shifts.These tables provide complementary evaluations beyond the SUN397 ablations.Source: Table 2 · p. 7, Table 3 · p. 7
• Mean-plus-variance alignment gives ComKD-CLIP its best performance among the tested IFAlign strategies.It jointly aligns feature center position and discreteness, supporting absorption of the teacher’s image-feature extraction knowledge.Source: The effectiveness of IFAlign &… · p. 7, Table 5 · p. 7

Knowledge Refinement Method

• KL divergence produces the best performance among the tested knowledge-refinement methods on SUN397.The paper attributes this to making student logits better approximate teacher logits.Source: Knowledge Refinement Method · p. 7, Table 6 · p. 7

Conclusion

• ComKD-CLIP comprehensively distills a large teacher CLIP into a smaller student while maintaining comparable performance with substantially fewer parameters.The framework targets deployment in resource-constrained environments.Source: Conclusion · p. 7
• IFAlign and EduAttention distill knowledge from the teacher’s text-image feature-fusion process, and teacher fusion results refine the transferred knowledge.The stated goal is accurate absorption of teacher knowledge by the smaller student.Source: Conclusion · p. 7
• Experiments across 11 datasets demonstrate the superior performance of ComKD-CLIP.The conclusion links this result to broader practical utility for CLIP technology.Source: Conclusion · p. 7