PaperlayerView exact revision on arXiv

Source-linked AI summary

MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representations

Jongsuk Kim, Qiyu Wu, Zhuoyuan Mao, Hiromi Wakaki, Junmo Kim, Yuki Mitsufuji

arXiv:2608.25575v1cs.CVcs.AI

TL;DR

CLIP often struggles with compositional vision-language understanding, while directly transferring knowledge from generative MLLMs to discriminative CLIP encoders presents an architectural mismatch. MLLMCLIP performs feature-level heterogeneous distillation and consistently outperforms prior CLIP-enhancement methods across compositionality, classification, and retrieval benchmarks.

  • Problem

    CLIP often exhibits bag-of-words behavior that limits compositionality, while distilling between generative MLLM decoders and discriminative CLIP encoders remains architecturally heterogeneous.

  • Method

    MLLMCLIP transfers MLLM hidden-state features directly into CLIP through attention-based token selection and a structure-aware CKA distillation loss.

  • Results

    Across 26 benchmarks spanning compositionality, classification, and retrieval, MLLMCLIP consistently outperforms prior CLIP-enhancement methods and achieves the highest average score across regimes.

  • Takeaways & Limitations

    Feature-level MLLM distillation can build lightweight vision-language encoders with strong compositional and general representation performance across diverse teacher families.

  • Takeaways & Limitations

    Teacher biases and reasoning failures can transfer to the student, and the optimal teacher may vary across downstream tasks.

Abstract

from arXiv · show

Pretrained vision-language models such as CLIP excel at zero-shot recognition but often fail at compositionality, particularly attribute-object and relational structures. Recent studies mitigate this issue by augmenting training with synthetic hard negatives generated by a cascade of large language models and text-to-image models, which incurs substantial pipeline overhead. We instead propose MLLMCLIP, a heterogeneous distillation framework that transfers multimodal knowledge directly from a generative Multimodal Large Language Model (MLLM) teacher into a discriminative CLIP student, bypassing synthetic data entirely. To bridge the architectural mismatch between the two paradigms, we introduce an attention-based per-layer token selection and a CKA-based distillation loss. Compared to prior CLIP-enhancement methods, MLLMCLIP achieves state-of-the-art compositional accuracy while delivering consistent gains on standard zero-shot classification and image-text retrieval, showing that feature-level distillation strengthens both compositional and general vision-language representation capability.

1 Introduction

CLIP’s bag-of-words behavior limits compositionality, while existing hard-negative data-level distillation relies on costly synthetic-data pipelines. MLLMCLIP instead distills hidden-state features from a generative MLLM into CLIP using token selection and CKA alignment, achieving broad benchmark gains.

  • MLLMCLIP transfers hidden-state features from a frozen generative MLLM teacher directly into a discriminative CLIP student in a single forward pass.
  • CLIP often exhibits bag-of-words behavior that prevents it from capturing compositionality.
  • Prior data-level distillation synthesizes hard-negative captions and images with external LLMs and text-to-image models, incurring iterative sampling and quality-control overhead.
  • The framework addresses architectural heterogeneity with attention-based teacher-token selection and a CKA-based loss aligning structural relationships.
  • MLLMCLIP is evaluated on 11 compositionality benchmarks, 13 zero-shot classification datasets, and 2 image-text retrieval benchmarks, outperforming prior CLIP-enhancement methods.
  • MLLMCLIP outperforms previous CLIP-enhancement methods on 26 benchmarks spanning compositionality, classification, and retrieval with diverse MLLM teachers.

2 Related Works

Prior work improves CLIP through data, training, and textual-supervision enhancements, while also identifying its bag-of-words limitation. Knowledge distillation research largely follows homogeneous teacher–student architectures, motivating distinction between homogeneous and heterogeneous regimes.

  • CLIP Enhancement: CLIP generalizability studies use data augmentation, improved training strategies, and enriched textual supervision such as paraphrased or longer captions.These approaches represent broad enhancements to CLIP rather than addressing a single architectural limitation.
  • CLIP Enhancement: Yuksekgonul et al. (2022) identifies CLIP’s tendency to behave like a bag-of-words model as a key limitation.The passage introduces this observation as a contrast to general enhancement methods.
  • Knowledge Distillation: Most prior distillation efforts are homogeneous, compressing a larger model into a smaller model within the same model family.The passage distinguishes distillation regimes according to the architectural relationship between teacher and student.
  • Knowledge Distillation: Homogeneous distillation has been explored for CLIP-to-CLIP, LLM-to-LLM, and MLLM-to-MLLM teacher–student pairs.The cited examples include Yang et al. (2024) and Chen et al. (2024), Chenglin et al. (2024) and Xu et al. (2024b), and Cai et al. (2025) and Xu et al. (2024a), respectively.

3 Does MLLM Possess Sufficient Compositionality?

The section first tests whether current MLLMs possess sufficient compositional understanding to serve as teachers. A SugarCrepe question-answering evaluation finds that base MLLMs are consistently strong, whereas embedding-tuned counterparts substantially lag, motivating direct distillation from base MLLMs.

  • Motivation: The pilot study evaluates whether recent MLLMs understand compositional structures well enough to transfer that knowledge to a CLIP student.The authors treat compositional competence in the teacher as a prerequisite for constructing the distillation framework.
  • QA-based Evaluation Protocol: SugarCrepe is reformulated as question answering, prompting an MLLM with an image and two captions to select the caption that correctly describes the image.This replaces the benchmark’s usual image-text similarity comparison with a single-letter answer.
  • Models and Procedure: The evaluation covers LLaVA-1.6, LLaMA-3.2-Vision, Qwen, and InternVL, alongside MLLM-based embedding models.The comparison includes recent open-source generative MLLMs and embedding-style models evaluated on the same benchmark.
  • Models and Procedure: MLLMs are evaluated twice with the ground-truth caption placed in swapped positions to account for position bias.The caption-order swap is applied during evaluation rather than changing the benchmark’s image-caption content.
  • Results: Table 1 shows consistently high compositional ability for MLLMs, while embedding-tuned counterparts substantially lag regardless of the underlying MLLM.The results motivate directly distilling compositional understanding from base MLLMs instead of relying on embedding-style fine-tuning.

4 Method

MLLMCLIP transfers multimodal knowledge from an MLLM teacher to a CLIP student through feature-level distillation. It bridges their architectural mismatch with auxiliary feature alignment, attention-guided token and layer selection, and CKA-based representation matching alongside contrastive learning.

  • Framework: MLLMCLIP distills multimodal knowledge from a generative MLLM teacher into a discriminative CLIP student at the feature level.The framework is teacher-agnostic and bypasses synthetic-data generation.
  • Student Architecture: The CLIP student uses separate image and text encoders, auxiliary layers to align teacher and student dimensions, and linear projection heads for common-space contrastive learning.Image inputs use patch embeddings and [CLS], while text inputs use tokenization with [SOS] and [EOS].
  • Teacher Architecture: The MLLM teacher jointly encodes image and text as one sequence through decoder layers, using a fixed compositionality-oriented text template.Image features are concatenated with [Image End], followed by text tokens, before decoder processing.
  • Token and Layer Selection: Attention-based token selection averages attention across heads and chooses the token receiving the highest maximum attention in each teacher layer.This selects the most attended token under the assumption that it carries salient multimodal information.
  • Token and Layer Selection: A subset S of |S| = Ls teacher layers is selected using fixed relative positions or uniform stride intervals to reduce distillation cost.The selected layers correspond to the number of student encoder layers.
  • Distillation Objective: The training objective combines CLIP-style InfoNCE contrastive learning with intermediate-feature distillation, primarily using a CKA loss for cross-architecture structural alignment.CKA compares centered batch representations through Gram matrices, uses a square-root loss variant, and averages across image and text modalities and selected layers.

5 Experiments

Experiments evaluate MLLMCLIP across compositionality, zero-shot classification, and retrieval benchmarks using five MLLM teachers distilled into CLIP Base students. MLLMCLIP consistently surpasses prior enhancement methods, while token selection, CKA distillation, offline features, and CLIP-style inference improve effectiveness or efficiency.

  • Experimental Setup: MLLMCLIP is evaluated on 11 compositionality benchmarks, 13 zero-shot classification datasets, and 2 image-text retrieval benchmarks using CC3M pretraining data.Captions are formatted with a structured template, and frozen-teacher per-layer features are extracted offline.
  • Experimental Setup: Five MLLM teachers are distilled into CLIP Base students with ViT-B/32 image and 12-layer Transformer text encoders.The teachers include LLaVA-1.6-Mistral-7B, Qwen3-VL-2B, Qwen3.5-2B, InternVL3-2B, and InternVL3.5-2B.
  • Main Results: MLLMCLIP outperforms previous CLIP-enhancement methods on the 11 compositionality benchmarks, whereas prior methods provide only modest gains over CLIP.The compared baselines are LaCLIP, NegCLIP, FSC-CLIP, and TripletCLIP, which use external language or image-generation supervision.
  • Main Results: Across all five teachers, MLLMCLIP improves over every previous method in both zero-shot classification and image-text retrieval.These evaluations cover 13 classification datasets and 2 retrieval datasets, indicating gains beyond compositionality-focused supervision.
  • Ablations: Attention-based teacher-token selection and CKA distillation address the mismatch between generative MLLMs and discriminative CLIP encoders.Fixed token positions improve over baseline, while MSE causes performance collapse and CKA consistently outperforms direct similarity approaches.
  • Efficiency: 3.6× is the approximate per-batch time cost of cascaded negative caption and image generation relative to feature extraction.Pre-extracted teacher features also permit larger student batch sizes under the same GPU memory budget.

6 Conclusion

MLLMCLIP transfers multimodal knowledge from an MLLM teacher to a CLIP student through feature-level distillation, addressing their architectural mismatch with attention-based token selection and a structure-aware CKA loss. The framework performs strongly on compositionality, zero-shot classification, and image-text retrieval.

  • Conclusion: MLLMCLIP transfers multimodal knowledge from an MLLM teacher into a CLIP student through feature-level distillation.The framework is designed to strengthen vision-language representations without changing the teacher-student paradigm described in the conclusion.
  • Conclusion: Attention-based teacher-token selection and a structure-aware CKA distillation loss address the architectural mismatch between MLLM teachers and CLIP students.These mechanisms bridge the heterogeneous architectures involved in distillation.
  • Conclusion: MLLMCLIP achieves strong performance across compositionality, zero-shot classification, and image-text retrieval.The conclusion presents these results as evidence of the potential of MLLM-guided distillation.

Limitations

MLLMCLIP’s performance is bounded by the capabilities of its teacher MLLM, whose biases or reasoning failures may transfer to the student. Teacher choice may vary across downstream tasks, and reported configurations use a single seed despite consistent gains across five MLLM teachers.

  • Teacher Dependence: Teacher MLLM capabilities cap MLLMCLIP performance because teacher biases or reasoning failures can transfer during distillation.The limitation concerns the direct transfer of teacher shortcomings to the student.
  • Teacher Dependence: The optimal teacher may vary across downstream tasks despite the proposed rigorous teacher-selection protocol.Teacher selection is therefore task-dependent rather than universally optimal.
  • Evaluation Protocol: Reported configurations use a single seed, trading repeated runs for teacher diversity, while gains remain consistent across all five MLLM teachers.The study reports each configuration once but observes consistent gains across the tested teacher set.

A Additional Ablation Studies

The ablations examine supervision source, teacher prompting, layer selection, and distillation weighting. They indicate that compositional prompting and balanced loss weighting matter, while broader prompt sweeps remain computationally impractical.

  • Data-level versus Feature-level Supervision: The controlled comparison uses Qwen3.5-2B as both hard-negative generator and feature-level teacher under an identical training budget.This isolates feature-level transfer from teacher identity, although the supplied passage does not include Table A’s results.
  • Teacher Prompt: A compositional contrastive prompt yields a small but consistent improvement over a plain caption prompt with the teacher and other settings fixed.The full quantitative result is not included in the supplied passage.
  • Teacher Prompt: A full prompt-by-prompt ablation is computationally prohibitive because each variant requires re-extracting teacher features over the entire corpus.The study therefore reports a single informative comparison and leaves broader prompt design to future work.
  • Weight Parameter: λ = 1 achieves the best zero-shot classification and retrieval, whereas λ = 10 marginally improves compositionality.λ = 0.1 provides insufficient supervision and λ = 100 over-constrains the student; both degrade performance across all metrics.

B Experimental Settings · B.1 Evaluation Benchmarks

The evaluation covers compositionality, retrieval, and zero-shot classification through 11 compositionality benchmarks, two retrieval datasets, and 13 classification datasets.

  • B.1 Evaluation Benchmarks: The study evaluates compositionality on 11 benchmarks, including ARO, CREPE, EQBEN, ImageCoDe, and SugarCrepe.The benchmark suite also includes SVO-Probes, VALSE, VL-Checklist, WhatsUp, Winoground, and SPEC.
  • B.1 Evaluation Benchmarks: The compositionality evaluation additionally includes SVO-Probes, VALSE, VL-Checklist, WhatsUp, Winoground, and SPEC.These benchmarks are listed alongside ARO, CREPE, EQBEN, ImageCoDe, and SugarCrepe.
  • B.1 Evaluation Benchmarks: Retrieval uses MSCOCO and Flickr-30K for zero-shot image-to-text and text-to-image retrieval.Both retrieval directions are evaluated on each listed dataset.
  • B.1 Evaluation Benchmarks: The retrieval benchmarks measure both zero-shot image-to-text and text-to-image retrieval.The datasets are MSCOCO and Flickr-30K.
  • B.1 Evaluation Benchmarks: Zero-shot classification is evaluated on 13 datasets, including Caltech101, CIFAR-10, CIFAR-100, Describable Textures, and EuroSAT-CLIP.The passage also lists FER-2013, Flower102, Food101, ImageNet-1K, KITTI-Distance, Oxford-IIIT Pet, RESISC45-CLIP, and PASCAL VOC2007.
  • B.1 Evaluation Benchmarks: The classification suite includes FER-2013, Flower102, Food101, ImageNet-1K, KITTI-Distance, Oxford-IIIT Pet, RESISC45-CLIP, and PASCAL VOC2007.These datasets complement Caltech101, CIFAR-10, CIFAR-100, Describable Textures, and EuroSAT-CLIP.

B.2 Implementation Details · B.3 Model Sources · LLM Usage

The paper specifies its feature-extraction prompt, training configuration, public model sources, and limited assistive use of LLMs. It also documents the evaluation setup for MLLM-based judges and embedding models on SugarCrepe.

  • B.2 Implementation Details: The teacher prompt asks whether a caption matches an image, then elicits a subtly misleading alternative while emphasizing attributes, actions, relations, and spatial arrangements.This prompt is the feature-extraction template used for the teacher.
  • B.2 Implementation Details: Training uses 8 NVIDIA A100 GPUs, AdamW, batch size 4096, learning rate 5 × 10−4, weight decay 0.5, and 30 epochs.A cosine scheduler with first-epoch linear warmup starts at 1 × 10−6 and decays to 1×10−5 by training end.
  • B.2 Implementation Details: Experiments use bfloat16 precision and, unless specified otherwise, set the distillation–contrastive loss-balancing weight to λ = 1.This fixed weight balances the two losses during training.
  • B.3 Model Sources: Table E records the public sources of all models used in the experiments.The paper provides model-source links rather than introducing additional proprietary model sources.
  • LLM Usage: LLMs assist only with code review, minor-bug detection, English proofreading, and grammar checks; they do not draft passages or create methods or results.The paper describes this use as limited and assistive.
  • B.3 Model Sources: Listed models include Qwen3-4B for hard-negative captions and SDXL-Turbo for hard-negative images, alongside multiple LLaVA, LLaMA, and Qwen vision-language models.The model inventory also includes Qwen3.5-2B and other listed vision-language checkpoints.
  • B.3 Model Sources: The source list further includes VLM2Vec variants, VLM2Vec-V2.0, and Qwen3-VL-Embedding models with 2B and 8B sizes.These models cover embedding backbones based on Qwen2-VL, LLaVA-Next, and Phi-3.5V.
  • LLM Usage: On SugarCrepe, judge models are queried twice with the ground-truth caption in either position, whereas embedding models report image–text similarity accuracy per task.The comparison covers MLLM-as-judge and MLLM-based embedding models across Replace, Swap, and Add tasks.
Loading 2608.25575v1…