ProCLIP: Progressive Vision-Language Alignment via LLM-based Embedder

• CLIP’s 77-token English-only supervision and limited fine-grained textual supervision hamper its ability to capture nuanced semantic details (Zhang et al., 2024; Tschannen et al., 2025; Hu et al., 2025).Source: §1 · p. 1
• ProCLIP retains more pretrained visual knowledge than LLM2CLIP in the CKA analysis of visual representations.Source: Figure 1 · p. 1, §1 · p. 2
• Long-CLIP extends input length but struggles with fine-grained and multilingual understanding (Zhang et al., 2024), while direct LLM alignment risks overfitting and weaker generalization, especially with scarce data (Cao et al., 2025; Huang et al., 2024).Source: §1 · p. 1
• ProCLIP distills CLIP text-encoder knowledge into an LLM adapter, then contrastively tunes representations with image-side self-distillation to safeguard pretrained knowledge.Source: §1 · pp. 1–2
• The authors report SOTA performance under identical data constraints and versatility across long-text, multilingual, and fine-grained scenarios with limited data.Source: §1 · p. 2
• 6.8%–13.5% gains in zero-shot classification accompany improvements across short/long-text, multilingual retrieval, and fine-grained understanding tasks.Source: §1 · p. 2

• CLIP’s fixed input length and capacity constrain long-form, multilingual, and fine-grained semantics; Long-CLIP and LoTLIP address long inputs but remain tied to CLIP’s text architecture (Zhang et al., 2024; Wu et al., 2024).Source: §2 · p. 2
• LLM-based embedders support multilingual, long-form, and complex inputs, but prior approaches often use coarse alignment strategies that degrade inherent generalization (Huang et al., 2024; Cao et al., 2025; Zhang et al., 2025a).Source: §2 · p. 2
• The proposed pipeline has two stages: representation inheritance through cross-architecture distillation, followed by contrastive tuning integrated with self-distillation regularization.Source: Figure 2 · p. 3
• TinyCLIP, CLIP-KD, and CLIP-CID use distillation for cross-modal interactions, teacher-student similarity, or semantic-prior transfer; ProCLIP targets forgetting during transition to LLM-based architectures (Wu et al., 2023a; Yang et al., 2024a; Yang et al., 2024b).Source: §2 · p. 3

3 Methodology

• ProCLIP uses a two-stage training framework: representation inheritance followed by contrastive tuning integrated with self-distillation regularization.The methodology first introduces preliminary concepts, then describes cross-architecture distillation and contrastive tuning.Source: §3 · p. 3

3.1 Preliminary

• CLIP’s 77-token text limit motivates using an LLM embedder, but direct contrastive tuning risks drift because its representations lack prior alignment with CLIP’s image encoder.The passage notes that limited fine-tuning data may not compensate for knowledge acquired during large-scale pretraining.Source: §3.1 · pp. 3–4

3.2 Stage I: Representation Inheritance via Cross-Architecture Distillation.

• Stage I distills CLIP text embeddings into projected LLM embeddings using instance-level alignment and pairwise-distance structure losses.The structural loss aligns inter-sample distances across the CLIP text and LLM embedding spaces.Source: §3.2 · p. 4

3.3 Stage II: Contrastive Tuning Integrated with Self-Distillation Regularization.

• Stage II applies InfoNCE to align image and projected LLM embeddings, while self-distillation regularizes the image encoder to mitigate forgetting of pretrained knowledge.The tuning objective combines the contrastive loss with a weighted regularization loss.Source: §3.3 · pp. 4–5

4.1 Experimental Setup

• Experiments scale training data from 3M to 30M and test classification, robustness, retrieval, multilingual retrieval, and fine-grained understanding across CLIP architectures.The setup uses CC3M, CC12M, and YFCC15M, with DreamLIP captions; the main CLIP embedder is LLaMA3-8B-CC.Source: §4.1 · pp. 5–6

4.2 Main Results

• 10.0%–13.5% average zero-shot classification gains over LLM2CLIP on 30M data accompany consistent retrieval and robustness improvements.ProCLIP also reports consistent retrieval gains across datasets and scales, and average robustness gains of 5.9%–9.3%.Source: Table 2 · p. 6, Table 3 · p. 6, §4.2 · p. 7
• 11.3%–17.7% gains over CLIP on SugarCrepe and further MMVP gains demonstrate improved fine-grained understanding.On MMVP, ProCLIP gains 3.0%, 2.2%, and 10.4% at 3M, 15M, and 30M data scales, respectively; it also outperforms compared methods across model sizes under the same or lower training costs.Source: Figure 4 · p. 6, §4.2 · p. 7, Figure 5 · p. 8

4.3 Ablation Study

• Ablations find that embedder choice affects classification more than retrieval, while the 4e-4 regularization weight is selected as the optimal balance.Llama3-CC performs best overall among tested embedders; excessively strong regularization hampers alignment. Instance and structural alignment losses transfer textual priors and global geometry, while self-distillation recovers classification after contrastive tuning compromises it.Source: §4.3 · pp. 7–8, Table 7 · p. 8
• Across tested base-model architectures, ProCLIP surpasses LLM2CLIP in classification and retrieval, though a classification gap to vanilla CLIP remains.The authors attribute the remaining gap to vanilla CLIP’s substantially larger data scale in zero-shot tasks.Source: §4.3 · p. 7, Table 6 · p. 8

5 Conclusion

• ProCLIP integrates CLIP image encoders with LLM-based embedders through a progressive two-stage alignment strategy.It first distills CLIP textual priors into the embedder, then performs contrastive tuning with self-distillation to prevent overfitting.Source: §5 · p. 8
• Instance semantic and structural alignment losses provide complementary distillation to maintain feature-space consistency.Source: §5 · p. 8
• Evaluations across various scales and architectures validate the efficacy and generality of the approach.Source: §5 · p. 9

6 Limitation

• The first stage incurs only modest additional computational overhead.Source: §6 · p. 9
• Unfreezing the vision encoder and performing online self-distillation in stage two significantly increases training cost.Source: §6 · p. 9
• ProCLIP adds approximately 35% computational overhead over the baseline; ViT-L training on 3M data takes 1.5 hours on 8× H100 GPUs.The authors describe this budget as manageable.Source: §6 · p. 9

A.1 Training Details

• ProCLIP’s MLP layers match the LLM2CLIP baseline configuration.Source: §A.1 · p. 12
• Both ProCLIP and the baseline use four linear layers in their MLPs.Source: §A.1 · p. 12
• Table 8 presents the hyperparameter configurations for ProCLIP’s two-stage training.Source: §A.1 · p. 12, Table 8 · p. 12

A.2 Details of Benchmarks.

• Zero-shot classification and linear probing are evaluated on 11 datasets, with prompts and dataset details listed in Table 9.The benchmark description also points to Table 16 for zero-shot prompts.Source: §A.2 · p. 12, Table 9 · p. 12
• Robustness is tested on five ImageNet variants, while cross-modal retrieval covers six datasets spanning short- and long-text settings.Retrieval settings are summarized in Table 10.Source: §A.2 · p. 12, Table 10 · p. 12
• XM3600 provides multilingual retrieval evaluation across 3,600 images and 36 languages.Fine-grained understanding is assessed on 150 MMVP-VLM samples covering nine patterns, and on SugarCrepe’s Add, Replace, and Swap negative types.Source: §A.2 · pp. 12–13, Table 11 · p. 13
• The benchmark suite also evaluates the fine-tuned vision encoder in LLaVA on SEED-Bench, GQA, VizWiz, PoPE, TextVQA, MMBench, and VQAv2.MMVP-VLM patterns include orientation, feature presence, state, quantity, positional relations, color, structure, text, and viewpoint.Source: §A.2 · pp. 12–13

Compared to Long-CLIP

• ProCLIP outperforms Long-CLIP on both short-text and long-text retrieval when both experiments use 1M data.The comparison is reported in Table 12.Source: §A.3 · p. 13

Linear Probe

• ProCLIP consistently achieves superior linear-probe performance across 11 datasets.The authors attribute this advantage to two-stage regularization that stabilizes training, prevents vision-encoder overfitting, and preserves generalization.Source: §A.3 · p. 13

Data and Model Scale

• Zero-shot IN-1k accuracy rises from 62.5 with 3M samples to 67.8 with 30M, and expanding the linear layers to 12 also yields gains.At the same data scale, ProCLIP consistently outperforms LLM2CLIP; with 1M CC3M samples it reaches 61.8 zero-shot IN-1k, with comparable or better zero-shot retrieval than LLM2CLIP.Source: §A.3 · p. 13, Table 14 · p. 14

MLLM Performance

• When integrated into an MLLM, ProCLIP improves over CLIP on most benchmarks but performs relatively comparably to LLM2CLIP.The authors attribute the lack of a significant MLLM-benchmark advantage to not placing additional emphasis on those downstream benchmarks; Table 13 covers the evaluation.Source: §A.3 · pp. 13–14, Table 13 · p. 14

A.4 Future Works.

• ProCLIP incurs increased computational overhead during training.Potential reductions include PEFT or partial vision-encoder fine-tuning, and replacing online with offline distillation.Source: §A.4 · p. 14
• A dedicated local alignment loss is proposed for future work to strengthen fine-grained visual-textual alignment on tasks requiring detailed spatial understanding..Source: §A.4 · p. 14
• Replacing the image encoder remains an open direction, motivated by the CLIP image encoder’s stated lack of locality.The authors leave thorough investigation of this option for future work.Source: §A.4 · p. 14

A.5 Visualization.

• Figure 7 presents four long-text cross-modal retrieval cases, and the accompanying text describes precise alignment with highly detailed descriptions..Source: §A.5 · p. 14, Figure 7 · p. 17