PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Contrastive Loss in CLIP Post-training: A Complementary Framework with Frozen Text Encoder

Zidan Wang, Yaqian Li, Xiaokai Zhang, Kaiwen Long, Kun He, Hanpeng Liu

arXiv:2610.11374v1cs.CVcs.AI

TL;DR

CLIP post-training is a lightweight alternative to costly retraining, but prior work argues contrastive learning forgets under small batches. This paper attributes InfoNCE forgetting primarily to temperature and proposes ComCLIP, which combines tempered contrastive learning with anchoring and relational distillation. ComCLIP trades retrieval strength for better linear-probe transferability while maintaining zero-shot parity with CLIP-Refine and showing no net change across eight LLaVA benchmarks.

  • Problem

    Prior CLIP post-training work argues that small batches make contrastive loss inadequate because insufficient negatives lead to catastrophic forgetting, motivating distillation-based alternatives.

    Source: §1 · pp. 1–2

  • Method

    ComCLIP freezes CLIP’s text encoder and trains its vision encoder with properly tempered contrastive loss, MSE anchoring to the original CLIP, and DINOv2 relational distillation.

    Source: §1 · p. 2

  • Results

    ComCLIP matches CLIP-Refine on zero-shot classification, improves linear-probe transferability and ViT-L/14 MMVP, trails on retrieval, and yields no net change across eight LLaVA benchmarks.

    Source: §5 · p. 10

  • Takeaways & Limitations

    In the tested LLaVA-1.5-7B configuration, ComCLIP serves as a drop-in vision encoder without re-alignment and does not change performance across eight VLM benchmarks.

    Source: §1 · p. 2

    The temperature analysis covers InfoNCE, not CLIP-Refine’s KL-based self-distillation or SigLIP’s sigmoid loss, and downstream evaluation uses only LLaVA-1.5-7B.

    Source: §5 · p. 10

Abstract

from arXiv · show

CLIP serves as a foundational vision-language model and the de facto vision encoder for downstream VLMs such as LLaVA. Post-training offers a lightweight route to refine CLIP, but recent work argues that the standard contrastive loss is unsuitable for post-training due to catastrophic forgetting under small batches, motivating designs that abandon the contrastive objective in favor of distillation. We revisit this premise and find that, for the InfoNCE objective, the reported forgetting is driven primarily not by insufficient negatives but by an inappropriate magnitude of the contrastive temperature $\tau$: with $\tau$ set sufficiently small, contrastive post-training improves rather than degrades the pretrained CLIP, which we explain through the temperature dependence of the InfoNCE gradient. Building on this finding, we propose \textbf{ComCLIP}, a lightweight single-epoch post-training recipe that freezes CLIP's text encoder---so the refined vision encoder is a drop-in replacement with unchanged architecture and inference cost---and trains the vision encoder with a properly-tempered contrastive loss, an MSE anchoring loss against the original CLIP, and a relational distillation loss from DINOv2. Over multiple seeds, ComCLIP matches the self-distillation baseline CLIP-Refine on zero-shot classification while significantly improving the transferability of visual features, measured by linear probing ($48.99$ vs.\ $42.28$ on ViT-B/16), and on ViT-L/14 it also improves MMVP over CLIP-Refine ($24.20$ vs.\ $19.01$); CLIP-Refine remains stronger on image-text retrieval. Used as a drop-in vision encoder for LLaVA-1.5-7B without re-aligning the projector or LLM, ComCLIP yields no net change across $8$ VLM benchmarks, i.e., the refinement does not break downstream compatibility. Code and models are available at https://github.com/showstarpro/ComCLIP.git.

1 Introduction

The introduction argues that lightweight post-training can refine CLIP while preserving compatibility with downstream VLMs. It finds that temperature, rather than insufficient negatives, drives reported InfoNCE forgetting and motivates ComCLIP’s combination of contrastive learning, anchoring, and relational distillation.

1 Introduction

  • CLIP is a de facto vision encoder for VLMs such as LLaVA, while retraining from scratch is costly and breaks compatibility with the CLIP-based ecosystem.The target is to improve feature transferability while keeping the refined encoder in the original CLIP image-text space, without retraining the projector or LLM.Source: §1 · p. 1
  • Prior methods avoid contrastive loss, arguing that small-batch post-training lacks negatives and causes catastrophic forgetting, even at batch size 2,048.CLIP-Refine and external-teacher methods instead use image-text soft labels or align CLIP representations with DINOv2.Source: §1 · pp. 1–2
  • With τ = 0.01, InfoNCE post-training raises 12-dataset zero-shot accuracy from 61.82 to 62.90, whereas τ = 0.07 lowers it to 57.90 under the same compute budget.The negative-pair gradient decays exponentially with its margin to the positive; small τ suppresses gradients on easy pairs and concentrates updates on hard negatives.Source: §1 · p. 2
  • ComCLIP freezes CLIP’s text encoder and combines properly tempered contrastive loss, MSE anchoring to the original CLIP, and DINOv2 relational distillation.The resulting vision encoder is a drop-in replacement with unchanged architecture and inference cost.Source: §1 · pp. 2–3
  • On ViT-B/16, zero-shot accuracy is at parity (63.00±0.13 vs. 62.87±0.11), ComCLIP leads linear probing (48.99±0.35 vs. 42.28±0.62), and CLIP-Refine leads retrieval.ComCLIP also improves ViT-L/14 MMVP over CLIP-Refine (24.20±0.86 vs. 19.01±1.86); in LLaVA-1.5-7B, it yields no net change across 8 VLM benchmarks.Source: §1 · pp. 2–3

2 Related Work

Related work situates ComCLIP among approaches to CLIP’s persistent visual limitations and lightweight post-training. The paper positions its fixed-text-encoder recipe as a way to improve visual-feature transferability while retaining a drop-in encoder, rather than maximizing fine-grained perception.

2 Related Work

  • CLIP retains the modality gap (Liang et al., 2022) and fine-grained visual blindness (Tong et al., 2024), which propagate into downstream VLMs (Li et al., 2024).Prior work improves CLIP through objectives, data, training recipes, and architectures, including SigLIP (Zhai et al., 2023), LaCLIP (Fan et al., 2023), CLIPA (Li et al., 2023a), and EVA-CLIP (Sun et al., 2023).Source: §2 · p. 3
  • Unlike LiT (Zhai et al., 2022), which freezes the image encoder and tunes text, ComCLIP freezes text and refines the image encoder to retain compatibility with the original CLIP text space.The paper names LLaVA’s projector as an example of a system built on that space.Source: §2 · p. 3
  • CLIP-Refine (Yamaguchi et al., 2025) abandons contrastive loss for image-text soft labels, while KUEA (Gong et al., 2025) aligns visual representations with DINOv2 (Oquab et al., 2024).Generative approaches DIVA (Wang et al., 2025), un2CLIP (Li et al., 2025), and GenHancer (Ma et al., 2025) target fine-grained perception; their training uses a large generative model.Source: §2 · p. 3
  • ComCLIP treats InfoNCE as viable post-training when temperature is appropriate, combining it with anchoring and DINOv2 relational distillation (Park et al., 2019; Tung and Mori, 2019; Zheng et al., 2021).Its stated target is single-epoch refinement that improves visual-feature transferability while remaining a drop-in replacement, not maximizing fine-grained perception.Source: §2 · p. 3

3 Method

ComCLIP post-trains CLIP’s vision encoder while freezing its text encoder, combining contrastive alignment, pretrained-feature anchoring, and DINOv2 relational supervision in the shared projected space.

3.2 Overview of ComCLIP

  • ComCLIP combines a properly tempered contrastive loss, pretrained-feature anchoring, and DINOv2 relational distillation while freezing CLIP’s text encoder.The three losses operate on the same post-projection embedding space.Source: §3 · p. 3, §3.2 · p. 4

3.1 Preliminaries

  • The post-training setup refines the pretrained image encoder on a small dataset while keeping the pretrained text encoder frozen and applying all losses to its projected embeddings.The projected image embedding lies in CLIP’s shared image-text contrastive space.Source: §3.1 · p. 4

3.2 Overview of ComCLIP

  • ComCLIP weights its three losses to preserve image-text alignment, anchor pretrained features, and transfer DINOv2’s relational structure.The losses are the contrastive objective, feature-level MSE, and relational distillation.Source: §3.2 · p. 4

3.6 Complementary Effects of the Three Losses

  • The three losses act on the same shared contrastive space, with contrastive and relational distillation shaping relative structure and MSE anchoring absolute geometry.The design rationale separates relational constraints from the absolute-position constraint.Source: §3.2 · p. 4, §3.6 · p. 6

3.3 Properly-Tempered Contrastive Loss

  • With InfoNCE, a smaller temperature concentrates gradient on hard negatives, leaving easy pretrained pairs largely unchanged; the authors recommend setting τ well below its pretraining value.At τ ≈ 0.07, many already-correct pairs still receive gradients, whereas a smaller τ makes gradients on easy negatives decay exponentially.Source: §3.3 · p. 5

3.4 MSE Anchoring Loss

  • The MSE anchoring loss matches trainable image embeddings to a frozen pretrained copy, providing a stable reference that limits drift during joint optimization.The anchor preserves the pretrained feature manifold in CLIP’s contrastive space.Source: §3.4 · p. 5

3.5 Relational Distillation from DINOv2

  • Relational distillation transfers DINOv2’s inter-sample structure by matching row-wise distributions derived from batch-wise similarity matrices.The DINOv2 features are computed without gradients, and the loss is applied to post-projection embeddings.Source: §3.5 · pp. 5–6

4 Experiments

Experiments compare ComCLIP with CLIP-Refine and other baselines across classification, retrieval, feature transfer, and downstream VLM compatibility. ComCLIP shows a linear-probing advantage over CLIP-Refine, while retrieval and other outcomes depend on the metric and setting.

4.1 Settings

  • Experiments use frozen text and teacher encoders, ViT-B/16 and ViT-L/14 students, and one-epoch post-training; evaluation spans five task families.The settings compare multiple datasets and baselines, and ComCLIP uses τ = 0.01 with unchanged architecture and parameter count.Source: §4.1 · pp. 6–7, §4.2 · p. 7

4.2 Main Results

  • 48.99±0.35 vs. 42.28±0.62 linear probing favors ComCLIP, while ViT-B/16 zero-shot is at parity and CLIP-Refine leads retrieval.Dataset-matched comparisons preserve ComCLIP's linear-probe advantage: +4.10 on ViT-B/16 CC3M and +7.45 on ViT-L/14 COCO Caption; larger corpora do not consistently improve results.Source: §4.2 · pp. 7–8, Table 2 · p. 8

4.3 Multi-seed Statistics

  • ViT-L/14 MMVP is 24.20±0.86 for ComCLIP versus 19.01±1.86 for CLIP-Refine; the authors make no MMVP claim for ViT-B/16.The ViT-L/14 comparison uses three seeds per method with both trained on CC3M; the ViT-B/16 results are indistinguishable.Source: §4.3 · p. 8, Table 2 · p. 8

4.4 Downstream VLM Compatibility

  • ComCLIP produces no net change relative to original CLIP in LLaVA-1.5-7B with its LLM and projector frozen, supporting compatibility but not accuracy gains without re-alignment.Across the nine scores, ComCLIP improves four, ties two, and degrades three; its mean per-benchmark relative change is +0.02%.Source: §4.4 · pp. 8–9, Table 3 · p. 8

4.5.1 Loss Decoupling

  • The full loss configuration reaches the highest linear-probe accuracy in the ablation, while Lclip+Lmse gives the highest zero-shot accuracy.The full objective reaches 48.74 linear-probe accuracy and 63.15 zero-shot accuracy; the table reports single runs except where seed variation is given.Source: §4.5.1 · p. 9, Table 4 · p. 9

4.5.2 Contribution of Lrkd

  • Adding Lrkd significantly improves ViT-L/14 linear probing, but leaves zero-shot unchanged and produces small, mixed retrieval changes.The ViT-B/16 linear-probe change has the same sign but is not significant; the authors retain Lrkd on the basis of linear-probe transferability only.Source: §4.5.2 · pp. 9–10

4.5.3 Effect of Contrastive Temperature

  • With Lclip alone, τ = 0.07 reduces zero-shot accuracy by 3.92 and Flickr30K I→T by 9.00, whereas tested 1/τ values from 50 to 200 improve both.At τ = 0.01, increasing batch size from 128 to 2,048 improves zero-shot by 1.03, less than the decline associated with τ = 0.07.Source: §4.5.3 · p. 10, Table 5 · p. 10

5 Conclusion and Limitations

ComCLIP is a single-epoch CLIP post-training recipe with a fixed text encoder; it matches CLIP-Refine on zero-shot classification while improving linear-probe transferability and ViT-L/14 MMVP, but trails on retrieval. Its limitations include evaluation on one VLM configuration and an InfoNCE-only temperature analysis.

5 Conclusion and Limitations

  • ComCLIP matches CLIP-Refine on zero-shot classification, improves linear-probe transferability on both backbones and MMVP on ViT-L/14, trails on retrieval, and yields no net change in LLaVA-1.5-7B.The evaluation uses a frozen projector and LLM.Source: §5 · p. 10
  • The temperature analysis covers InfoNCE, and its downstream VLM evaluation uses only LLaVA-1.5-7B; other VLM architectures would better establish generality.Without projector re-alignment, ComCLIP preserves but does not improve downstream VLM accuracy; whether re-alignment converts linear-probe gains into VLM gains is untested.Source: §5 · p. 10

A Experiments Settings

  • Evaluation covers 12-dataset zero-shot classification, linear probing on five datasets, and eight multimodal large-language-model benchmarks.The zero-shot set includes ImageNet-1K, CIFAR10/100, Caltech101, FER2013, OxfordPets, DTD, RESISC45, EuroSAT, PCAM, ImageNet-Sketch, and ImageNet-O.Source: A Experiments Settings · p. 15

B Post-training Performance Results based on SigLIP

On a SigLIP ViT-SO400M/14 backbone, ComCLIP post-trained with InfoNCE improves zero-shot classification and retrieval over pretrained SigLIP while leaving linear probing unchanged. The experiment does not test the temperature finding with SigLIP’s sigmoid loss.

B Post-training Performance Results based on SigLIP

  • 71.24 → 72.15 zero-shot classification and 70.87 → 70.88 linear probing; retrieval also improves over pretrained SigLIP with ComCLIP.MMVP changes from 39.26 to 41.48, within observed seed variance; the experiment uses InfoNCE on a SigLIP backbone, not SigLIP’s sigmoid loss.Source: B Post-training Performance Results… · p. 15, Table 6 · p. 15

C Disentangling Temperature from Batch Size and Learnability

Controlled studies find that small-temperature InfoNCE improves zero-shot accuracy across tested batch sizes, while learnability of temperature has little effect once initialized small. The additional ablations examine teacher choice, distillation location, and hyperparameter settings.

C Disentangling Temperature from Batch Size and Learnability

D Contribution of Lrkd and Choice of Teacher

  • DINOv2-Base gives the highest ViT-B/16 linear-probe accuracy (49.28), while MAE-Large gives the highest zero-shot accuracy (63.51) in single runs; ViT-L/14 linear probing is 56.91 vs. 56.99 for DINOv2-Base and Large.The results support Lrkd even with matched teacher and student capacity but do not select a teacher size; DINOv2-Large is used for uniformity.Source: Table 9 · p. 16, D Contribution of Lrkd and Choice of… · pp. 16–17

E Where to Apply RKD

  • Post-projection and pre-projection RKD perform comparably on ViT-B/16, with differences within about 0.8 points on zero-shot classification, retrieval, and linear probing.The MMVP difference is within seed-to-seed variance; the post-projection variant is adopted so all three losses act in the same embedding space.Source: Table 10 · p. 17, E Where to Apply RKD · p. 17

F Hyperparameter Selection Protocol

  • The temperature τ = 0.01 was selected on ViT-B/16 with CC3M and transferred unchanged to ViT-L/14, COCO Caption, and CC12M.Among tested loss ratios, 1.0:1.0:0.5 gives the best linear-probe accuracy and 1.0:1.0:1.0 the best zero-shot accuracy; the latter is the default for simplicity.Source: F Hyperparameter Selection Protocol · pp. 17–18

G Results After 1 or 2 epochs of Post-training

On ViT-B/16 post-trained with CC3M, extending ComCLIP from one to two epochs did not significantly improve overall performance, so the paper uses one epoch throughout.

G Results After 1 or 2 epochs of Post-training

  • Extending ComCLIP training from one to two epochs on ViT-B/16 with CC3M did not significantly improve overall performance.Table 12 compares both durations against the pretrained OpenAI CLIP baseline; MMVP is within seed-to-seed variance and is not highlighted.Source: Table 12 · p. 18, G Results After 1 or 2 epochs of… · p. 18

H Geometric Metric for Preserving the Pre-trained Image-Text Feature Space

The text-anchored KL diagnostic measures how well post-training preserves the image-text feature manifold, while training curves show how the contrastive loss responds to temperature. Small temperatures reduce drift and improve loss convergence, though the KL is not monotonic in temperature.

H Geometric Metric for Preserving the Pre-trained Image-Text Feature Space

  • The text-anchored KL compares image-feature distributions before and after post-training against frozen text features; lower values indicate better preservation of the contrastive-space manifold.The diagnostic is called Text-Anchored KL Divergence.Source: H Geometric Metric for Preserving the… · p. 18
  • Text Anchor KL falls from 5.50 at τ = 0.07 to 0.48 with the adopted small-τ mechanism, indicating less manifold drift relative to frozen text anchors.KL rises again at τ = 0.005, alongside a slight decline in zero-shot accuracy.Source: H Geometric Metric for Preserving the… · p. 18

I Training Curves of Lclip under Different τ Settings

  • With τ = 0.07, Lclip fails to reach a lower value within 3,000 steps; decreasing τ improves convergence, with logit_scale = 100 optimal.At logit_scale 142 and 200, Lclip increases slightly but still converges to a favorable level within 3,000 steps.Source: I Training Curves of Lclip under… · p. 18

J Detailed Zero-shot Experimental Results

Across 12 zero-shot classification benchmarks, ComCLIP and CLIP-Refine are at parity on ViT-B/16 in multi-seed results, while retrieval outcomes vary by architecture. Detailed results also report dataset-specific zero-shot performance and retrieval across two datasets.

J Detailed Zero-shot Experimental Results

  • 63.00±0.13 vs. 62.87±0.11 on 12-dataset zero-shot accuracy: ComCLIP and CLIP-Refine are at parity on ViT-B/16, with no significant difference.The reported values are multi-seed results; Table 14 provides per-dataset single-run results across the 12 benchmarks.Source: J Detailed Zero-shot Experimental… · p. 18, Table 14 · p. 19
  • Table 13’s Text Anchor KL reports how closely post-trained visual features preserve the structure of frozen text features, with lower values indicating less structural drift.The diagnostic covers models post-trained using only Lclip across temperature values.Source: Table 13 · p. 19
  • For Lclip training, logit_scale = 100 gives the most optimal loss-convergence curve among the temperatures shown.Figure 4 compares Lclip over training steps under different temperature settings.Source: Figure 4 · p. 19
  • On ImageNet-1K, CLIP-Refine trained on CC3M scores 61.59 versus 68.35 for original CLIP; ComCLIP varies by at most 0.2 across three post-training datasets.ComCLIP and CLIP-Refine trained on COCO Caption are at parity in average zero-shot accuracy on ViT-B/16.Source: J Detailed Zero-shot Experimental… · pp. 18–19

K Detailed Image-text Retrieval Experimental Results

L Detailed Linear Probe Experimental Results

The detailed linear-probe results compare model architectures, post-training datasets, and epochs across five tests. ComCLIP leads CLIP-Refine on ViT-B/16, while ViT-L/14 results vary by baseline and data overlap.

L Detailed Linear Probe Experimental Results

  • Table 16 covers five linear-probe tests across model architectures, training datasets, and epochs.The table presents detailed results for baselines and ComCLIP.Source: L Detailed Linear Probe Experimental… · p. 20
  • On ViT-B/16, ComCLIP exceeds CLIP-Refine in average linear-probe accuracy, with a significant multi-seed advantage that also holds in dataset-matched comparisons.ComCLIP also exceeds KUEA and CLIP-Refine across all three post-training datasets.Source: L Detailed Linear Probe Experimental… · p. 20
  • On ViT-L/14, KUEA trained on ImageNet-1K has the best average linear-probe accuracy; ComCLIP ranks second and varies by at most 1.3 across datasets.The passage notes that overlap between KUEA’s post-training data and the ImageNet-1K probe may help explain its result.Source: L Detailed Linear Probe Experimental… · p. 20

M Detailed MMVP Experimental Results

The MMVP results distinguish ViT-B/16 from ViT-L/14 comparisons with CLIP-Refine, and the small per-category sample size limits how single-run category values should be read.

M Detailed MMVP Experimental Results

  • On ViT-L/14, ComCLIP scores 24.20±0.86 on MMVP versus 19.01±1.86 for CLIP-Refine trained on the same data.These are multi-seed means; generative enhancement methods report higher MMVP on ViT-L/14.Source: M Detailed MMVP Experimental Results · p. 20
  • On ViT-B/16, ComCLIP (14.81±2.97) and CLIP-Refine (16.05±1.14) are indistinguishable on MMVP.The reported values are multi-seed means.Source: M Detailed MMVP Experimental Results · p. 20
  • MMVP evaluates 135 image pairs total and only 15 per category, so single-run per-category values move in steps of 6.67 points and should not be interpreted individually.Table 17 presents detailed MMVP results across model architectures, training datasets, and epochs.Source: M Detailed MMVP Experimental Results · p. 20

N Comparison with Generative-Model-Based Enhancement

This section compares generative-model enhancement methods with ComCLIP, describing their tradeoffs and ComCLIP’s scope, while also stating that the research involved neither human subjects nor a core non-standard LLM component.

N Comparison with Generative-Model-Based Enhancement

  • DIVA reports unchanged zero-shot accuracy (27-dataset average 69.3 →69.3; ImageNet-1K 75.5 →75.5) and slightly improved COCO retrieval, while GenHancer reports changes under 0.3 points on ImageNet-1K, SUN397, and COCO retrieval.The authors cannot resolve a third-party reproduction’s report of a large GenHancer ImageNet drop; un2CLIP reduces ImageNet-1K accuracy (75.5 →62.4) while improving text-to-image retrieval.Source: N Comparison with… · pp. 21–22
  • On MMVP, GenHancer (31.9) and un2CLIP (32.6) exceed ComCLIP (24.20±0.86), while DIVA (25.9) is comparable; base values differ across papers.Table 18 cautions that evaluation protocols and base values differ, so each row should be read relative to its own base.Source: Table 18 · p. 21, N Comparison with… · p. 22
  • ComCLIP’s supported contributions are its temperature diagnosis and improved linear-probe transferability, not maximizing fine-grained perception; it uses fixed text encoding without a generative model in the loop.A matched retraining comparison was outside the authors’ compute class, so Table 18 uses methods’ published OpenAI CLIP ViT-L/14 numbers.Source: N Comparison with… · pp. 21–22

O Broader Impacts

  • ComCLIP refines CLIP in one epoch on public data, reducing compute and energy costs versus training from scratch, and its frozen text encoder enables use in CLIP-based systems without retraining them.The paper notes potential downstream benefits for accessibility tools and visual question answering.Source: O Broader Impacts · p. 22
  • ComCLIP inherits biases from pretrained CLIP and DINOv2 and web-scraped CC3M/CC12M data, does not explicitly mitigate them, and could be misused for surveillance; practitioners are urged to evaluate fairness and safety.Source: O Broader Impacts · p. 22

NeurIPS Paper Checklist

  • The paper reports releasing code and post-trained checkpoints and providing experimental settings for reproducibility; main-claim comparisons use three-seed means and standard deviations with two-sided Welch t-tests.Remaining ablations are single runs due to compute cost, and the authors state they draw no conclusions from differences within seed variance.Source: NeurIPS Paper Checklist · pp. 24–25

10. Broader impacts

  • The paper reports no crowdsourcing or research involving human subjects.The response marks this item N/A.Source: §10 · p. 28
  • The paper states that its core method development does not involve LLMs as an important, original, or non-standard component.The LLM-usage declaration is marked N/A.Source: §10 · p. 29

Related papers

Loading 2610.11374v1…