PaperlayerView exact revision on arXiv

Source-linked AI summary

Bigger Text Encoders Can Hurt CLIP Zero-Shot Performance

Samir Char, Carles Domingo-Enrich, Randall Balestriero

arXiv:2609.05730v1cs.CVcs.AI

TL;DR

Prior CLIP scaling work treats total model size as a single variable, leaving the capacity split between encoders unexplored. This study independently scales vision and text encoders and examines zero-shot performance, overfitting, and embedding geometry. It finds that oversized text encoders can degrade zero-shot performance, while modality-specific weight decay can recover and improve it.

  • Problem

    Prior scaling studies treat total CLIP model size as one variable, leaving the effect of allocating capacity between vision and text encoders unexamined.

    Source: §1 · pp. 1–2

  • Method

    The study independently varies vision and text encoder sizes in CLIP models and evaluates zero-shot and linear-probe performance across downstream datasets.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    30–55% fewer total parameters match ViT-B/16 zero-shot performance in two architectures, while oversized text encoders degrade zero-shot performance for most vision encoders.

    Source: §1 · p. 2, §8 · p. 11

  • Takeaways & Limitations

    The findings identify parameter-efficient encoder allocations and modality-specific weight decay as ways to mitigate zero-shot degradation.

    Source: §1 · p. 2, §8 · p. 11

    Models were trained only from scratch on CC3M and CC12M and evaluated on classification and retrieval, leaving other pretraining and application settings untested.

    Source: §9 · p. 11

Abstract

from arXiv · show

Contrastive Language-Image Pretraining (CLIP) is a building block of many machine learning applications. Scaling laws have guided resource allocation for large-scale training, yet prior work treats total CLIP model size as a single variable, without exploring how the capacity split between encoders impacts downstream performance. Here, we train multiple CLIP models with different vision and text encoder sizes, revealing that for most vision encoders, there is an optimal text encoder size beyond which zero-shot performance degrades---even as total parameter count increases. Exploiting this behavior yields efficient configurations that match the zero-shot performance of the standard ViT-B/16 architecture with up to 55% fewer parameters. We further show that this degradation stems from overfitting induced by the oversized text encoder, and that using modality-specific weight decay coefficients not only recovers but improves performance across all degraded configurations. A geometric analysis reveals a trade-off in which scaling the text encoder improves embedding uniformity but worsens cross-modal alignment; we further show that these metrics are predictive of zero-shot performance. We hope these findings motivate CLIP architectures and training methods that counteract this degradation, a prerequisite for scaling CLIP reliably and efficiently.

1 Introduction

Prior CLIP scaling work leaves the capacity split between vision and text encoders unexamined, despite their different modality characteristics. This study independently scales both encoders and finds that oversized text encoders can hurt zero-shot performance, while modality-specific weight decay can recover performance.

1 Introduction

  • The capacity split is an unexamined design choice in prior CLIP scaling work, which treats total model size as a single variable.The authors also note that modalities can overfit and generalize at different rates.Source: §1 · pp. 1–2
  • The study trains a full grid crossing five vision encoders with six text encoders on CC3M and CC12M to measure zero-shot and linear-probe performance.The encoders span 10M–352M and 3M–145M parameters, respectively.Source: §1 · p. 2
  • 30–55% fewer total parameters match ViT-B/16 zero-shot performance in two architectures identified by exploiting asymmetric encoder scaling.For most vision encoders, performance degrades beyond an optimal text encoder size; this pattern is absent in linear probing.Source: §1 · p. 2
  • Increasing text-encoder weight decay recovers and can improve zero-shot performance, while increasing only vision-encoder weight decay degrades it.Equal decay underperforms text-only decay in most scenarios, or performs comparably.Source: §1 · p. 2
  • Scaling the text encoder improves image and text embedding uniformity but worsens image-text alignment; raising its weight decay mitigates this trade-off.The reported recovery of alignment preserves or improves image uniformity.Source: §1 · p. 2

2 Related Work

Prior work suggests encoder capacity allocation matters, but does not systematically isolate each encoder’s size in controlled contrastive pre-training. This paper connects that gap to CLIP scaling laws, heterogeneous regularization, and embedding-geometry diagnostics.

2 Related Work

  • Prior asymmetric-capacity evidence includes vision-only scaling, highly unequal encoder sizes, and text-only training, but not controlled isolation in contrastive pre-training.SmolVLM studies a vision-language model with a text decoder rather than contrastive pre-training.Source: §2 · p. 2
  • Prior CLIP scaling-law studies treat vision and text encoders as one parameter pool and do not derive specific scaling laws for their independent effects.Related work reports power-law and inverse-scaling findings, while this paper studies each encoder independently.Source: §2 · p. 2
  • Figure 1 frames the capacity-allocation gap with a grid of 30 architectures and reports that standard training can produce a rise, peak, and decline in zero-shot performance as text size grows.The figure caption says modality-specific weight decay removes the degradation.Source: §2 · p. 3
  • Heterogeneous weight-decay methods regularize parts of networks differently; this study uses modality-specific decay during scratch training to control encoder capacity and overfitting.Earlier CLIP work applied asymmetric decay to encoders initialized from pretrained weights.Source: §2 · p. 3
  • Alignment measures positive-pair embedding closeness, uniformity measures spread across the unit hypersphere, and RankMe estimates effective dimensionality.The paper uses these as diagnostics for scaling behavior and notes that prior work had not studied independent encoder scaling’s effect on embedding geometry.Source: §2 · p. 3

3 Methodology

The methodology constructs a controlled grid of CLIP encoder sizes and evaluates the resulting models across zero-shot and linear-probe tasks. Shared architecture settings and evaluation protocols are used to assess the effects of encoder scaling.

3.1 Asymmetric architecture grid

  • 30 configurations cross five vision encoders spanning 10M–352M parameters with six text encoders spanning 3M–145M parameters.The architectures span below and above the ViT-B/16 baseline, whose encoders have 86M and 63M parameters.Source: §3.1 · pp. 3–4, Table 1 · p. 4
  • All models share a 49,408-token vocabulary, 77-token context, 16-pixel patch size, 224 × 224 input resolution, and 512-dimensional shared embedding.Encoder scaling changes layers and attention heads while deriving width and MLP hidden dimension from those choices.Source: Table 1 · p. 4, §3.1 · p. 4
  • The 30 architectures are pretrained on CC3M and CC12M, yielding 60 models in total.Training hyperparameters follow Fan et al. (2023).Source: §3.1 · p. 4

3.2 Evaluation protocols

  • Evaluation covers 40 downstream datasets, including 28 zero-shot datasets corresponding to 30 tasks and 31 linear-probe tasks.Average performance uses each task’s preferred metric, rescaled to a percentage.Source: §3.2 · p. 4
  • Zero-shot classification compares image embeddings with class representations averaged across prompt templates, whereas linear probes train on frozen image features without text prompts.The zero-shot templates follow Radford et al. (2021).Source: §3.2 · p. 4

4 Larger text encoders hurt zero-shot performance

Across both pretraining datasets, most vision encoders have an optimal text-encoder size beyond which zero-shot performance declines, while linear-probe performance does not decline. Smaller text encoders also yield configurations matching ViT-B/16 zero-shot performance with fewer parameters.

4 Larger text encoders hurt zero-shot performance

  • Across CC3M and CC12M, most vision encoders show an optimal text-encoder size beyond which zero-shot performance degrades; the pattern is strongest for smaller vision encoders and fine-grained tasks.For the largest vision encoder, degradation is absent on CC12M, though larger text encoders might eventually induce it.Source: §4 · p. 5
  • On CC3M, the largest vision encoder does not achieve the best zero-shot performance, consistent with overfitting on the smaller dataset.This is an exception to the broader scaling trends.Source: §4 · p. 5
  • Unlike zero-shot performance, linear-probe performance does not degrade as the text encoder scales, instead improving or at worst saturating.Source: §4 · p. 6
  • Base–Nano and Tiny–Nano match or exceed Base–Base’s 37.7% average zero-shot performance at 38.1% and 37.9%, using 30.6% and 54.7% fewer parameters, respectively.Figure 3 plots average zero-shot performance against total parameters and marks the Base–Base baseline and Pareto frontier.Source: §4 · p. 6, Figure 3 · p. 6

5 Parameter allocation between encoders matters beyond total count

How CLIP capacity is divided between vision and text encoders predicts zero-shot performance better than total parameter count alone. The fitted model describes asymmetric scaling and a penalty for encoder-size mismatch, while the observed best pairings favor larger vision encoders and comparatively smaller text encoders.

5 Parameter allocation between encoders matters beyond total count

  • For every vision encoder except Colossal, the zero-shot-optimal text encoder is under 40M parameters; Colossal’s best pairing uses 145M text parameters.The optimal vision encoder for any text encoder is at least 194M parameters, and for four of six text encoders it is 352M.Source: §5 · p. 6
  • The vision–text parameter split explains more zero-shot-performance variance than total parameters alone: adjusted R2 rises from 0.312 to 0.693, and LOOCV R2 from 0.240 to 0.543.The total-parameter fit uses a linear-log model; the improved fit includes vision parameters, text parameters, and an interaction term.Source: §5 · p. 6
  • Removing the capacity-mismatch penalty fits substantially worse (LOOCV R2=0.826), underscoring the importance of appropriate encoder capacity; these findings are presented as an interpretable description of the data, not a definitive scaling law.Figure 4(b) shows predicted average zero-shot performance from the fitted regression across vision and text encoder sizes, with 30 architectures overlaid as points colored by observed performance.Source: §5 · p. 7, Figure 4 · p. 7
  • The fitted model treats vision capacity as a linear-log ceiling, text capacity as a saturating factor, and encoder-size mismatch as a penalty.The authors present this model as an interpretable description of their data, not a definitive scaling law.Source: §5 · p. 7

6 Modality-specific weight decay mitigates text-encoder overfitting

Oversized text encoders overfit for a fixed vision encoder, and modality-specific weight decay can mitigate the resulting zero-shot degradation. Increasing text-only weight decay improves degraded configurations, whereas increasing vision-only weight decay can worsen performance.

6 Modality-specific weight decay mitigates text-encoder overfitting

  • For a fixed vision encoder, larger text encoders lower CC12M training loss even as zero-shot performance declines beyond its optimum, indicating overfitting.Oversized text encoders also reach lower training loss but higher held-out loss on MSCOCO and Flickr30k.Source: §6 · p. 8
  • At λ=1.0, text-only weight decay yields 39.3% average zero-shot performance versus 38.5% for joint decay, while vision-only decay degrades performance by up to 0.7 pp.Text-only decay also surpasses Base–Tiny’s 38.2%, the best model using the Base vision encoder.Source: §6 · p. 9
  • With text weight decay λt=1.0 and vision weight decay λv=0.5, degraded Giant-text configurations gain +2.7 pp for Atto–Giant to +1.7 pp for Giant–Giant.Giant–Giant reaches 41.5%, matching Colossal–Giant’s 41.4% despite using a roughly half-sized vision encoder (194M vs. 352M parameters).Source: §6 · p. 9
  • Modality-specific weight decay improves zero-shot performance while increasing training loss in degraded configurations, consistent with reduced modality-specific overfitting.The paper’s stated principles are to favor larger vision than text encoders, size both appropriately, and account for capacity mismatch.Source: §6 · p. 9

7 Encoder scaling and regularization reshape embedding geometry

Text-encoder scaling trades improved embedding uniformity for worse image–text alignment, and this geometry reflects the observed zero-shot performance patterns. Alignment and image uniformity together also predict average zero-shot performance across the evaluated held-out datasets.

7 Encoder scaling and regularization reshape embedding geometry

  • Scaling the text encoder improves image and text uniformity but worsens image–text alignment; the alignment degradation lessens with larger vision encoders.Image uniformity saturates at the Base text encoder size of 63M parameters, while text uniformity continues to improve.Source: §7 · p. 10
  • Modality-specific weight decay improves image–text alignment while maintaining or improving image uniformity, typically at the expense of text uniformity.The reported geometry change applies to degraded configurations on MSCOCO and Flickr30k.Source: §7 · p. 10, Figure 7 · p. 10
  • Alignment and image uniformity predict average zero-shot performance with LOOCV R2=0.810 on MSCOCO and LOOCV R2=0.877 on Flickr30k.The leave-one-vision-out regression captures the overall trend, though it predicts degradation for Colossal that is not observed in practice.Source: §7 · p. 10
  • The alignment–uniformity trade-off coincides with zero-shot saturation and degradation, especially when the vision encoder is not large enough.Linear-probe inference depends only on the vision encoder, consistent with text scaling improving image uniformity without degrading linear-probe performance.Source: §7 · p. 10

8 Conclusion

The study finds that CLIP’s encoder capacity must be split appropriately: oversized text encoders can reduce zero-shot performance, while modality-specific weight decay can recover it. The authors also identify parameter-efficient configurations and directions for future work.

8 Conclusion

  • Two architectures match ViT-B/16 zero-shot performance while using 30–55% fewer total parameters.The authors report that oversized text encoders can degrade zero-shot performance, and modality-specific weight decay recovers and improves performance across degraded architectures.Source: §8 · p. 11
  • Oversized text encoders worsen image-text alignment while improving embedding uniformity; modality-specific weight decay recovers alignment while preserving or improving uniformity.The authors report that these geometric metrics predict zero-shot performance.Source: §8 · p. 11
  • Future work could test regularization or optimization changes, architectural modifications, richer captions, broader dataset scales and qualities, and heterogeneous modalities.The authors state that resolving the degradation could support more resilient scaling, while finding optimal encoder allocations could yield parameter-efficient recipes as long as it persists.Source: §8 · p. 11

9 Limitations

The study’s limitations concern its training data, evaluated tasks, initialization, weight-decay stability, and significance testing. The authors describe the primary impacts as positive and say the approach can lower compute barriers and reduce environmental footprint.

9 Limitations

  • The study trains only on CC3M and CC12M, evaluates classification and retrieval, and pre-trains from scratch; larger datasets, other applications, or initialized encoders may change the findings.Modality-specific weight decay sometimes caused training instability or collapse, and computational cost prevented significance tests across multiple seeds for all analyses.Source: §9 · p. 11

Broader Impact Statement

  • Asymmetrically scaling CLIP’s encoders can match or exceed strong baselines with fewer parameters, lowering the stated barrier to training from scratch under limited compute and reducing environmental footprint.The authors do not anticipate significant risks beyond those already associated with CLIP-style models.Source: Broader Impact Statement · p. 11

A Implementation details

The implementation varies encoder size by changing attention heads and layer counts while keeping per-head dimensions fixed, and applies training recipes to both datasets. Weight decay can be assigned separately to vision and text parameter groups.

A Implementation details

  • Encoder variants change attention-head and layer counts while keeping per-head dimension at 64 and setting the MLP hidden dimension to four times model width.The baseline pairs an 86M-parameter vision encoder with a 63M-parameter text encoder.Source: A Implementation details · p. 19
  • Both datasets use the same training recipes as Fan et al. (2023), with runs conducted on four to eight machines equipped with eight A100 GPUs each.Table A1 provides an overview of the pre-training hyperparameters for CC3M and CC12M.Source: A Implementation details · p. 19, Table A1 · p. 19
  • Weight decay excludes one-dimensional parameters, then groups remaining parameters into vision and text encoder sets, including each encoder’s embeddings, attention and MLP matrices, and projection.The vision group also includes patch-embedding convolutions; the text group includes token embeddings.Source: A Implementation details · p. 19
  • Vision and text groups use constant coefficients λv and λt, while baseline and joint-decay runs set λv = λt.Source: A Implementation details · p. 20

B.1 Pre-training datasets

Pre-training uses archived CC3M and CC12M versions with fewer samples than their original releases, and resizes images to a fixed shorter-side dimension. Evaluation datasets are organized by task and protocol, with exclusions for incompatible or near-random tasks.

B.1 Pre-training datasets

  • The archived CC3M and CC12M versions contain fewer samples than the original releases and may yield minor performance differences; the copies used retain 2.9 million and 10.9 million samples, respectively.The study uses versions downloaded and stored on Hugging Face in 2021.Source: §B.1 · p. 20
  • Both datasets’ images are resized so their shorter side equals 256 pixels.Source: §B.1 · p. 20

B.2 Downstream datasets

  • The evaluation tables describe dataset task types, sizes, classes, metrics, and zero-shot or linear-probe inclusion; dataset groups for per-group analysis are not exhaustive.The zero-shot evaluation excludes nine VTAB datasets and FER2013 because they are known to perform no better than random guessing.Source: §B.2 · p. 20, Table A2 · p. 21, Table A3 · p. 21
  • Linear-probe evaluation excludes Flickr30k and MSCOCO as retrieval datasets, plus several distribution-shift, scene-understanding, and ObjectNet datasets for incompatibility with the benchmark library.Source: §B.2 · p. 20

C.1 Robustness of the zero-shot degradation to the training seed

These analyses test the zero-shot degradation across seeds and training conditions, compare it with linear-probe behavior, and examine overfitting and embedding geometry. They also evaluate modality-specific weight decay and geometry metrics as ways to characterize or predict zero-shot performance.

C.1 Robustness of the zero-shot degradation to the training seed

  • For Base–Tiny versus Base–Giant, the three-seed paired test reports t(2)=3.29, p=0.04, Cohen’s dz=1.901, and a one-sided 95% CI of [+0.09, ∞).The passage describes Base–Tiny as the best configuration and Base–Giant as the degraded one.Source: Table A4 · p. 22, §C.1 · p. 22

C.2 Zero-shot and linear probe performance for sub-groups; C.4 Text encoder scaling drives linear probe gains without degradation

  • Across task groups, zero-shot performance peaks and degrades most on fine-grained tasks, while text scaling does not degrade linear probes; on CC3M, the two largest vision encoders overfit.For ImageNet and fine-grained groups, linear-probe performance saturates or improves monotonically as the text encoder scales.Source: Figure A1 · p. 23, Figure A2 · p. 24, §C.4 · p. 27

C.3 CLIP learning curves; C.6 Text encoder overfitting on held-out MSCOCO and Flickr30k datasets

  • On CC3M, validation accuracy saturates or declines after 20 epochs despite falling training loss; oversized Giant text encoders also have higher held-out than best-pairing contrastive loss.On CC12M, training loss and validation accuracy continue improving throughout training, and CC3M overfitting is milder with the Base vision encoder.Source: §C.3 · p. 24, §C.6 · p. 28

C.5 Predicting zero-shot performance from encoder size

  • Among five encoder-size models, the saturating-ceiling model with a capacity-mismatch penalty best predicts average zero-shot performance across the 30 CC12M architectures.The table reports in-sample, adjusted, and leave-one-out cross-validated R2 fit statistics.Source: §C.5 · p. 27, Table A5 · p. 28

C.7 Modality-specific weight decay

  • Raising text-encoder weight decay improves zero-shot performance, while raising vision-encoder weight decay degrades it; held-out validation loss falls even as training loss rises.The reported ablations include every degraded configuration with the Giant text encoder.Source: §C.7 · pp. 29–30, Table A7 · p. 30

C.8.1 Uniformity and Alignment; C.8.3 Impact of modality-specific weight decay on embedding geometry

  • Vision scaling improves alignment with larger text encoders but worsens it with the smallest; the weight-decay analysis compares geometry shifts across two held-out sets and both uniformity measures.The weight-decay comparison moves from λt=0.5, λv=0.5 to λt=1, λv=0.5.Source: Figure A10 · p. 32, §C.8.3 · p. 32, Figure A13 · p. 34

C.8.2 Predicting zero-shot performance from uniformity and alignment

  • No single geometry metric predicts zero-shot performance well, but image uniformity and alignment together explain most variance; the model generalizes to unseen vision encoders and captures the overall curve.Adding text uniformity does not help because it is collinear with image uniformity.Source: §C.8.2 · p. 32, Table A8 · p. 32

C.8.4 RankMe; C.8.5 Modality gap

  • Text scaling raises the effective rank of image and text embeddings, whereas modality gap generally falls as either encoder scales and does not reproduce zero-shot’s non-monotonic pattern.RankMe is less sensitive to vision-encoder size than to text-encoder size.Source: §C.8.4 · p. 35, Figure A14 · p. 35, §C.8.5 · p. 35
Loading 2609.05730v1…