PaperlayerView exact revision on arXiv

Source-linked AI summary

An Inverse Scaling Law for CLIP Training

Xianhang Li, Zeyu Wang, Cihang Xie

arXiv:2305.07017v2cs.CV

TL;DR

CLIP training is prohibitively costly, limiting its broader exploration. This paper studies inverse scaling in CLIP and introduces CLIPA, which reduces token lengths as encoder size grows. CLIPA’s G/14 model reaches 83.0% zero-shot top-1 ImageNet-1k accuracy while training about 33× faster than OpenCLIP.

  • Problem

    CLIP’s prohibitively high training cost limits wider exploration of the model.

    Source: §1 · p. 1

  • Method

    CLIPA studies inverse scaling by reducing image and text token lengths as encoder size grows, using image resizing and syntax masking as effective strategies.

    Source: §1 · p. 2, §8 · p. 10

  • Results

    CLIPA G/14 reaches 83.0% zero-shot top-1 ImageNet-1k accuracy and trains about 33× faster than OpenCLIP.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings support faster CLIP training, including for researchers with limited computational resources.

    Source: §8 · p. 10

    On ARO relational-understanding benchmarks, CLIPA-B/16 slightly trails OpenCLIP-B/16, and both models remain somewhat limited.

    Source: §7 · p. 9

Abstract

from arXiv · show

CLIP, one of the pioneering foundation models that connect images and text, has enabled many recent breakthroughs in computer vision. However, its associated training cost is prohibitively high, imposing a significant barrier to its widespread exploration. In this paper, we present a surprising finding that there exists an inverse scaling law for CLIP training, whereby the larger the image/text encoders used, the shorter the sequence length of image/text tokens that can be applied in training. Moreover, we showcase that the strategy for reducing image/text token length plays a crucial role in determining the quality of this scaling law. As a result of this finding, we are able to successfully train CLIP even with limited computational resources. For example, using 8 A100 GPUs, our CLIP models achieve zero-shot top-1 ImageNet-1k accuracies of 63.2% in ~2 days, 67.8% in ~3 days, and 69.3% in ~4 days. Our method also works well when scaling up -- with G/14, we register a new record of 83.0% ImageNet-1k zero-shot accuracy, and meanwhile accelerate the training by ~33x compared to its OpenCLIP counterpart. By reducing the computation barrier associated with CLIP, we hope to inspire more research in this field, particularly from academics. Our code is available at https://github.com/UCSC-VLAA/CLIPA.

1 Introduction

CLIP training costs restrict broader exploration, despite the model’s role in image-text research. The paper finds that larger encoders can use shorter token sequences and introduces CLIPA for efficient training at scale.

1 Introduction

  • Replicating OpenCLIP-B/32’s 62.9% zero-shot top-1 ImageNet-1k accuracy requires 36 hours on 128 A100 GPUs, limiting broader CLIP exploration.The paper links this barrier to scaling costs that make replication and follow-on research challenging.Source: §1 · pp. 1–2
  • L/16 can use image/text token lengths of 50/6 versus S/16’s 101/16 while keeping zero-shot ImageNet-1k accuracy within 1% of full-resolution training.The paper finds that token-reduction strategies preserving more semantic information tend to yield better scaling effects.Source: §1 · p. 2, Figure 1 · p. 2
  • With eight A100 GPUs, CLIPA-L/16 reaches 67.8% zero-shot top-1 ImageNet-1k accuracy in about three days; CLIPA G/14 reaches 83.0% and trains about 33× faster than OpenCLIP.CLIPA-L/16’s GPU-hour cost is over 16× lower than OpenCLIP-B/16’s in the reported comparison.Source: §1 · p. 2
  • The authors hope lower training costs will encourage researchers with limited computational resources to explore CLIP and foundation-model training.They frame the work as a way to broaden participation in research.Source: §1 · p. 2

2 Related Works

Prior work has pursued efficient CLIP training through data reduction and token reduction. This paper builds on FLIP while examining semantic-preserving strategies and CLIP’s token-length scaling behavior.

2 Related Works

  • Efficient CLIP training work has reduced or curated data, while FLIP removes 50–75% of image patches for faster training [21] [29].The authors describe their approach as based on FLIP.Source: §2 · pp. 2–3
  • The paper extends FLIP by exploring semantic-preserving token-reduction strategies and reducing inputs to as few as 17 image tokens and 8 text tokens, with acceleration up to 25×.The authors distinguish this token-level approach from data-focused efficiency methods.Source: §2 · pp. 2–3
  • Unlike language-model scaling analyses that use fewer training samples or compare model sizes, this work examines performance drops for the same CLIP model across token lengths.The paper positions its analysis as a distinct way to study sample efficiency in image-text training.Source: §2 · p. 3

3 Reducing Image/Text Tokens

The study evaluates eight strategies—four each for images and text—because their effects in CLIP differ from their effects in masked image or language modeling. The experiments compare token-reduction approaches under a common training setup.

3 Reducing Image/Text Tokens

  • The study compares four image-reduction strategies and four text-reduction strategies, whose effects in CLIP differ from their effects in masked image or language modeling.The evaluated methods include random, grid, and block masking for images, and truncation, random masking, block masking, and syntax masking for text.Source: §3 · p. 3, §3.3 · p. 4

3.1 Training Setup

  • The training setup follows FLIP, using LAION-400M for 6.4 epochs and a 0.36-epoch full-resolution fine-tuning stage, with batch size 32k.The experiments use a vanilla ViT visual encoder and a non-autoregressive Transformer text encoder.Source: §3.1 · p. 3

3.2 Image

  • At 75% image masking, block masking achieves 68.5% zero-shot top-1 ImageNet-1k accuracy, versus 67.6% for random masking and 67.3% for grid masking.This differs from masked image modeling, where random masking is generally most effective; the authors suggest structured patterns may better support CLIP’s information-extraction objective.Source: §3.2 · pp. 3–4
  • Resizing images to 112×112, computationally equivalent to 75% masking, yields 68.9% zero-shot top-1 ImageNet-1k accuracy for L/16.The authors use anti-aliasing bilinear interpolation to preserve image quality and report that resizing surpasses the mask strategies.Source: §3.2 · p. 4

3.3 Text

  • At a maximum text length of 8, truncation scores 68.2% versus 67.8% for random masking, while block masking also scores 68.2%.Block masking preserves consecutive text sequences and outperforms random masking by 0.4%.Source: §3.3 · p. 4
  • With text length capped at 8, syntax masking achieves 69.0% zero-shot top-1 ImageNet-1k accuracy, the best result among the tested text strategies.It prioritizes retaining nouns, then adjectives, then other words.Source: §3.3 · p. 4

4 An Inverse Scaling Law

The section finds that larger CLIP encoders can use shorter image and text token sequences with smaller performance drops, though the effect depends on the reduction strategy. It also reports the pattern for ConvNeXt models.

4.1 Image

  • ImageNet-1k accuracy generally decreases as image-token length shrinks, with block masking as an exception: L/16 accuracy rises 0.3% when tokens fall from 197 to 99.The image study compares random masking, block masking, image resizing, and grid masking while holding the maximum text length at 32.Source: §4.1 · pp. 4–5
  • With image resizing from 197 to 17 tokens, the ImageNet-1k performance drop falls from 6.2% for S/16 to 4.3% for B/16 and 3.0% for L/16.Across all four image-token reduction strategies, drops become smaller as model size increases; more information-preserving resizing permits shorter sequences than random masking at the same 2% drop threshold.Source: §4.1 · p. 5
  • On COCO retrieval with random masking from 197 to 17 image tokens, image/text retrieval drops shrink from 6.6%/7.1% for S/16 to 5.8%/5.9% for B/16 and 4.6%/4.1% for L/16.Robustness evaluations on ImageNet-V2, ImageNet-R, ImageNet-A, and ImageNet-Sketch show the same trend in most cases: larger models have smaller relative drops.Source: §4.1 · p. 5

4.2 Text

  • With text length 4, scaling from S/16 to L/16 reduces performance drops for truncation, syntax masking, block masking, and random masking, respectively, from 5.7% to 5.2%, 3.4% to 2.0%, 4.3% to 2.9%, and 5.9% to 5.1%.Syntax masking is most effective at very short lengths; sensitivity stays relatively low through 8 tokens, then drops abruptly, while retrieval shows little drop at 16 tokens and no evident scaling benefit at 8 or fewer.Source: §4.2 · pp. 6–7

4.3 ConvNeXt

  • At a 1.5% performance-drop threshold, ConvNeXt-T requires 112 × 112 inputs, whereas ConvNeXt-B works with 96 × 96 inputs, confirming inverse scaling for this architecture.The study uses image resizing as a viable ConvNeXt strategy because masking offers only modest speedup without efficient sparse-convolution support.Source: §4.3 · p. 7

5 Training CLIP with Limited Resources

CLIPA trains CLIP with reduced image and text token lengths before full-length fine-tuning, targeting limited compute. On an 8-A100 machine, it compares favorably with OpenCLIP in accuracy and GPU hours, and larger models further improve the compute-accuracy trade-off.

5 Training CLIP with Limited Resources

  • L/16 with 17 image tokens outperforms standard B/16 with 197 tokens by 2.5% while achieving a 1.7× speedup.Reducing text tokens can further accelerate training, especially with a large text encoder such as H/14.Source: §5 · p. 8
  • CLIPA uses image resizing and text truncation for reduced-token pre-training, followed by full-token fine-tuning, with training on an 8-NVIDIA-A100 machine.The setup also adds color-jitter and grayscale augmentation and uses global average pooling in ViT.Source: §5 · p. 8, Table 1 · p. 8
  • CLIPA-B/16 exceeds OpenCLIP-B/32 by 0.3% in zero-shot ImageNet-1k classification with ~10× fewer GPU hours, while CLIPA-L/16 exceeds OpenCLIP-B/16 by 0.7% with 17× fewer GPU hours.CLIPA-B/16 trains in ~2 days and CLIPA-L/16 in ~3 days on an 8-A100 machine.Source: §5 · p. 8, Table 1 · p. 8
  • With image size 84 and text length 8, CLIPA-H/14 reaches 72.8% zero-shot top-1 ImageNet-1k accuracy, on par with OpenCLIP-L/14 at ~25× lower computational cost.Reducing text length from 32 to 8 yields an additional ~1.3× training speedup.Source: §5 · p. 9

6 CLIPA at Scale

At larger scale, CLIPA matches OpenCLIP-H/14 on LAION-2B with substantially less computation and reports an open-source ImageNet-1k zero-shot record with G/14. The section also reports results on VTAB.

6 CLIPA at Scale

  • Extending H/14 fine-tuning from 128M to 512M samples at 224 × 224, then adding 128M samples at 336 × 336, improves accuracy from 77.7% to 79.1%.The updated schedule uses random masking at both resolutions, reducing training overhead with little-to-no performance drop.Source: §6 · p. 9
  • On LAION-2B at 224 × 224 resolution, CLIPA-H/14 matches OpenCLIP-H/14 with ~1/15 of its training computation and exceeds FLIP-H/14 at 79.1% ImageNet-1k accuracy while being 6× faster.The reported OpenCLIP training cost is ~5,600 GPU-days, and the stated CLIPA saving is ~5,230 GPU-days.Source: §6 · p. 9
  • On DataComp-1B, CLIPA-G/14 reaches 83.0% open-sourced ImageNet-1k zero-shot accuracy with ~33× less computation than OpenCLIP-G/14.The models use the same number of seen samples from the dataset.Source: §6 · p. 9
  • On the highly diverse and challenging VTAB vision-task benchmark, CLIPA-H/14 achieves comparable or even superior performance while requiring significantly less training cost, demonstrating good generalizability.Source: §6 · p. 9

7 Limitation

The reported limitation concerns CLIP’s relational, attribute, and order understanding: on ARO, OpenCLIP-B/16 slightly outperforms CLIPA-B/16, and both remain limited. The authors point to NegCLIP as a lightweight fine-tuning strategy, while leaving the initial ARO underperformance for future work.

7 Limitation

  • On ARO, OpenCLIP-B/16 slightly outperforms CLIPA-B/16, and both models remain limited in capturing relationships, attributes, and order information.The authors leave CLIPA’s initial ARO underperformance for future work, although NegCLIP nearly doubles both models’ performance on COCO-Order and Flickr30k-Order.Source: §7 · pp. 9–10
  • NegCLIP is described as a lightweight, additional fine-tuning stage for text understanding, with near-doubling reported on COCO-Order and Flickr30k-Order for both OpenCLIP and CLIPA.The supplied VTAB table caption identifies a separate zero-shot top-1 accuracy comparison on LAION-2B-trained models.Source: §7 · p. 10, Table 3 · p. 10

8 Conclusion

The paper identifies an inverse scaling law for CLIP: larger models require fewer input tokens during training. It finds token-reduction strategy matters, with image resizing and text syntax masking giving the best overall scaling quality.

8 Conclusion

  • Larger CLIP models require fewer input tokens during training, and the choice of token-reduction strategy affects scaling quality.Resizing image inputs and syntax masking text provide the best overall scaling quality among the eight strategies studied.Source: §8 · p. 10
  • Image resizing and text syntax masking yield the best overall scaling quality among the eight token-reduction strategies studied.The paper links this result to the importance of preserving semantic information during efficient CLIP training.Source: §8 · p. 10
  • The authors hope the findings encourage researchers with limited computational resources to explore CLIP training.Source: §8 · p. 10

9 Broader Impact

The authors say inverse scaling makes proficient CLIP training possible on a modest budget and reduces training compute. They also caution that models trained on public web-scale data may reflect the data’s biases or harmful content.

9 Broader Impact

  • The authors say inverse scaling enables proficient CLIP models to be trained on a modest budget.Source: §9 · p. 10
  • The method curtails tens of thousands of GPU/TPU hours, with implications for energy consumption and carbon emissions.Source: §9 · p. 10
  • Because training uses publicly available web-scale datasets, model weights may reflect biases or harmful content in those datasets.The authors advise care when interpreting model outputs and deploying them in real-world applications.Source: §9 · p. 10

A Implementation Details

The implementation uses multiple visual model scales and specified pre-training and fine-tuning configurations. It evaluates zero-shot ImageNet-1k accuracy and presents an alternative view showing that larger models lose less performance when trained with fewer tokens.

A.1 Architectures

  • The visual encoders use S/16, B/16, and L/16 scales at a shared patch size, while CLIPA uses ViT with global average pooling.The text encoder is a non-autoregressive Transformer with a WordPiece tokenizer; shorter inputs are zero-padded to the model’s maximum length.Source: §A.1 · p. 14, Table 4 · p. 14

A.2 Hyper-parameters

  • Pre-training uses an 8e-6 base learning rate and a 32k batch size, with color jitter and grayscale augmentation.The reported color jitter strength is 0.32 at probability 0.8, and grayscale augmentation probability is 0.2.Source: §A.2 · p. 14, Table 5 · p. 14
  • Fine-tuning uses full-resolution 224 × 224 images, text up to length 32, and 4000 steps with an 800-step warm-up.Due to limited resources, CLIPA-B/16 and CLIPA-L/16 use fine-tuning batch sizes of 8k and 7k; an 8k batch is reported as competitive with 32k.Source: §A.2 · p. 14, Table 6 · p. 15
  • The paper implements JAX and PyTorch codebases; most experiments use TPU-V3, while CLIPA-B/16 and CLIPA-L/16 use A100 GPUs.Source: §A.2 · p. 14

A.3 Evaluation Setting

  • ImageNet-1k zero-shot evaluation uses 80 prompt templates and OpenCLIP benchmarking, with center-cropped 224 × 224 inputs or resized 336 × 336 inputs.Source: §A.3 · p. 15

B.1 Alternative view of inverse scaling law

  • Larger models show a significantly smaller performance decrease when fewer tokens are used for pre-training.Figure 9 plots model size against performance drop with separate lines for input image-token counts; Figure 10 plots total pre-training tokens against accuracy.Source: Figure 9 · p. 15, Figure 10 · p. 15, §B.1 · p. 16

B.2 Ablation

Ablations report that scaling model size, data, and training schedule improves CLIPA performance, while fine-tuning choices affect accuracy and speed. ImageNet-1k analyses also show that fine-tuning improves results after token reduction and narrows the pre-training gap between ConvNeXt and ViT.

B.2 Ablation

  • Global average pooling improves ImageNet-1k zero-shot accuracy by approximately 1.5%, and stronger augmentation adds approximately 0.8%.Together, they yield a reported 1.8% improvement; fine-tuning CLIPA-L/16 at batch size 7k causes only a marginal performance decrease.Source: §B.2 · p. 16
  • 77.9% performance is reached after a fivefold longer schedule and approximately 13B seen samples; CLIPA-H/14 exceeds its FLIP counterpart by 0.3% with one-third the training budget.Scaling L/16 to H/14 raises performance from 69.3% to 72.8%, and switching from LAION-400M to LAION-2B adds another 1.3%.Source: §B.2 · p. 16
  • At 25% masking, random and block fine-tuning slightly improve performance over full-resolution fine-tuning, while higher masking ratios perform worse.Random masking consistently outperforms grid and block masking in the reported comparison.Source: §B.2 · p. 17, Table 9 · p. 17
  • 30% random masking gives a 0.1% performance improvement and 1.3× fine-tuning speedup, while a 4× schedule adds 0.6%.Further schedule increases produce no substantial performance gains.Source: Table 10 · p. 17, §B.2 · p. 18
  • Progressive fine-tuning improves performance by 0.2% and speeds fine-tuning by 1.5× compared with direct fine-tuning at 336 × 336.The schedule uses 224 × 224 inputs with 30% masking for the first 512M samples, then 336 × 336 inputs with 40% masking for 128M samples.Source: Table 10 · p. 17, §B.2 · p. 18

B.3 ImageNet-1k

  • Fine-tuning pre-trained models with 17 image tokens yields gains of 18.4%, 17.0%, and 18.6% for S/16, B/16, and L/16 with block masking.Source: Table 11 · p. 17, §B.3 · p. 18
  • Text inputs shorter than 8 tokens have noticeably lower pre-training performance because evaluation prompts are often longer than 8 tokens.Fine-tuning with a maximum text length of 32 significantly improves model performance.Source: Table 12 · p. 17, §B.3 · p. 18
  • At 64 × 64 input size, ConvNeXt-B outperforms ViT-B by approximately 3.5% before fine-tuning, but the performance gap narrows after fine-tuning.Source: §B.3 · p. 18, Table 13 · p. 18

B.4 Zero-shot retrieval and robustness

This section reports how varying text-token lengths relates to zero-shot image/text retrieval and robustness. The evaluations cover COCO retrieval and robustness benchmarks from the ImageNet family.

B.4 Zero-shot retrieval and robustness

  • Zero-shot image/text retrieval is evaluated on COCO.The passage points to Table 15 for the results but supplies no performance values.Source: §B.4 · p. 19
  • Zero-shot robustness is evaluated on ImageNet-V2, ImageNet-R, ImageNet-A, and ImageNet-Sketch.The passage points to Table 15 for the results but supplies no performance values.Source: §B.4 · p. 19
  • The section studies the scaling effect of text-token length on zero-shot image/text retrieval and robustness benchmarks.Table 15 presents these evaluations.Source: Table 15 · p. 19, §B.4 · p. 19
Loading 2305.07017v2…