PaperlayerView exact revision on arXiv

Source-linked AI summary

Reproducible scaling laws for contrastive language-image learning

Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, Jenia Jitsev

arXiv:2212.07143v2cs.LGcs.AIcs.CV

TL;DR

Prior scaling-law research largely focused on unimodal learning or private data and models, leaving contrastive language-image learning systematically understudied. This paper studies CLIP scaling with public LAION data and OpenCLIP across model, data, and training scales, finding power-law relationships across multiple downstream tasks and task-dependent differences between OpenCLIP and OpenAI CLIP.

  • Problem

    Prior empirical scaling-law research focused mainly on unimodal language or vision models and private data or models, leaving contrastive language-image learning without systematic investigation.

  • Method

    The study trains OpenCLIP models on LAION datasets while varying model scale, data scale, and samples seen, then evaluates zero-shot classification, retrieval, linear probing, and fine-tuning.

  • Results

    Power-law scaling appears across several downstream tasks, while OpenCLIP and OpenAI CLIP show distinct scaling advantages depending on the task and pre-training dataset.

  • Takeaways & Limitations

    Scaling behavior is task- and training-distribution-dependent, so conclusions from one CLIP training distribution may not transfer directly to another.

  • Takeaways & Limitations

    The study samples the scale space sparsely, cannot fully tune hyperparameters at larger scales, and has limited OpenAI CLIP measurements because WIT-400M is private.

Abstract

from arXiv · show

Scaling up neural networks has led to remarkable performance across a wide range of tasks. Moreover, performance often follows reliable scaling laws as a function of training set size, model size, and compute, which offers valuable guidance as large-scale experiments are becoming increasingly expensive. However, previous work on scaling laws has primarily used private data \& models or focused on uni-modal language or vision learning. To address these limitations, we investigate scaling laws for contrastive language-image pre-training (CLIP) with the public LAION dataset and the open-source OpenCLIP repository. Our large-scale experiments involve models trained on up to two billion image-text pairs and identify power law scaling for multiple downstream tasks including zero-shot classification, retrieval, linear probing, and end-to-end fine-tuning. We find that the training distribution plays a key role in scaling laws as the OpenAI and OpenCLIP models exhibit different scaling behavior despite identical model architectures and similar training recipes. We open-source our evaluation workflow and all models, including the largest public CLIP models, to ensure reproducibility and make scaling laws research more accessible. Source code and instructions to reproduce this study will be available at https://github.com/LAION-AI/scaling-laws-openclip

1 Introduction

This study addresses the lack of systematic, reproducible scaling-law evidence for contrastive language-image learning by using public data and open-source models. It evaluates how model, data, and compute scale relate to downstream performance and finds task-dependent scaling behavior across training distributions.

  • Prior scaling-law research focused mainly on private data and models or unimodal language and vision, leaving contrastive language-image learning insufficiently studied.
  • The study uses public LAION data and OpenCLIP to vary model, data, and samples-seen scale while evaluating classification, retrieval, linear probing, and fine-tuning.
  • The results motivate improving image-text datasets and evaluating them across downstream tasks because scaling behavior differs by task and pre-training source.
  • Scaling model size, data size, and samples seen leads to better zero-shot classification, with OpenAI WIT models showing stronger scaling than LAION-trained models.
  • LAION-trained models show stronger scaling than OpenAI WIT-trained CLIP models for zero-shot image retrieval, opposite to the classification trend.

2 Background and related work

Earlier scaling-law work established power-law relationships for language and vision models, including downstream transfer. In language-vision learning, private data and heterogeneous training procedures have limited systematic comparison and reproducibility.

  • Empirical studies connect model, data, and training-compute scale to performance through power-law relationships and enable predictions at larger scales.
  • Downstream-task scaling studies find benefits from increasing model, data, and training-compute scale, although upstream performance does not always correlate with downstream performance.
  • CLIP demonstrated strong zero-shot transfer and robustness to distribution shift using weakly aligned image-text data.
  • Prior language-vision studies often used private datasets, supervised pre-training, or heterogeneous multi-stage procedures, complicating conclusions about scale and reproduction.
  • Earlier open image-text datasets were substantially smaller than the private datasets used to train large-scale language-vision models.

3 Datasets and Methods

The authors build reproducible OpenCLIP scaling experiments with open LAION datasets while systematically varying model scale, data scale, and training duration. They use separate training schedules and distributed infrastructure to measure performance across these dimensions.

  • LAION-400M and LAION-5B provide open image-text data, with the English LAION-5B subset containing 2.32 billion samples referred to as LAION-2B.
  • The experiments vary visual encoders from ViT-B/32 through ViT-g/14, datasets from LAION-80M through LAION-2B, and samples-seen scales of 3B, 13B, and 34B.
  • Larger H/14 and g/14 model scales receive restricted measurements because compute constraints limit sampling density.
  • The training procedure uses larger global batch sizes, adapted learning rates, and control experiments to assess whether batch-size differences affect scaling measurements.
  • Each samples-seen scale is trained separately with a cosine-annealing schedule adapted to its duration, while selected hyperparameters are tuned for validation loss and stability.
  • OpenCLIP enables distributed training on supercomputers, with experiments using up to 1520 NVIDIA A100 GPUs.

4 Scaling laws for different downstream tasks

Across zero-shot transfer, retrieval, linear probing, and fine-tuning, increasing model, data, and compute scale generally improves downstream performance, often following power-law trends. Scaling behavior and bottlenecks depend on the task and pre-training data distribution.

  • Zero-shot transfer and robustness: Scaling model, data, and samples seen consistently improves zero-shot ImageNet accuracy, reaching 78% with ViT-H/14 trained on LAION-2B for 34B samples.Accuracy follows power laws, with the strongest reported result using the largest total pre-training compute.
  • Zero-shot transfer and robustness: α_openCLIP = −0.11 and α_CLIP = −0.16 for zero-shot top-1 ImageNet, while robustness coefficients are α_openCLIP = −0.13 and α_CLIP = −0.24.The OpenAI CLIP models show a scaling advantage over OpenCLIP for these tasks.
  • Zero-shot transfer and robustness: 79.1% top-1 accuracy is the predicted ImageNet performance for a well-tuned ViT-g/14 using 2B data and 34B samples seen.The power-law extrapolation predicts stronger performance at larger samples-seen scales.
  • Retrieval: OpenCLIP retrieval performance follows power laws and scales more favorably than WIT-400M CLIP on MS-COCO and Flickr30K.The reported coefficients are α_openCLIP = −0.08 and α_CLIP = −0.05 for MS-COCO, and α_openCLIP = −0.19 and α_CLIP = −0.10 for Flickr30K.
  • Linear probing: Increasing model, data, and scale consistently improves linear-probe accuracy on ImageNet, CIFAR100, and VTAB, including few-shot and full-data regimes.For models trained on the same distribution, scaling follows a linear trend on a log-log plot.
  • Fine-tuning: Fine-tuning accuracy improves with scale across downstream tasks, but the slope relating accuracy to pre-training compute varies by task and pre-training dataset.ImageNet fine-tuning compares OpenCLIP models trained on 2B data and 34B samples seen with OpenAI CLIP under matched evaluation settings.

5 Discussion

Scaling improves multiple downstream tasks, but the relationship depends on the scaling dimension, task, and pre-training dataset. The study also uses observed scaling laws for near-range performance predictions while noting important experimental limitations.

  • Power-law scaling yields consistent improvements in zero-shot classification, retrieval, linear probing, and fine-tuning as model, data, and training-sample scales increase.
  • Bottlenecks can hide dataset benefits: LAION-2B improves performance over LAION-400M when the number of samples seen reaches 34B.At lower sample counts, the larger dataset's benefit remains hidden; ViT-L/14 similarly requires LAION-2B to reveal gains from more samples seen.
  • A ViT-G/14 model trained on LAION-2B with 34B samples is predicted to reach 79.1% ImageNet top-1 accuracy, but extrapolation is limited by saturation effects.The authors state that extrapolation is more reliable near measured scales and that the estimate is modest relative to models using private labeled data and additional training stages.
  • OpenCLIP has stronger scaling for zero-shot retrieval, whereas OpenAI CLIP has stronger scaling for zero-shot ImageNet classification.The comparison involves LAION-400M/2B for OpenCLIP and private WIT-400M for OpenAI CLIP.
  • The authors hypothesize that dataset composition contributes to task-specific scaling differences between OpenCLIP and OpenAI CLIP.LAION-400M/2B was filtered using OpenAI ViT-B/32 similarity measurements, which may favor retrieval tasks.
  • The study's scaling-law evidence is constrained by sparse scale sampling, limited hyperparameter tuning, few OpenAI data points, and potentially undetected downstream duplicates.The authors rely on early-stage control experiments and prior tuning to suggest that each scale is not far from optimal.

6 Conclusion

The paper systematically studies how model, data, and sample scale affect contrastive language-image performance across tasks and adaptation methods. It finds power-law relationships, task-specific differences between OpenCLIP and OpenAI CLIP, and measurable performance at larger predicted scales.

  • The study examines scaling effects across zero-shot classification, retrieval, few- and full-shot linear probing, and fine-tuning.
  • Downstream performance follows a power-law relationship with model, data, and samples-seen scale across a broad range of settings.
  • OpenCLIP-LAION and OpenAI-WIT-400M models show distinct task-dependent scaling benefits.
  • Scaling-law predictions estimate 81.9% zero-shot top-1 ImageNet accuracy for ViT-G/14 trained from scratch on 68B image-text samples.
  • More data points at smaller and intermediate scales could clarify optimal configurations under fixed compute, while robustness scaling and separate encoder scaling remain open directions.

Supplementary: Reproducible scaling laws for contrastive language-image learning

The supplementary material describes the supercomputer infrastructure used for training, including its A100 GPU capacity and energy-efficiency context.

  • JUWELS Booster provides 3,744 NVIDIA A100 GPUs across 936 compute nodes, with four GPUs per node.Each A100 has 40 GB of memory and 19.5 TFLOP/s of FP64 Tensor Core performance.
  • The A100 GPUs reach peak efficiency of 48.75 GFLOP/(s W), and the machine ranked first in the Green500 list in November 2020.The comparison benchmark cited is 25 GFLOP/(s W) for the first 100 Top500 machines.

A.2 Scaling and training time

The supplementary experiments assess distributed OpenCLIP training efficiency and the implementation of sharded contrastive loss. They also document instability at larger model scales and its resolution through increased precision.

  • Scaling and training time: Distributed ViT-L/14 OpenCLIP training remains above approximately 84% efficiency at 1,024 GPUs, close to ideal linear scaling.Efficiency is defined as 100 × T(N)/(N×T(1)), with T(N) the measured throughput for N GPUs.
  • Sharding contrastive loss: CLIP's large InfoNCE batch requires computation sharding because similarity scores otherwise require O(N^2) memory per worker.The original CLIP training setup uses a batch size of 32,768, motivating distributed computation.
  • Sharding contrastive loss: Sharding reduces similarity-matrix memory by computing local features against all features, but differentiable all-gather is required for correct gradients.The differentiable torch.distributed.nn.functional.all_gather implementation is used instead of the standard nondifferentiable operation.
  • Scaling and training time: Figure 6a reports raw throughput, while Figure 6b reports speedup and efficiency relative to single-node training as GPU count varies.Each node contains four GPUs.
  • Training instabilities: NaN values and loss spikes emerged during ViT-L/14, H/14, and g/14 training as parameterization increased.
  • Training instabilities: Increasing precision resolved the observed instabilities: AMP with bfloat16 over float16, or float32 with tensor-float32, prevented the reported issues.Even smaller ViT-B models can become unstable when learning rate and batch size are sufficiently large.

B.1 Datasets employed in experiments.

The experiments use open LAION datasets for pre-training and a broad collection of downstream classification, retrieval, and fine-tuning datasets. A pHash-based check measures overlap between downstream data and LAION-400M.

  • Pre-training datasets: LAION-400M contains 414M English image-text pairs, while LAION-5B contains over 5.8B multilingual examples and includes the 2.32B-sample English LAION-2B subset.These datasets were assembled from Common Crawl links and filtered with an OpenAI ViT B/32 model.
  • Downstream datasets: Downstream evaluation covers 35 classification tasks, MS-COCO and Flickr30K retrieval, and ImageNet-12k and ImageNet-22k fine-tuning data.ImageNet-12k contains 12M training and 470K validation examples; ImageNet-22k contains 14M examples.
  • Dataset overlap: The duplication check applies perceptual hashing to LAION-400M and downstream images, then counts matching hashes.The check uses pHash’s discrete cosine transform method.
  • Dataset overlap: Around 1% of images were detected as duplicates overall, increasing to 3.80% for ImageNet-R and 5.15% for ImageNet-Sketch.Figure 7 illustrates duplicates under blurring, text blitting, color changes, cropping, and scaling, alongside false positives.

B.2.1 Predictions derived from scaling laws

The study extrapolates downstream performance by fitting power laws to Pareto-frontier models. Predictions cover larger-scale ImageNet classification and MS-COCO retrieval, while supplementary material documents fine-tuning and patched-model analyses.

  • Scaling-law extrapolation: Power-law fits on the Pareto frontier are used to predict performance at larger model and sample scales.Because trained models have misaligned GMAC budgets, the analysis bins compute and fits a line in log-log space to the best performance in each bin.
  • ImageNet predictions: 79.73% (+1.76%) zero-shot top-1 accuracy is predicted for H/14 at 68B samples seen on ImageNet.This is compared with the trained H/14 model at 34B samples seen.
  • MS-COCO predictions: 75.10% (+1.60%) image retrieval Recall@5 is predicted for H/14 at 68B samples seen on MS-COCO.The comparison is with the trained H/14 model at 34B samples seen.
  • Largest-scale predictions: 81.92% zero-shot top-1 accuracy on ImageNet and 76.99% image retrieval Recall@5 on MS-COCO are predicted for G/14 at 68B samples seen.For g/14 at 68B samples seen, the predictions are 75.85% (+3.45%) MS-COCO Recall@5 and a corresponding ImageNet prediction reported in the same extrapolation analysis.
  • Patching and supplementary analyses: Model patching interpolates fine-tuned and zero-shot weights using θpatched = (1 − α)θzero-shot + αθfine-tuned, with α selected under an ImageNet accuracy constraint.The coefficient is chosen to maximize average accuracy across eight downstream tasks while ImageNet accuracy decreases by no more than one percentage point.

B.2.3 Control experiments

Control experiments examine batch size, data-scale construction, trial variance, sampling strategy, and supplementary downstream results. Additional evidence compares scaling behavior across architectures, datasets, and adaptation methods.

  • Control experiments: Batch-size controls evaluate zero-shot ImageNet top-1 accuracy for LAION-400M at 13B samples seen using 32K–45K and 86K–88K global batches.The experiments test whether large distributed-training batch sizes affect the observed results.
  • Control experiments: A random 400M subset of LAION-2B is compared with LAION-400M at the same 13B-sample scale to assess whether they represent comparable data scales.LAION-400M and LAION-2B were independently composed from Common Crawl rather than one being a subset of the other.
  • Supplementary results: Supplementary tables report zero-shot, fine-tuned, patched, ImageNet robustness, VTAB+, and linear-probing results across the downstream evaluations.The fine-tuning and patching tables specify joint eight-task evaluation and frozen zero-shot classification heads.
  • Control experiments: Two identical ViT-B/32 pre-training trials show approximately 0.1% ImageNet zero-shot variation, smaller than variations observed when changing scale.Both trials use LAION-400M, 13B samples seen, and the same hyperparameters.
  • Scaling comparisons: OpenAI WIT-trained models scale more strongly for zero-shot classification, whereas LAION-trained OpenCLIP models scale more strongly for retrieval, despite architectural differences.This pattern holds for both OpenAI ResNets and ViTs trained on WIT compared with OpenCLIP models trained on LAION.

C Code and Data availability

The study makes its code, models, dataset-access instructions, and evaluation materials available to support reproduction and further analysis. The release is framed as a research resource with explicit safety and energy considerations.

  • Availability: Source code for experiments and figure production will be released at the project repository.The repository will also provide links to pretrained models and instructions for obtaining LAION-400M and LAION-5B.
  • Availability: All datasets used in the study are openly available and listed with references to their original work.The release materials support reuse of the study’s data and models.
  • Safety and impact: The released pretrained language-vision models are intended as research artefacts, not for end products or sensitive applications such as medical imaging or security surveillance.The paper notes potential misuse and calls for institutional rules governing sensitive applications.
  • Safety and impact: Large-scale pre-training has high computational energy costs, while releasing reusable models can reduce repeated training across downstream research.The paper also notes energy savings from data-efficient downstream transfer.
Loading 2212.07143v2…