PaperlayerView exact revision on arXiv

Source-linked AI summary

CLIPA-v2: Scaling CLIP Training with 81.1% Zero-shot ImageNet Accuracy within a \$10,000 Budget; An Extra \$4,000 Unlocks 81.8% Accuracy

Xianhang Li, Zeyu Wang, Cihang Xie

arXiv:2306.15658v1cs.CV

TL;DR

CLIP’s computational requirements hinder wider exploration, while CLIPA [12] showed that larger encoders can train with fewer input tokens. CLIPA-v2 extends that inverse scaling law to finetuning and scales experiments across models, data, and schedules; it reports 81.1% zero-shot ImageNet accuracy within $10,000, with 81.8% achievable after an additional $4,000.

  • Problem

    CLIP’s demanding computational requirements hinder its widespread exploration.

    Source: §1 · p. 1

  • Method

    CLIPA-v2 tests inverse scaling during full-resolution finetuning and scales CLIPA across model size, data, and training schedule.

    Source: §1 · p. 1

  • Results

    81.1% zero-shot ImageNet accuracy within $10,000 beats the best publicly available OpenCLIP model by 1.0% at ~39× lower computation cost; an additional $4,000 raises accuracy to 81.8%.

    Source: §1 · p. 2, §3 · p. 3

  • Takeaways & Limitations

    Applying the inverse scaling law during finetuning further reduces the computations required for CLIP training.

    Source: §3 · p. 2

Abstract

from arXiv · show

The recent work CLIPA presents an inverse scaling law for CLIP training -- whereby the larger the image/text encoders used, the shorter the sequence length of image/text tokens that can be applied in training. This finding enables us to train high-performance CLIP models with significantly reduced computations. Building upon this work, we hereby present CLIPA-v2 with two key contributions. Technically, we find this inverse scaling law is also applicable in the finetuning stage, enabling further reduction in computational needs. Empirically, we explore CLIPA at scale, extending the experiments up to the H/14 model with ~13B image-text pairs seen during training. Our results are exciting -- by only allocating a budget of \$10,000, our CLIP model achieves an impressive zero-shot ImageNet accuracy of 81.1%, surpassing the prior best CLIP model (from OpenCLIP, 80.1%) by 1.0% and meanwhile reducing the computational cost by ~39X. Moreover, with an additional investment of $4,000, we can further elevate the zero-shot ImageNet accuracy to 81.8%. Our code and models are available at https://github.com/UCSC-VLAA/CLIPA.

1. Introduction

CLIPA-v2 extends CLIPA’s inverse-scaling approach to finetuning and scales experiments across model size, data, and schedule. It reports 81.1% zero-shot ImageNet accuracy within a $10,000 budget, outperforming the best publicly available CLIP model from OpenCLIP while reducing computation cost.

1. Introduction

  • CLIPA [12] found that larger encoders can use fewer input tokens for training, addressing CLIP’s computational demands.The introduction frames CLIP’s computational requirements as a barrier to its wider exploration.Source: §1 · p. 1
  • The inverse scaling law also applies during full-resolution finetuning, further reducing CLIPA’s training cost.CLIPA-v2 investigates scaling up to H/14, DataComp-1B [6] and LAION-2B [22], and approximately 13B samples seen.Source: §1 · p. 1
  • Table 1 explores scaling data, model, and schedule, using 36 image and 8 text tokens for H/14 pretraining and 256 image and 32 text tokens for finetuning.The listed image resolutions are 84 × 84 for pretraining and 224 × 224 for finetuning.Source: Table 1 · p. 2
  • 81.1% zero-shot ImageNet accuracy within $10,000 beats the best publicly available OpenCLIP model by 1.0% and reduces computation cost by ~39×.An additional $4,000 raises accuracy to 81.8%.Source: §1 · p. 2

2. Background

CLIP’s strong zero-shot capability and versatility come with substantial training costs. CLIPA [12] uses an inverse scaling law to reduce those costs, and this work builds on CLIPA to improve efficiency and scale it up.

2. Background

  • CLIP’s training cost is a barrier for researchers seeking to train strong models, despite its zero-shot capability and versatility.CLIP’s success is associated with the scale of its data and model.Source: §2 · p. 2
  • This work builds on CLIPA [12], focusing on further efficiency and scaling it up.Source: §2 · p. 2

3. Experiments

The experiments test inverse scaling during finetuning, scale CLIPA across model size, data, and schedule, and compare CLIPA-v2 with other CLIP models on performance and computation cost. They report zero-shot ImageNet top-1 accuracy.

Inverse scaling law in the finetuning stage

  • At 50% token retention, H/14 drops 0.4% in performance, versus 0.8% for L/16, 1.1% for B/16, and 1.8% for S/16.Across the tested models, larger models show lower performance drops at the same input-token count; the experiments use random masking and 128M finetuning samples.Source: §3 · p. 2, Figure 2 · p. 2

Scaling up CLIPA

  • Scaling from L/14 to H/14 raises performance from 69.3% to 72.8%; LAION-2B adds 1.3%, and a fivefold schedule reaches 77.9% after approximately 13B samples.The scaled CLIPA H/14 model exceeds its FLIP counterpart by 0.3% while using one-third of the training budget.Source: §3 · pp. 2–3

Comparison with OpenCLIP

  • On DataComp-1B, CLIPA-v2 H/14 pretrained on 70 × 70 images reaches 81.1% zero-shot ImageNet accuracy within $10,000, 1.0% above OpenCLIP G/14’s 80.1%.Source: §3 · p. 3

Ablation

  • At 25% masking, random and block finetuning slightly improve performance over full-resolution finetuning, while higher masking ratios hurt and random masking performs best overall.The comparison uses H/14 on LAION-2B.Source: §3 · p. 3, Table 2 · p. 3
  • 30% masked random finetuning is the default, improving performance by 0.1% and speeding finetuning 1.3×; a 4× schedule adds 0.6%, with no substantial gains from longer schedules.The experiments also report progressive finetuning with a slight 0.2% performance improvement and 1.5× speedup over direct finetuning at 336 × 336.Source: §3 · p. 3

Comparison with OpenCLIP

  • 79.1% vs. 78.0% zero-shot ImageNet accuracy is reported for CLIPA-v2 H/14 and OpenCLIP H/14 on LAION-2B, with CLIPA-v2 using approximately 18× less training cost.The comparison also reports CLIPA-v2 H/14 at 81.1% on DataComp-1B within $10,000.Source: §3 · p. 3, Table 4 · p. 3
  • 81.8% zero-shot ImageNet accuracy follows an additional $4,000 investment in larger-resolution pretraining and progressive finetuning.Pretraining image size increases from 70 to 84, with progressive finetuning at 336 resolution.Source: §3 · p. 3
Loading 2306.15658v1…