PaperlayerView exact revision on arXiv

Source-linked AI summary

EVA-CLIP: Improved Training Techniques for CLIP at Scale

Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, Yue Cao

arXiv:2303.15389v1cs.CV

TL;DR

Scaling CLIP training is computationally expensive and can be unstable, motivating more efficient training techniques. EVA-CLIP combines improved initialization, optimization, token masking, and attention methods, achieving strong zero-shot results with reduced training costs. Its largest model reaches 82.0% ImageNet-1K zero-shot top-1 accuracy using 9 billion seen samples, while smaller models also perform strongly across broad benchmarks.

  • Problem

    Training CLIP models at scale is computationally costly and can suffer from instability, despite CLIP’s strong zero-shot transferability.

    Source: §1 · p. 1, §2 · p. 1

  • Method

    EVA-CLIP combines pre-trained EVA initialization, the LAMB optimizer, random image-token masking, and flash attention for CLIP training.

    Source: §1 · p. 1, §2 · pp. 2–3

  • Results

    82.0% zero-shot top-1 accuracy on ImageNet-1K val is achieved by the 5.0B-parameter EVA-02-CLIP-E/14+ using 9 billion seen samples.

    Source: §1 · p. 1

  • Takeaways & Limitations

    EVA-CLIP reduces computational cost while delivering strong zero-shot performance across image, video, and retrieval benchmarks.

    Source: §2 · p. 2, §3 · p. 4

Abstract

from arXiv · show

Contrastive language-image pre-training, CLIP for short, has gained increasing attention for its potential in various scenarios. In this paper, we propose EVA-CLIP, a series of models that significantly improve the efficiency and effectiveness of CLIP training. Our approach incorporates new techniques for representation learning, optimization, and augmentation, enabling EVA-CLIP to achieve superior performance compared to previous CLIP models with the same number of parameters but significantly smaller training costs. Notably, our largest 5.0B-parameter EVA-02-CLIP-E/14+ with only 9 billion seen samples achieves 82.0 zero-shot top-1 accuracy on ImageNet-1K val. A smaller EVA-02-CLIP-L/14+ with only 430 million parameters and 6 billion seen samples achieves 80.4 zero-shot top-1 accuracy on ImageNet-1K val. To facilitate open access and open research, we release the complete suite of EVA-CLIP to the community at https://github.com/baaivision/EVA/tree/master/EVA-CLIP.

1. Introduction

CLIP learns visual representations by bridging images and language through contrastive pre-training, but scaling its training remains computationally costly and unstable. EVA-CLIP addresses this challenge with techniques intended to improve training efficiency, stability, and zero-shot performance.

  • CLIP bridges vision and language through large-scale contrastive image-text pre-training to learn visual representations.Source: §1 · p. 1
  • Training CLIP at scale remains difficult because large models and batches require substantial computation and can introduce instability.Source: §1 · p. 1
  • EVA-CLIP combines representation-learning, optimization, and augmentation techniques to reduce training costs, stabilize training, and improve zero-shot performance.Source: §1 · p. 1
  • 82.0% zero-shot top-1 accuracy on ImageNet-1K val is achieved by the 5.0B-parameter EVA-02-CLIP-E/14+ using 9 billion seen samples.Source: §1 · p. 1

2. Approach

EVA-CLIP targets the computational demands and instability of large-batch CLIP training with initialization, optimization, token-masking, and attention techniques. These choices are associated with reduced computational cost, stabilized training, and strong zero-shot performance across benchmarks.

Motivation

  • Large-batch CLIP training requires substantial resources and can become unstable as models scale.Source: §2 · p. 1

Results

  • EVA-CLIP achieves superior zero-shot performance across a broad range of benchmarks while significantly reducing computational cost.Source: §2 · p. 2

Better Initialization

  • Pre-trained EVA representations initialize the image encoder to combine contrastive semantic information with geometric and structural features from masked image modeling.Source: §2 · pp. 2–3

Optimizer

  • LAMB is used for large-batch training, with adaptive elementwise updates and layer-wise learning rates intended to improve efficiency and convergence.Source: §2 · p. 3

Token Masking

  • Randomly masking 50% of image tokens halves time complexity and permits a 2× larger batch without additional memory costs.Source: §2 · p. 3

3. Experiments

Experiments evaluate EVA-CLIP’s training design through ablations, computational measurements, architecture comparisons, and broad zero-shot benchmarks. The results show improved accuracy and efficiency across image, video, and retrieval tasks, with performance depending partly on model and text-encoder capacity.

System-level Comparison

  • 80.9% averaged zero-shot top-1 accuracy across six benchmarks is achieved by EVA-02-CLIP-E/14+, surpassing OpenCLIP-G/14 by 4.7 points.On ImageNet, it surpasses OpenCLIP-G/14 by 1.9%.Source: §3 · p. 4
  • 77.5% averaged accuracy across 27 zero-shot image-classification benchmarks is achieved by EVA-02-CLIP-E/14+, while EVA-02-CLIP-L/14+ improves 1.2 points over OpenCLIP-H/14.The L/14+ model uses approximately half the model size and one-fifth the image-text pairs.Source: §3 · p. 4

Zero-shot Video Recognition

  • Video evaluation uses one center frame per video, making the reported UCF-101 and Kinetics results image-classification evaluations.The reported metrics are top-1 accuracy for UCF-101 and the mean of top-1 and top-5 accuracy for the Kinetics datasets.Source: §3 · p. 4

System-level Comparison

  • EVA-CLIP outperforms competitors at base and large sizes on zero-shot Flickr30K and COCO retrieval, while E/14+ improves retrieval over EVA-02-CLIP-E/14.The paper attributes retrieval differences partly to text-encoder capacity and training-sample count.Source: §3 · p. 4

Ablation Study

  • A 0.7% ImageNet zero-shot top-1 increase is obtained by replacing AdamW with LAMB at the same seen-sample count, while 50% masking provides a 2× speedup with a 0.7% accuracy decrease.The ablation evaluates EVA initialization on LAION-400M.Source: §3 · p. 5

Computation Costs

  • EVA-CLIP-B/16 converges within 300 hours on 16 NVIDIA 40GB-A100 GPUs, while EVA-CLIP-g/14 trains 12B samples in under 25 days on 64 such GPUs.The reported batch sizes are 32k and 65k, respectively.Source: §3 · p. 5

Related papers

Loading 2303.15389v1…