Source-linked AI summary
EVA-CLIP: Improved Training Techniques for CLIP at Scale
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, Yue Cao
TL;DR
Scaling CLIP training is computationally expensive and can be unstable, motivating more efficient training techniques. EVA-CLIP combines improved initialization, optimization, token masking, and attention methods, achieving strong zero-shot results with reduced training costs. Its largest model reaches 82.0% ImageNet-1K zero-shot top-1 accuracy using 9 billion seen samples, while smaller models also perform strongly across broad benchmarks.
Problem
Training CLIP models at scale is computationally costly and can suffer from instability, despite CLIP’s strong zero-shot transferability.
Method
EVA-CLIP combines pre-trained EVA initialization, the LAMB optimizer, random image-token masking, and flash attention for CLIP training.
Results
82.0% zero-shot top-1 accuracy on ImageNet-1K val is achieved by the 5.0B-parameter EVA-02-CLIP-E/14+ using 9 billion seen samples.
Takeaways & Limitations
EVA-CLIP reduces computational cost while delivering strong zero-shot performance across image, video, and retrieval benchmarks.
Abstract
from arXiv · showhide
Contrastive language-image pre-training, CLIP for short, has gained increasing attention for its potential in various scenarios. In this paper, we propose EVA-CLIP, a series of models that significantly improve the efficiency and effectiveness of CLIP training. Our approach incorporates new techniques for representation learning, optimization, and augmentation, enabling EVA-CLIP to achieve superior performance compared to previous CLIP models with the same number of parameters but significantly smaller training costs. Notably, our largest 5.0B-parameter EVA-02-CLIP-E/14+ with only 9 billion seen samples achieves 82.0 zero-shot top-1 accuracy on ImageNet-1K val. A smaller EVA-02-CLIP-L/14+ with only 430 million parameters and 6 billion seen samples achieves 80.4 zero-shot top-1 accuracy on ImageNet-1K val. To facilitate open access and open research, we release the complete suite of EVA-CLIP to the community at https://github.com/baaivision/EVA/tree/master/EVA-CLIP.
1. Introduction
CLIP learns visual representations by bridging images and language through contrastive pre-training, but scaling its training remains computationally costly and unstable. EVA-CLIP addresses this challenge with techniques intended to improve training efficiency, stability, and zero-shot performance.
- CLIP bridges vision and language through large-scale contrastive image-text pre-training to learn visual representations.
- Training CLIP at scale remains difficult because large models and batches require substantial computation and can introduce instability.
- EVA-CLIP combines representation-learning, optimization, and augmentation techniques to reduce training costs, stabilize training, and improve zero-shot performance.
- 82.0% zero-shot top-1 accuracy on ImageNet-1K val is achieved by the 5.0B-parameter EVA-02-CLIP-E/14+ using 9 billion seen samples.
2. Approach
EVA-CLIP targets the computational demands and instability of large-batch CLIP training with initialization, optimization, token-masking, and attention techniques. These choices are associated with reduced computational cost, stabilized training, and strong zero-shot performance across benchmarks.
- Motivation: Large-batch CLIP training requires substantial resources and can become unstable as models scale.
- Results: EVA-CLIP achieves superior zero-shot performance across a broad range of benchmarks while significantly reducing computational cost.
- Better Initialization: Pre-trained EVA representations initialize the image encoder to combine contrastive semantic information with geometric and structural features from masked image modeling.
- Optimizer: LAMB is used for large-batch training, with adaptive elementwise updates and layer-wise learning rates intended to improve efficiency and convergence.
- Token Masking: Randomly masking 50% of image tokens halves time complexity and permits a 2× larger batch without additional memory costs.
3. Experiments
Experiments evaluate EVA-CLIP’s training design through ablations, computational measurements, architecture comparisons, and broad zero-shot benchmarks. The results show improved accuracy and efficiency across image, video, and retrieval tasks, with performance depending partly on model and text-encoder capacity.
- System-level Comparison: 80.9% averaged zero-shot top-1 accuracy across six benchmarks is achieved by EVA-02-CLIP-E/14+, surpassing OpenCLIP-G/14 by 4.7 points.On ImageNet, it surpasses OpenCLIP-G/14 by 1.9%.
- System-level Comparison: 77.5% averaged accuracy across 27 zero-shot image-classification benchmarks is achieved by EVA-02-CLIP-E/14+, while EVA-02-CLIP-L/14+ improves 1.2 points over OpenCLIP-H/14.The L/14+ model uses approximately half the model size and one-fifth the image-text pairs.
- Zero-shot Video Recognition: Video evaluation uses one center frame per video, making the reported UCF-101 and Kinetics results image-classification evaluations.The reported metrics are top-1 accuracy for UCF-101 and the mean of top-1 and top-5 accuracy for the Kinetics datasets.
- System-level Comparison: EVA-CLIP outperforms competitors at base and large sizes on zero-shot Flickr30K and COCO retrieval, while E/14+ improves retrieval over EVA-02-CLIP-E/14.The paper attributes retrieval differences partly to text-encoder capacity and training-sample count.
- Ablation Study: A 0.7% ImageNet zero-shot top-1 increase is obtained by replacing AdamW with LAMB at the same seen-sample count, while 50% masking provides a 2× speedup with a 0.7% accuracy decrease.The ablation evaluates EVA initialization on LAION-400M.
- Computation Costs: EVA-CLIP-B/16 converges within 300 hours on 16 NVIDIA 40GB-A100 GPUs, while EVA-CLIP-g/14 trains 12B samples in under 25 days on 64 such GPUs.The reported batch sizes are 32k and 65k, respectively.