PaperlayerView exact revision on arXiv

Source-linked AI summary

EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameters

Quan Sun, Jinsheng Wang, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Xinlong Wang

arXiv:2402.04252v1cs.CV

TL;DR

Vision foundation models used in multimodal systems remain much smaller than LLMs. This paper scales CLIP through EVA’s weak-to-strong approach; its 18-billion-parameter model reaches 80.7% average zero-shot top-1 accuracy across 27 image-classification benchmarks, with scaling performance still improving.

  • Problem

    Vision foundation models continue to operate at a much smaller scale than LLMs, although they serve as foundational vision encoders for multimodal models.

    Source: §1 · p. 1

  • Method

    The authors use EVA’s weak-to-strong cycle, distilling a large EVA from a small EVA-CLIP and using it to initialize a larger EVA-CLIP.

    Source: §2 · p. 2

  • Results

    80.7% average zero-shot top-1 accuracy across 27 image-classification benchmarks, exceeding DFN5B-CLIP-H/14+ by +1.5% and InternVL-C by +2.7%.

    Source: §3 · p. 3

  • Takeaways & Limitations

    EVA-CLIP-18B demonstrates the potential of EVA-style visual scaling, and the authors release code and weights to encourage further research.

    Source: §1 · p. 2, §5 · p. 5

    The scaling study keeps the training dataset largely fixed; the authors note that scaling datasets could further unleash the method’s scaling power.

    Source: §2 · p. 2

Abstract

from arXiv · show

Scaling up contrastive language-image pretraining (CLIP) is critical for empowering both vision and multimodal models. We present EVA-CLIP-18B, the largest and most powerful open-source CLIP model to date, with 18-billion parameters. With only 6-billion training samples seen, EVA-CLIP-18B achieves an exceptional 80.7% zero-shot top-1 accuracy averaged across 27 widely recognized image classification benchmarks, outperforming its forerunner EVA-CLIP (5-billion parameters) and other open-source CLIP models by a large margin. Remarkably, we observe a consistent performance improvement with the model size scaling of EVA-CLIP, despite maintaining a constant training dataset of 2-billion image-text pairs from LAION-2B and COYO-700M. This dataset is openly available and much smaller than the in-house datasets (e.g., DFN-5B, WebLI-10B) employed in other state-of-the-art CLIP models. EVA-CLIP-18B demonstrates the potential of EVA-style weak-to-strong visual model scaling. With our model weights made publicly available, we hope to facilitate future research in vision and multimodal foundation models.

1. Introduction

Vision encoders used in multimodal models remain much smaller than LLMs. EVA-CLIP-18B scales CLIP to 18 billion parameters, reports strong zero-shot results, and makes its code and weights available to support further research.

1. Introduction

  • Vision foundation models remain much smaller than LLMs scaled to around 100B parameters or higher, despite their foundational role in multimodal models.CLIP models provide visual representations, while LLMs serve as a general interface for reasoning across modalities.Source: §1 · p. 1
  • 80.7% average zero-shot top-1 accuracy across 27 image-classification benchmarks; EVA-CLIP-18B surpasses EVA-02-CLIP-E/14+ and other open-source CLIP models.Table 2 reports top-1 accuracy across the 27 datasets.Source: §1 · p. 1, Table 2 · p. 2
  • EVA-CLIP performance consistently improves with scaling, with no observed sign of saturation.Figure 1 presents scaling behavior averaged across 27 image-classification benchmarks.Source: Figure 1 · p. 1, §1 · p. 2
  • The authors open-source the training code and model weights to encourage research and development in vision and multimodal foundation models.Source: §1 · p. 2

2. Weak-to-Strong Vision Scaling

EVA-CLIP scaling follows a weak-to-strong cycle: a large EVA model learns from a smaller EVA-CLIP, then initializes training of a larger EVA-CLIP. The authors keep the training dataset largely fixed while scaling model size.

2. Weak-to-Strong Vision Scaling

  • A large EVA model is distilled from a small EVA-CLIP, then initializes contrastive pre-training of a larger EVA-CLIP; the cycle continues by distilling a larger EVA.EVA-18B learns to reconstruct masked image-text-aligned vision features from EVA-02-CLIP-E/14+; EVA-18B then initializes EVA-CLIP-18B pre-training.Source: §2 · p. 2
  • The scaling cycle keeps the training dataset largely fixed to demonstrate the model-scale philosophy, while dataset scaling may further unleash its power.Source: §2 · p. 2
  • EVA-CLIP-8B follows similar pre-training methods, and experiments show sustained performance improvement with progressive weak-teach-strong scaling.Source: §2 · p. 2

3. Experiments

Experiments report broad results for EVA-CLIP-18B across zero-shot classification, retrieval, video, robustness, linear probing, and 3D representation learning. The authors note that more benchmarks covering more image distributions would make robustness and zero-shot evaluation more comprehensive.

Zero-Shot Image Classification

  • 80.7% top-1 accuracy averaged across 27 zero-shot image-classification benchmarks exceeds DFN5B-CLIP-H/14+ by +1.5% and InternVL-C by +2.7%.The Birdsnap evaluation used 2,195 test images because downloads were limited.Source: §3 · p. 3

Robustness

  • 83.6% top-1 accuracy averaged across six ImageNet benchmarks and ObjectNet comes with only a 0.2% gap from ImageNet-1K.The reported gap is the difference between averaged accuracy and ImageNet-1K accuracy; lower is better.Source: §3 · p. 4, Table 6 · p. 4
  • +8.2 (+6.3) average top-1 accuracy over BASIC-L on 17 datasets, despite lower performance on ImageNet variants.The parenthetical average excludes Birdsnap because the methods used different test sizes.Source: §3 · p. 4, Table 6 · p. 4

Zero-Shot Video Classification

  • Sampling 8 or 16 frames improves accuracy by +4.7% / +4.8% averaged across four video benchmarks versus one frame; EVA-CLIP-18B also outperforms other CLIP models.The video evaluation reports top-1 accuracy for UCF-101 and the mean of top-1 and top-5 accuracy for the three Kinetics benchmarks.Source: §3 · pp. 3–4, Table 5 · p. 3

Zero-Shot Image-Text Retrieval

  • 87.8% average recall across zero-shot image- and text-retrieval benchmarks on Flickr30K and COCO significantly outperforms competitors.Source: Table 3 · p. 3, §3 · p. 4

Evaluation Scope

  • The authors recommend adding benchmarks across more image distributions for more robust and comprehensive robustness and zero-shot evaluation.Source: §3 · p. 4

Linear Probing on ImageNet-1K

  • 88.9% average top-1 accuracy in ImageNet-1K linear probing surpasses InternVL-C by 0.7%.The table caption reports continuous top-1 accuracy improvement as EVA-CLIP scales.Source: §3 · p. 4, Table 7 · p. 4

3D Representation

  • Scaling EVA-CLIP improves 3D representation learning, and Uni3D-base with EVA-CLIP-18B sets new records on ModelNet and ScanObjectNN.The evaluation follows the Uni3D setting.Source: §3 · p. 5, Table 8 · p. 5

4. Ablation Studies

Ablations report gains from video data and higher image resolution. The reported resolution gains include settings with low global batch sizes.

Video Data

  • +0.7 with one middle frame and +0.8 with 8 frames are the video-data gains averaged across UCF-101 and Kinetics-400/600/700.Table 9 reports averages for image classification, video classification, and image-text retrieval.Source: §4 · p. 5, Table 9 · p. 5

Image Resolution

  • +0.9 average top-1 accuracy for EVA-CLIP-8B when resolution increases from 224^2 to 448^2.Table 10 reports zero-shot performance on ImageNet variants and ObjectNet.Source: §4 · p. 5, Table 10 · p. 5
  • +0.5 average top-1 accuracy for EVA-CLIP-8B when resolution increases from 224^2 to 336^2, including at a 24k global batch size.The reported batch sizes are 24k for EVA-CLIP-8B and 23k for EVA-CLIP-18B.Source: §4 · p. 5

5. Conclusion

EVA-CLIP-18B advances open-source CLIP scaling, with reported gains across image, video, and 3D benchmarks and no sign of performance saturation as model size increases.

5. Conclusion

  • EVA-CLIP-18B is presented as the largest and most performant open-source CLIP model, with 18 billion parameters.Source: §5 · p. 5
  • The model advances state of the art across prevalent benchmarks in image, video, and 3D domains.Source: §5 · p. 5
  • Increasing EVA-CLIP model size consistently boosts performance, with no sign of saturation.Source: §5 · p. 5

Supplementary Material

The supplementary material identifies the datasets used to evaluate EVA-CLIP models.

Supplementary Material

  • Table 11 lists the datasets used to evaluate EVA-CLIP models.Source: Table 11 · p. 9
  • The listed datasets are evaluation resources for EVA-CLIP models.Source: Table 11 · p. 9
  • The supplementary table provides a reference to the evaluation datasets.Source: Table 11 · p. 9

6. Training Settings

Detailed training settings for EVA-CLIP-8B and EVA-CLIP-18B are provided in Tables 12 and 13.

6. Training Settings

  • Detailed training settings are provided for EVA-CLIP-8B and EVA-CLIP-18B.Source: §6 · p. 9
  • The training settings cover two models: EVA-CLIP-8B and EVA-CLIP-18B.Source: §6 · p. 9
  • Tables 12 and 13 contain the detailed training settings.Source: §6 · p. 9

7. Image Transformations for Evaluation

The section examines how image-resizing choices affect zero-shot evaluation. EVA-CLIP-18B retains nearly the same average accuracy across transformations in zero-shot image and video classification, while reported results use the better-performing transformation and use direct resizing for zero-shot retrieval.

7. Image Transformations for Evaluation

  • EVA-CLIP-18B has the same average top-1 accuracy across image transformations in zero-shot image classification and nearly the same average accuracy across transformations in zero-shot image/video classification.Different transformations can significantly affect classification, particularly on ObjectNet; gaps are also noted for VOC2007 classification and Flickr30K and COCO retrieval.Source: §7 · pp. 9–10
  • The evaluation compares direct resizing to a fixed size with resizing by shortest side followed by a center crop.Table 14 investigates the impact of these two transformations on zero-shot evaluation.Source: §7 · p. 9, Table 14 · p. 10
  • Reported results select the better-performing transformation, while zero-shot retrieval specifically uses direct resizing to a fixed size.The two transformations can produce a significant performance gap in zero-shot retrieval on Flickr30K and COCO.Source: §7 · pp. 9–10
Loading 2402.04252v1…