PaperlayerView exact revision on arXiv

Source-linked AI summary

EVA-02: A Visual Representation for Neon Genesis

Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, Yue Cao

arXiv:2303.11331v2cs.CVcs.CL

TL;DR

The paper addresses the limited accessibility and transparency of state-of-the-art visual representations that depend on large models, infrastructure, and private data. EVA-02 combines optimized plain Vision Transformers with masked image modeling from a strong CLIP vision encoder, achieving strong transferable performance across vision tasks with moderate model sizes and public data.

  • Problem

    State-of-the-art visual representations often require very large models, substantial infrastructure, and private web-scale training data, limiting affordability, transparent evaluation, and broad access.

  • Method

    EVA-02 robustly optimizes plain Vision Transformers and pre-trains them with masked image modeling to reconstruct features from a strong CLIP vision encoder.

  • Results

    EVA-02 outperforms larger state-of-the-art specialized models across representative vision tasks, including 90.0 fine-tuning top-1 accuracy on ImageNet-1K with 304M parameters and publicly accessible data.

  • Takeaways & Limitations

    EVA-02 provides performant, affordable, and transferable vision representations across model sizes from 6M to 304M parameters, while EVA-02-CLIP offers language-aligned features for future multimodal systems.

  • Takeaways & Limitations

    With EVA-CLIP targets, base-sized plain ViT accuracy initially decreases from 85.0 to 84.0 under the stated 300-epoch ImageNet-1K pre-training setup.

Abstract

from arXiv · show

We launch EVA-02, a next-generation Transformer-based visual representation pre-trained to reconstruct strong and robust language-aligned vision features via masked image modeling. With an updated plain Transformer architecture as well as extensive pre-training from an open & accessible giant CLIP vision encoder, EVA-02 demonstrates superior performance compared to prior state-of-the-art approaches across various representative vision tasks, while utilizing significantly fewer parameters and compute budgets. Notably, using exclusively publicly accessible training data, EVA-02 with only 304M parameters achieves a phenomenal 90.0 fine-tuning top-1 accuracy on ImageNet-1K val set. Additionally, our EVA-02-CLIP can reach up to 80.4 zero-shot top-1 on ImageNet-1K, outperforming the previous largest & best open-sourced CLIP with only ~1/6 parameters and ~1/6 image-text training data. We offer four EVA-02 variants in various model sizes, ranging from 6M to 304M parameters, all with impressive performance. To facilitate open access and open research, we release the complete suite of EVA-02 to the community at https://github.com/baaivision/EVA/tree/master/EVA-02.

1. Introduction

EVA-02 targets the gap between high-performing but resource-intensive vision models and affordable, transparent alternatives by combining optimized plain ViTs with MIM from a strong CLIP vision encoder. Across representative vision tasks, it achieves strong performance with moderate model sizes and publicly accessible data.

  • Motivation: EVA-02 addresses the growing gap between state-of-the-art vision models and models affordable to the wider research community.Large models require substantial computation, time, infrastructure, and often private web-scale data, limiting transparent evaluation and broad access.
  • Approach: EVA-02 uses robustly optimized plain Vision Transformers trained with masked image modeling to learn transferable representations from a strong CLIP vision encoder.The approach targets masked-out CLIP vision features and is designed around moderate model sizes.
  • Results: EVA-02 outperforms prior state-of-the-art approaches across visual tasks while using fewer compute budgets and resources than leading models with billions of parameters.Reported evaluations include classification, CLIP zero-shot transfer, object detection, instance segmentation, and semantic segmentation.
  • Results: 304M parameters yield 90.0 fine-tuning top-1 accuracy on ImageNet-1K validation using 38 million publicly accessible data.The 22M-parameter small variant reaches 85.8 fine-tuning top-1 accuracy under the same publicly accessible-data setting.
  • Results: 80.4 zero-shot top-1 on ImageNet-1K is achieved by EVA-02-CLIP, surpassing CLIP-Giant with approximately 1/6 the parameters and image-text training data.The comparison concerns the previous largest and best open-sourced CLIP identified in the passage.
  • Accessibility: The EVA-02 series spans 6M to 304M parameters and aims to make performant state-of-the-art representations more accessible for research and practice.The paper frames its contribution as identifying a robust, effective recipe rather than proposing an entirely new method.

2. Approach

EVA-02 combines plain ViT architectural refinements with masked image modeling from a giant EVA-CLIP teacher, targeting robust transferable visual representations. Scaling pre-training, model size, resolution, and intermediate fine-tuning yields strong ImageNet performance, including 90.0 top-1 accuracy with 304M parameters.

  • Architectural improvements: EVA-02 uses a plain ViT-based architecture with SwiGLU FFN, sub-LN, 2D RoPE, and Xavier normal initialization.These modifications form the Transform Vision (TrV) configuration.
  • Architectural improvements: 1.6 points improved fine-tuning accuracy, from 84.0 to 85.6, after introducing the final TrV configuration under the pilot setup.The architectural modifications were designed not to add parameters or FLOPs relative to the baseline.
  • Pre-training strategy: 1.3-point improvement over BEiTv2 emerged when TrV with EVA-CLIP used 1600 epochs of pre-training, indicating that stronger teachers require more extensive training.With 150 epochs of pure MIM pre-training on IN-21K, base-sized TrV reached 87.0 top-1 accuracy and exceeded the cited BEiTv2 setup.
  • Scaling: 90.0 top-1 accuracy on IN-1K was reached by the 304M-parameter EVA-02 after scaling model size, resolution, and label-based intermediate fine-tuning.Pure MIM-pre-trained representations also achieved competitive performance without additional intermediate fine-tuning.
  • MIM pre-training: Masked image modeling regresses masked-out language-aligned EVA-CLIP vision features from visible image patches.The target representation comes from the publicly accessible one-billion-parameter EVA-CLIP vision tower, with block-wise masking at a 40% ratio.
  • Model variants and data: 38 million publicly accessible images comprise the merged pre-training dataset for EVA-02-L, while four variants span 6M to 304M parameters.The variants are EVA-02-Ti, -S, -B, and -L; EVA-02-L combines IN-21K, CC12M, CC3M, COCO, ADE20K, Object365, and OpenImages images.

3. Experiments and Evaluation

EVA-02 is evaluated across classification, robustness, zero-shot vision-language transfer, detection, instance segmentation, and semantic segmentation, where it generally outperforms same- or larger-sized counterparts. These results are obtained with moderate models and publicly accessible or limited training resources.

  • Image classification: 90.0 fine-tuning top-1 accuracy is achieved by EVA-02-L with 304M parameters on the ImageNet-1K validation set.The model outperforms several larger state-of-the-art models trained with more data.
  • Image classification: EVA-02 performs favorably in lightweight settings, with tiny and small plain ViT variants overcoming the expected weakness of limited inductive biases.The result is attributed to extensive masked image modeling pre-training.
  • Robustness: EVA-02 achieves the highest averaged accuracy and smallest validation-set performance gap across ImageNet-1K variants.Models are fine-tuned on the original training set and evaluated on variants without specialized fine-tuning.
  • Zero-shot evaluation: 80.4 top-1 accuracy is reached in ImageNet-1K zero-shot classification with the large EVA-02-CLIP model.Base and large models achieve 74.7 and 80.4 top-1 accuracy, respectively; the large model also improves average performance across 26 additional zero-shot benchmarks.
  • Zero-shot evaluation: EVA-02-CLIP outperforms same-sized competitors in zero-shot image and text retrieval and is effective on zero-shot video recognition benchmarks.Retrieval performance remains competitive, with language-encoder capacity suggested as a factor in task differences.
  • Detection and instance segmentation: EVA-02 outperforms same- and larger-sized counterparts in object detection and instance segmentation, particularly on LVIS.The comparison includes ViTDet with Cascade Mask R-CNN counterparts.
  • Semantic segmentation: Pure MIM-pre-trained EVA-02 achieves up to 60.1 mIoU in single-scale semantic segmentation and new state-of-the-art results on COCO-Stuff-164K and ADE20K.The representation outperforms same-sized BEiTv2 models and larger pre-trained counterparts in reported comparisons.

4. Related Work

Related work motivates EVA-02 through the value of careful scaling and evaluation rather than novelty alone. The paper positions thorough MIM evaluation as a way to narrow the gap between state-of-the-art and accessible visual representations.

  • Scaling and empirical evaluation: Prior representation-learning advances show that scaling and careful measurement of training choices can produce major quantitative improvements without entirely new ideas.The discussion cites GPT-style scaling and RoBERTa’s replication and measurement study as examples.
  • Accessible representations: EVA-02 applies this evaluation-oriented perspective to MIM visual representation learning to bridge performance and accessibility.The stated target is the gap between large-scale state-of-the-art models and affordable models for the wider research community.

5. Discussion and Conclusion

The paper frames EVA-02 as a robust optimization and evaluation of MIM with CLIP vision features, producing affordable transferable representations. It also identifies alternate MIM and CLIP training as a bootstrapped direction for future scaling.

  • Discussion and conclusion: EVA-02 shows that robustly optimized MIM pre-training with CLIP vision features can produce affordable, transferable representations outperforming larger specialized models.The contribution is explicitly presented as an in-depth evaluation rather than an entirely new architecture or method.
  • Discussion and conclusion: Base- and large-sized EVA-02 models can yield compact, expressive CLIP representations for modular, reusable, and scalable model design.The findings are also presented as a reference for future work on model and representation scaling.
  • Discussion and conclusion: Alternate training of pure MIM and vision-language CLIP representations improves both MIM and CLIP performance in a bootstrapped manner.The paper identifies this as a promising scalable pre-training approach for representations of various sizes.

A.1. Architecture

EVA-02’s architecture incorporates Transformer design choices from language models into plain ViTs, including SwiGLU, sub-LN, and 2D RoPE. The resulting configuration improves accuracy while preserving practical compute and parameter targets.

  • Feedforward network: SwiGLU replaces the original ViT FFN’s first transformation with a SiLU-gated linear unit.Its output combines SiLU(xU) and xV element-wise before multiplication by W.
  • Feedforward network: The SwiGLU hidden dimension is reduced by 2/3 to keep parameters and computation constant relative to the original MLP FFN.The reduction applies to the output dimensions of U and V and the input dimension of W.
  • Normalization: EVA-02 uses sub-LN for base and large blocks, while tiny and small models use the default pre-LN configuration.The inner attention layer normalization is omitted in the sub-LN configuration.
  • Position embedding: RoPE rotates input embeddings without changing their norm, making attention depend on relative positions through q and k vectors.This design is compatible with fused high-performance multi-head self-attention operators.

A.2. Additional Results for Image Classification

Additional experiments show that sufficiently pre-trained EVA-02 representations remain effective across training choices and can outperform prior approaches without intermediate fine-tuning.

  • EVA-02 representations without IN-21K intermediate fine-tuning outperform some previous leading approaches.
  • Precisions and optimizers: Fine-tuning remains robust across fp16 and bf16 numerical precisions and Lion, AdamW, and SGD optimizers.
  • Precisions and optimizers: Using SGD for fine-tuning causes only a little performance drop.
  • The student is the master: MIM is distinguished from conventional knowledge distillation in the pre-training and fine-tuning paradigm.

A Case Study

The case study examines whether unlabeled development and test images used during MIM pre-training contaminate downstream detection and segmentation evaluations. Across the reported settings, the impact on final performance is minor.

  • The study finds that data contamination during MIM pre-training has a minor impact when representations transfer to object detection and instance segmentation.
  • Experimental setup: Two EVA-02-L models use either Merged-38M unlabeled images or ImageNet-21K images for MIM pre-training, with both trained for 1M steps and batch size 2k.
  • Transfer settings: The transfer study evaluates direct MIM-to-O365, direct MIM-to-LVIS, and MIM-to-O365-to-LVIS settings.
  • Evaluation benchmarks: O365 evaluation uses its test set, while LVIS evaluation uses its validation set, a long-tail benchmark with approximately 20k images and 0.25M annotated instances across more than 1.2k categories.
  • Motivation: The contamination arose because Merged-38M included unlabeled development or test images, prompting concerns about downstream benchmark validity.

A.4. Implementation Details

The implementation details specify the evaluation settings, model configurations, and training procedures used across classification, detection, instance segmentation, and semantic segmentation experiments.

  • The appendix summarizes training, evaluation settings, configurations, and hyper-parameters for EVA-02 experiments.
  • Training settings: The reported settings cover MIM pre-training, ImageNet-21K intermediate fine-tuning, ImageNet-1K fine-tuning, enhanced CLIP training, and O365 intermediate fine-tuning.
  • Model configuration: The model configurations include EVA-02-Ti, -S, -B, and -L, with peak learning rates of 2e-4, 1e-4, 1e-4, and 7e-5, respectively.
  • Optimization: The default optimization setup uses AdamW, cosine decay, β1 = 0.9, β2 = 0.999, ϵ = 1e-8, and weight decay 0.05.
Loading 2303.11331v2…