PaperlayerView exact revision on arXiv

Source-linked AI summary

EVA: Exploring the Limits of Masked Visual Representation Learning at Scale

Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, Yue Cao

arXiv:2211.07636v2cs.CVcs.CLcs.LG

TL;DR

Large-scale vision pretraining still relies heavily on supervised or weakly supervised labeled data, while the scalability of pure masked modeling remains uncertain. EVA addresses this gap by reconstructing masked CLIP vision features with a vanilla ViT, scaling to one billion parameters using public unlabeled images. It achieves strong results across vision tasks and also improves the stability and efficiency of giant CLIP training.

  • Problem

    Competitive billion-scale vision models still heavily rely on supervised or weakly supervised training with large labeled datasets, leaving the scalability of pure masked image modeling uncertain.

  • Method

    EVA is a one-billion-parameter vanilla ViT pretrained to reconstruct masked image-text aligned CLIP vision features conditioned on visible image patches.

  • Results

    EVA achieves strong performance across image, video, detection, and segmentation benchmarks using 29.6 million public unlabeled images, while EVA initialization improves giant CLIP training.

  • Takeaways & Limitations

    Masked visual feature modeling can scale to billion-parameter vision representation learning and support vision-centric multimodal training without heavy supervised pretraining.

  • Takeaways & Limitations

    The authors caution that a zero APmask gap does not mean EVA solves LVIS, and rare-category performance still trails common and frequent categories.

Abstract

from arXiv · show

We launch EVA, a vision-centric foundation model to explore the limits of visual representation at scale using only publicly accessible data. EVA is a vanilla ViT pre-trained to reconstruct the masked out image-text aligned vision features conditioned on visible image patches. Via this pretext task, we can efficiently scale up EVA to one billion parameters, and sets new records on a broad range of representative vision downstream tasks, such as image recognition, video action recognition, object detection, instance segmentation and semantic segmentation without heavy supervised training. Moreover, we observe quantitative changes in scaling EVA result in qualitative changes in transfer learning performance that are not present in other models. For instance, EVA takes a great leap in the challenging large vocabulary instance segmentation task: our model achieves almost the same state-of-the-art performance on LVISv1.0 dataset with over a thousand categories and COCO dataset with only eighty categories. Beyond a pure vision encoder, EVA can also serve as a vision-centric, multi-modal pivot to connect images and text. We find initializing the vision tower of a giant CLIP from EVA can greatly stabilize the training and outperform the training from scratch counterpart with much fewer samples and less compute, providing a new direction for scaling up and accelerating the costly training of multi-modal foundation models. To facilitate future research, we release all the code and models at https://github.com/baaivision/EVA.

1. Introduction

EVA explores whether masked visual representation learning can scale to billion-parameter vision models using publicly accessible unlabeled data. It achieves strong transfer across diverse vision tasks and extends to multimodal CLIP training.

  • Motivation and approach: EVA scales a vanilla ViT to one billion parameters through masked image modeling with strong transfer across downstream vision tasks.The approach uses masked image-text aligned vision features as the reconstruction target.
  • Vision benchmarks: Using 29.6 million publicly accessible unlabeled images, EVA reaches 89.7% top-1 accuracy on ImageNet-1K.
  • Vision benchmarks: EVA reports strong results across object detection, instance segmentation, semantic segmentation, and video action recognition benchmarks.Reported results include 62.2 APbox and 55.0 APmask on LVISv1.0, 53.4 mIoUss on COCO-Stuff, and 89.7% top-1 accuracy on Kinetics-400.
  • Scaling effects: EVA reaches nearly the same APmask on LVISv1.0 with more than 1,200 categories as on COCO with 80 categories.Both are reported at 55.0 APmask on the validation set.
  • Scaling effects: The authors report that scaling EVA produces qualitative transfer changes not observed in smaller-scale models, including a breakthrough on large-vocabulary instance segmentation.
  • Multimodal extension: Initializing a 1.1-billion-parameter CLIP image encoder from EVA outperforms training from scratch with fewer samples and less compute while stabilizing training.

2. Fly EVA to the Moon

The authors compare masked modeling objectives for visual transfer and select direct reconstruction of masked CLIP vision features. They then scale this objective to EVA without semantic tokenization or a supervised pretraining stage.

  • Pilot experiments: The pilot study evaluates tokenized semantic feature recovery and feature distillation as candidate masked image modeling objectives.
  • Pilot experiments: The CLIP feature tokenization process is unnecessary for achieving good downstream performance.
  • Pilot experiments: Feature distillation fails to provide consistent performance gains as pretraining becomes longer.
  • Selected objective: EVA therefore reconstructs masked CLIP vision features conditioned on visible image patches, an objective chosen for scaling.
  • Scaling objective: The objective scales to billion-parameter models and tens of millions of unlabeled images without semantic feature quantization or explicit image-text paired pretraining data.

2.2. Pre-training

EVA is a one-billion-parameter vanilla ViT trained to reconstruct masked CLIP vision features from visible patches. Its pretraining uses publicly accessible image data and standard optimization and infrastructure choices.

  • Architecture: EVA uses a vanilla ViT architecture with 1.0 billion parameters.
  • Pretraining objective: The pretraining target is the masked-out image-text aligned vision feature from the publicly available OpenAI CLIP-L/14 vision tower.
  • Pretraining objective: EVA masks 40% of input patches block-wise and reconstructs their normalized projected features using negative cosine similarity loss.
  • Pretraining data: The pretraining dataset contains 29.6 million publicly accessible images drawn from CC12M, CC3M, COCO, ADE20K, ImageNet-21K, and Object365.
  • Pretraining data: The CLIP prediction targets were trained on a 400-million-image image-text dataset, so EVA implicitly exploits knowledge from that dataset.
  • Optimization: EVA is optimized with Adam, decoupled weight decay, cosine learning-rate decay, stochastic depth, and RandResizeCrop augmentation.
  • Infrastructure: The implementation uses PyTorch, DeepSpeed ZeRO stage 1, NVIDIA A100-SXM4-40GB GPUs, and stable fp16 training.

2.3. Evaluation on Downstream Tasks

EVA achieves strong transfer performance across image, video, detection, segmentation, and zero-shot multimodal evaluations using publicly accessible data. Its scaling produces especially notable gains in large-vocabulary instance segmentation and supports efficient billion-scale CLIP training.

  • Image Classification: EVA achieves 89.7% top-1 accuracy on ImageNet-1K using a linear classifier and publicly available data.The model also establishes state-of-the-art self-supervised results on ImageNet-1K.
  • Image Classification: EVA achieves the smallest performance gap across six ImageNet-1K validation variants while maintaining the highest averaged accuracy.The comparison uses models with differing data, resolutions, sizes, and architectures, so absolute accuracies are not directly comparable.
  • Video Action Recognition: EVA achieves competitive video recognition performance, including 88.4% top-1 accuracy on K-400 without K-722 intermediate fine-tuning.It also outperforms recent video-specific or large foundation models in the reported comparisons.
  • Object Detection & Instance Segmentation: EVA establishes state-of-the-art object detection and instance segmentation results on COCO and LVIS using the canonical R-CNN framework.Compared with larger or specialized alternatives, EVA uses streamlined pre-training and avoids image-text paired data and semantic feature tokenization during pre-training.
  • Object Detection & Instance Segmentation: EVA achieves the same APmask on LVIS and COCO with single-scale evaluation, largely closing the usual performance gap between the two benchmarks.The authors caution that a zero APmask gap does not mean EVA solves LVIS, because LVIS annotations can have higher quality and may yield higher APmask than COCO.
  • Semantic Segmentation: EVA achieves strong semantic segmentation results on ADE20K and COCO-Stuff-164K, while performing slightly below BEiT-3 on ADE20K.The authors suspect the lower ADE20K performance is partly due to weakened architectural configurations.
  • Contrastive Language-Image Pre-training with Zero-shot Classification Evaluation: EVA-CLIP reaches 72.7% averaged zero-shot top-1 accuracy across 12 benchmarks and the smallest distribution-shift gap at 2.5%.It performs best on 10 of 12 zero-shot classification benchmarks and achieves 78.2% zero-shot top-1 accuracy on ImageNet-1K without training labels.
  • Contrastive Language-Image Pre-training with Zero-shot Classification Evaluation: Initializing a 1.1B EVA CLIP-g from CLIP-L enables large performance improvements and provides an efficient, scalable CLIP training approach.The authors report that this approach addresses training instability and efficiency challenges in billion-scale CLIP training.

3. Related Work

Masked image modeling has progressed from pixel and token prediction toward large-scale masked image-text aligned feature prediction. EVA explores this approach with vanilla ViTs, which become competitive at sufficient model and data scales.

  • Masked image modeling learns visual representations by predicting masked contents from visible context.
  • The BEiT family improves MIM through masked visual token prediction, while other work explores pixel and feature regression at smaller scales.
  • EVA extends large-scale MIM pre-training through masked image-text aligned feature prediction.
  • At sufficient model and data scales, ViTs outperform ConvNets because ConvNets lack scalable pre-training tasks and retain built-in inductive biases.

4. Conclusion

EVA is a one-billion-parameter vanilla ViT showing that simple masked feature modeling can scale with minimal vision priors. It achieves strong results across diverse downstream vision tasks and aims to connect vision and language through masked modeling.

  • EVA is a one-billion-parameter vanilla ViT designed to explore the limits of masked visual representation learning.
  • Simple masked feature modeling scales on an architecture with minimal vision priors.
  • EVA attains excellent results across a representative and diverse set of downstream vision tasks.
  • The work aims to bridge vision and language research through masked modeling.

A. Appendix

The appendix summarizes configurations for EVA’s pre-training and downstream evaluations.

  • The appendix summarizes configurations for image classification, video action classification, object detection and instance segmentation, and semantic segmentation.

A.1. Image Classification

Image-classification fine-tuning hyperparameters are reported separately for ImageNet-21K and ImageNet-1K.

  • Fine-tuning hyperparameters for ImageNet-21K and ImageNet-1K are provided in separate tables.

A.2. Video Action Classification

Video action classification uses two-stage fine-tuning: intermediate training on a cleaned, merged Kinetics dataset, followed by dataset-specific fine-tuning with more frames and multi-view inference.

  • A.2. Video Action Classification: EVA is first intermediate-fine-tuned on cleaned Kinetics-722, which combines K-400, K-600, and K-700 training samples.K-722 contains 0.63M training videos spanning 722 human action classes after removing leaked and duplicated videos.
  • A.2. Video Action Classification: The intermediate video stage uses 2242 resolution with 8 input frames.
  • A.2. Video Action Classification: Each target dataset receives a second fine-tuning stage using 16 frames at 2242 resolution.Sparse frame sampling is used, followed by four temporal clips and three spatial crops during testing; final predictions ensemble all trials.

A.3. Object Detection & Instance Segmentation

Object detection and instance segmentation use intermediate fine-tuning on Objects365, followed by largely shared COCO and LVIS configurations with longer training for LVIS.

  • A.3. Object Detection & Instance Segmentation: Objects365 intermediate fine-tuning runs for 380k iterations with batch size 128.Training begins at 10242 resolution for 320k iterations, then increases to 12802 for adaptation to COCO and LVIS.
  • A.3. Object Detection & Instance Segmentation: COCO and LVIS fine-tuning uses an initialized learning rate of 2.5e-5, reduced by 10 for the final 5k iterations.
  • A.3. Object Detection & Instance Segmentation: COCO and LVIS use almost identical hyper-parameters, except for long-tailed recognition techniques and training duration.Repeat-factor sampling and federated loss are used for long-tailed recognition; COCO trains for 45k steps and LVIS for 75k steps.

A.4. Semantic Segmentation

Semantic segmentation follows ViT-Adapter settings with Mask2Former as the segmentation head and uses COCO-Stuff pre-trained weights for ADE20K initialization.

  • A.4. Semantic Segmentation: Semantic segmentation uses ViT-Adapter-style settings with Mask2Former as the segmentation head.
  • A.4. Semantic Segmentation: ADE20K training is initialized with COCO-Stuff pre-trained weights.
Loading 2211.07636v2…