PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Language-Image Pre-training via Masking

Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, Kaiming He

arXiv:2212.00794v2cs.CV

TL;DR

Large-scale CLIP training is computationally expensive, motivating more efficient ways to process image-text pairs. FLIP randomly removes image patches during training, achieving similar accuracy in over 3× less wall-clock time and higher accuracy at the same epoch count while remaining 2–3× faster.

  • Problem

    Large-scale language-supervised vision training requires substantial computation, creating a need for more efficient CLIP training.

  • Method

    FLIP randomly masks and removes a large portion of image patches during CLIP training to reduce computation and process more sample pairs.

  • Results

    Removing 50%-75% of patches reduces computation by 2-4×; FLIP reaches similar accuracy in >3× less wall-clock time and higher accuracy at the same epoch count while remaining 2-3× faster.

  • Takeaways & Limitations

    FLIP offers a competitive, more practical alternative to CLIP for large-scale training by substantially reducing wall-clock time, energy use, and commercial cost.

  • Takeaways & Limitations

    Larger batches remain constrained by available memory, with some configurations requiring more devices or memory optimization.

Abstract

from arXiv · show

We present Fast Language-Image Pre-training (FLIP), a simple and more efficient method for training CLIP. Our method randomly masks out and removes a large portion of image patches during training. Masking allows us to learn from more image-text pairs given the same wall-clock time and contrast more samples per iteration with similar memory footprint. It leads to a favorable trade-off between accuracy and training time. In our experiments on 400 million image-text pairs, FLIP improves both accuracy and speed over the no-masking baseline. On a large diversity of downstream tasks, FLIP dominantly outperforms the CLIP counterparts trained on the same data. Facilitated by the speedup, we explore the scaling behavior of increasing the model size, data size, or training length, and report encouraging results and comparisons. We hope that our work will foster future research on scaling vision-language learning.

Meta AI, FAIR · 1. Introduction

FLIP makes CLIP training more efficient by randomly removing image patches, enabling larger batches and more image-text pairs within the same wall-clock budget. It improves the speed–accuracy trade-off, transfers strongly across downstream tasks, and supports controlled scaling studies.

  • 1. Introduction: Language-supervised pre-training produces versatile representations with strong zero-shot transfer, high-quality text-to-image generation, and benefits for multimodal and unimodal visual tasks.CLIP established this approach as a simple yet powerful representation-learning methodology; text-to-image generation examples include [53].
  • 1. Introduction: Language supervision captures objects, scenes, actions, context, and relations at multiple granularities, making large-scale training essential for vision-language capabilities.The original CLIP models used 400 million data for 32 epochs, equivalent to 10,000 ImageNet epochs.
  • 1. Introduction: With 50%–75% masking, FLIP trains over 3× faster to similar accuracy and reaches higher accuracy than CLIP at the same epochs while remaining 2–3× faster.Figure 1 evaluates ViT-L/16 on LAION-400M using ImageNet-1K zero-shot accuracy across 256 TPU-v3 cores.
  • 1. Introduction: Wall-clock training time remains a major bottleneck for exploring how vision-language learning scales, even with high-end infrastructure.This bottleneck motivates more efficient pre-training methods.
  • 1. Introduction: Randomly removing a large portion of image patches creates a trade-off between examining each pair carefully and processing more image-text pairs.The method is inspired by the sparse computation of Masked Autoencoders (MAE) [29].
  • 1. Introduction: Removing 50%–75% of patches reduces computation 2–4× and permits 2–4× larger batches with little extra memory, benefiting contrastive learning.The larger batches improve accuracy through contrastive-learning behavior [30] [11].
  • 1. Introduction: FLIP dominantly outperforms OpenCLIP and the authors’ CLIP reproduction across diverse downstream datasets and transfer scenarios when pretrained on the same LAION-400M data.These comparisons indicate that faster training can coincide with accuracy gains.
  • 1. Introduction: FLIP scaling experiments vary model size, dataset size, and training-schedule length, finding accuracy gains from model and data scaling, including data gains without extra training cost.The analyses use carefully controlled experiments.

2. Related Work

Prior work established masking for representation learning and efficient sparse encoding, including masked language modeling, vision, and multimodal extensions. FLIP applies high-ratio image-patch masking to large-scale CLIP training without reconstruction, emphasizing scaling rather than autoencoding.

  • Learning with masking: Masking has supported unsupervised representation learning through denoising autoencoders, BERT-style masked language modeling [18], and vision methods predicting missing regions, pixels, patches, or features [20] [29] [71] [6] [66].
  • Learning with masking: MAE [29] reduces training time and memory by applying the ViT encoder [20] only to visible content, with high masking ratios benefiting accuracy; its design extends across multiple modalities.Extensions include videos, point clouds, graphs, audio, visual control, and vision-language applications [61] [22] [49] [59] [9] [32] [4] [47] [13] [35] [70] [57] [23] [41] [31] [19].
  • Learning with masking: FLIP targets scaling large-scale CLIP training through sparse computation, unlike prior vision-language masking works [23] [41] [31] [19], which were limited in scale and did not define this focus.FLIP does not reconstruct masked content and is not an autoencoder; related masking for self-supervised contrastive learning [69] was limited by image-only scaling behavior.
  • Language-supervised learning: CLIP and related methods [37] [51] popularized language-supervised visual representation learning by contrasting image-text pairs, alongside generative approaches [17] [65] [2] [74].FLIP focuses on CLIP and may extend to generative methods in future work [74].

3. Method

FLIP masks and removes image patches during CLIP training to reduce computation, enabling more image-text pairs or larger contrastive batches under fixed resource budgets. The method uses masked ViT encoding without reconstruction and can optionally unmask for improved accuracy–time trade-offs.

  • Method overview: Under fixed wall-clock time or memory, masking trades encoding density for more image-text pairs or a larger-batch contrastive learning signal.The method is designed to spend computation more efficiently across samples and batch negatives.
  • Image masking: FLIP randomly masks image patches before ViT encoding, reducing image-encoding cost while enabling larger batches at similar memory use.Masking 50% or 75% of patches reduces time complexity to 1/2 or 1/4 and permits 2× or 4× larger batches.
  • Text masking: Text masking is optional and used only for ablation because the smaller text encoder makes its speedup unhelpful for the overall accuracy–time trade-off.Unlike BERT [18], masked text tokens are not replaced with a learned mask token.
  • Objective: FLIP trains image and text encoders with a batch-based contrastive loss, omitting MAE-style reconstruction because it is unnecessary for good zero-shot transfer.Removing the decoder and reconstruction loss provides additional speedup.
  • Unmasking: The masked-image encoder can process intact images directly, while brief 0%-masking continuation training reduces the masking-induced distribution gap and improves the accuracy–time trade-off.Direct application to intact images requires no architectural changes; this setting is used as an ablation baseline.

4. Experiments

Across classification, robustness, retrieval, and captioning evaluations, FLIP is a competitive or superior alternative to CLIP, especially when pre-training data is controlled. Masking also yields a strong accuracy–training-time trade-off, while text masking and inference-time masking are not worthwhile in the evaluated settings.

  • Masking and batch size: Masking 50% preserves comparable accuracy at batch size 16k, reaching 68.5% versus 68.6% for the unmasked baseline, whereas increasing batch size consistently improves accuracy.Scaling batch size with the masking ratio roughly maintains memory footprint, but large-batch settings can reach the memory limit of the infrastructure.
  • Text masking: Text masking reduces accuracy by 2.2% under random masking, but prioritizing padding tokens limits degradation to 0.4%; because speed gains are marginal, text masking is omitted thereafter.The text encoder accounts for only 4.4% of unmasked image-encoder computation, explaining the weak efficiency trade-off.
  • Inference and reconstruction: Inference-time masking causes a large accuracy drop, such as 7.3%, and ensembling complementary masked views narrows but does not eliminate the gap versus full-view inference.Reconstruction loss also slightly harms zero-shot and ImageNet fine-tuning accuracy, so the final system omits it for simplicity and a better accuracy–time trade-off.
  • Masking efficiency: With the same 32-epoch schedule, 50% masking is ∼1% more accurate than CLIP and 2× faster, while similar accuracy can be achieved with speedups exceeding 3×.Masking 50% and 75% require only 0.50× and 0.33× of baseline wall-clock training time, respectively.
  • Classification: 74.6% accuracy for ViT-L/14 is 1.8% above OpenCLIP and 1.5% above the CLIP reproduction, reducing the gap to original CLIP to 0.7%.FLIP also reaches 83.6% in ImageNet linear probing, 1.0% above its CLIP counterpart and 0.6% above transferred original-CLIP features.
  • Downstream comparisons: FLIP outperforms same-data CLIP counterparts across downstream evaluations and is dominantly better than OpenCLIP and the CLIP reproduction after isolating pre-training-data effects.Pre-training data systematically affects task performance: WIT helps Aircraft, Country211, and SST2, whereas LAION helps Birdsnap, SUN397, and Cars.

5. Discussion and Conclusion

FLIP preserves CLIP’s simple, scalable design while providing a 2-3× speedup or more and outperforming CLIP counterparts trained on the same LAION data. The study identifies data scaling as a favorable dimension, while noting remaining training costs and dataset biases.

  • Discussion and Conclusion: FLIP retains CLIP’s simple design while extending its scalability, building on language’s richer supervision than classical closed-set labels.The discussion frames scaling as increasing model capacity or information through data.
  • Discussion and Conclusion: 2-3× speedup or more reduces wall-clock time substantially while saving energy and commercial cost.At the study’s scale, the reduction can reach the order of thousands of TPU/GPU-days.
  • Discussion and Conclusion: FLIP outperforms its CLIP counterparts pre-trained on the same LAION data, while training-data differences create systematic gaps across several tasks.The study uses controlled comparisons with CLIP baselines and contrasts LAION-based models with original WIT-based models.
  • Discussion and Conclusion: Data scaling improves accuracy without extra training or inference cost, and FLIP enables scaling beyond the study’s explored range.The paper presents controlled experiments on scaling behavior and favors data scaling as a dimension.
  • Broader impacts: Training costs remain sizable despite FLIP reducing energy and carbon costs to 1/2-1/3, and model weights may inherit biases from the public dataset.Comparisons using identical data better isolate method properties, whereas comparisons across different training data must account for dataset biases.

A. Implementation Details · A.1. Pre-training

Pre-training uses CLIP-following ViT image encoders with smaller text encoders, trained on TPU-v3 systems under a default configuration with scalable learning rates. Unmasked tuning modifies the base learning rate and warmup schedule, while bfloat16 provides only a modest speedup.

  • A.1. Pre-training: The image encoder follows CLIP and uses ViT-B, ViT-L, or ViT-H architectures with global average pooling.Patch sizes are 16 for ViT-B/L and 14 for ViT-H, following [20].
  • A.1. Pre-training: The corresponding text encoder is smaller, and the model includes image and text encoders plus output projections.Table 9 reports the component and total model sizes.
  • A.1. Pre-training: ViT-B and ViT-L training uses 256 TPU-v3 cores, whereas ViT-H training uses 512 cores.
  • A.1. Pre-training: The default configuration applies linear learning-rate scaling: lr = base lr×batchsize / 256 [24].This rule permits batch-size changes in ablations without additional learning-rate search.
  • A.1. Pre-training: Training uses float32 by default, while bfloat16 yields only an approximately 1.1× speedup.The observed speedup is consistent with results reported in Google’s blog 4.
  • A.1. Pre-training: Unmasked tuning follows the default pre-training configuration but lowers the base learning rate to 4e-8 and shortens warmup to 25.6M samples.Unmasked tuning disables masking during pre-training.

A.2. ImageNet Classification

ImageNet evaluation includes zero-shot classification with recommended prompt templates and supervised linear-probing and fine-tuning settings following prior work [29]. The zero-shot protocol uses seven templates from an 80-template set [52], while all 80 produce similar results with slower inference.

  • Zero-shot: Zero-shot evaluation uses 7 recommended prompt templates from the 80 provided by.Using all 80 templates gives similar results but is slower at inference.
  • Linear probing and fine-tuning: Linear probing and fine-tuning follow the setting of [29], with results reported in Tables 11 and 12.

A.3. Zero-shot Retrieval

Zero-shot retrieval is evaluated on Flickr30K and COCO using cosine similarity between image and text embeddings, without prompts.

  • A.3. Zero-shot Retrieval: Evaluation uses the standard Flickr30K and COCO [42] zero-shot retrieval benchmarks.Their test sets contain 1K and 5K image-text pairs, respectively.
  • A.3. Zero-shot Retrieval: The protocol extracts image and text embeddings from the corresponding encoders.This follows the protocol established in CLIP.
  • A.3. Zero-shot Retrieval: Retrieval ranks candidate image-text pairs by cosine similarity without using prompts.

A.4. Zero-shot Robustness Evaluation … A.7. Visual Question Answering

The merged sections specify evaluation protocols for zero-shot robustness and additional datasets, then describe fine-tuning setups for image captioning and visual question answering. They also define dataset-specific metrics, generalization tests, model architectures, optimization schedules, and input resolutions.

  • A.4. Zero-shot Robustness Evaluation: Zero-shot robustness evaluation follows OpenCLIP data preparation and uses seven prompts, except IN-R uses all 80 stronger prompts; ObjectNet uses class names without prompts and YTBB uses VOC prompts.These prompt choices follow, while the dataset preparation and split follow OpenCLIP.
  • A.5. More Zero-shot Datasets: Additional zero-shot datasets use prompt and data-preparation procedures from prior work, with metrics varying by dataset, including mean per-class accuracy, mean top-1/top-5 accuracy, ROC AUC, 11-point mAP, and top-1 accuracy.The procedures follow,, and [46].
  • A.5. More Zero-shot Datasets: Birdsnap results are limited by dataset shrinkage: 1,850 test images were available, compared with 2,149 in and 2,443 originally.This limitation affects the available evaluation set rather than the model method.
  • A.6. Captioning: Captioning uses a sequence-to-sequence transformer atop the ViT encoder, projecting image features to 384 dimensions, adding a 3-layer encoder, and replacing FLIP’s text encoder with a randomly initialized 3-layer decoder.The architecture follows and uses 384-width, 6-head transformer components.
  • A.6. Captioning: Captioning is trained end-to-end on COCO for 20 epochs with teacher-forced word-level cross-entropy, AdamW, separate learning rates for new and pretrained parameters, warmup, and cosine decay; CIDEr optimization from is omitted.Inference uses autoregressive decoding, with evaluation on COCO test metrics and direct transfer to nocaps [1] without further fine-tuning.
  • A.7. Visual Question Answering: VQA casts answer prediction as classification, combining ViT image features with RoBERTa question representations through a 4-layer multimodal fusion Transformer and a two-layer MLP, following [21].The fusion Transformer has 768-dimensional representations, 12 heads, and merged attention [21].
  • A.7. Visual Question Answering: VQA is fine-tuned end-to-end with soft-score binary sigmoid loss, batch size 256, differential learning rates, weight decay, warmup, cosine decay, and image sizes of 512×512 for ViT-L/16 or 448×448 for ViT-H/14.Pretrained ViT learning rates are 1e-5 for ViT-L and 5e-6 for ViT-H.
  • A.7. Visual Question Answering: VQA models are trained for 20 epochs on VQAv2 train+val augmented with Visual Genome question-answer pairs [39], and evaluated on the test-dev split through the evaluation server.The training procedure follows.
Loading 2212.00794v2…