PaperlayerView exact revision on arXiv

Source-linked AI summary

MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training

Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, Oncel Tuzel

arXiv:2311.17049v2cs.CVcs.CLcs.LG

TL;DR

Large CLIP encoders offer zero-shot performance but their size and latency make mobile deployment challenging, while efficient-model development faces compute and accuracy hurdles. MobileCLIP pairs mobile-oriented image-text encoders with a reinforced dataset that stores captioning-model and CLIP-ensemble knowledge. The paper reports state-of-the-art latency–accuracy tradeoffs and 10×–1000× learning efficiency versus DataComp.

  • Problem

    Large CLIP models have high memory and latency overhead that challenges mobile deployment, and efficient-model development is hindered by costly training and poor small-scale accuracy signals.

    Source: §1 · p. 1

  • Method

    MobileCLIP combines mobile-friendly image-text encoders with multi-modal dataset reinforcement using synthetic captions and embeddings from an ensemble of pretrained CLIP models.

    Source: §1 · p. 2

  • Results

    MobileCLIP achieves state-of-the-art latency–accuracy tradeoffs on zero-shot tasks, while DataCompDR reports 10×–1000× learning efficiency compared with DataComp.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The paper reports that reinforced data supports efficient CLIP learning and mobile-oriented models with strong zero-shot latency–accuracy tradeoffs.

    Source: §6 · p. 8

Abstract

from arXiv · show

Contrastive pretraining of image-text foundation models, such as CLIP, demonstrated excellent zero-shot performance and improved robustness on a wide range of downstream tasks. However, these models utilize large transformer-based encoders with significant memory and latency overhead which pose challenges for deployment on mobile devices. In this work, we introduce MobileCLIP -- a new family of efficient image-text models optimized for runtime performance along with a novel and efficient training approach, namely multi-modal reinforced training. The proposed training approach leverages knowledge transfer from an image captioning model and an ensemble of strong CLIP encoders to improve the accuracy of efficient models. Our approach avoids train-time compute overhead by storing the additional knowledge in a reinforced dataset. MobileCLIP sets a new state-of-the-art latency-accuracy tradeoff for zero-shot classification and retrieval tasks on several datasets. Our MobileCLIP-S2 variant is 2.3$\times$ faster while more accurate compared to previous best CLIP model based on ViT-B/16. We further demonstrate the effectiveness of our multi-modal reinforced training by training a CLIP model based on ViT-B/16 image backbone and achieving +2.9% average performance improvement on 38 evaluation benchmarks compared to the previous best. Moreover, we show that the proposed approach achieves 10$\times$-1000$\times$ improved learning efficiency when compared with non-reinforced CLIP training. Code and models are available at https://github.com/apple/ml-mobileclip .

1. Introduction

MobileCLIP targets mobile deployment by addressing the latency and size of existing CLIP models and the compute and accuracy challenges of efficient-model development. It combines mobile-oriented architectures with multi-modal reinforced training, reporting improved latency–accuracy tradeoffs and learning efficiency.

1. Introduction

  • Mobile deployment is challenging because large CLIP models have high size and latency, while efficient-model development faces costly training and weak small-scale accuracy signals.The authors also identify reduced model capacity as a source of subpar accuracy that can be improved through training methods.Source: §1 · pp. 1–2
  • MobileCLIP achieves a state-of-the-art latency–accuracy tradeoff on zero-shot tasks, including a new best ViT-B/16-based CLIP model.Figure 1 compares public CLIP models with DataCompDR-trained MobileCLIP; latency is measured on iPhone 12 Pro Max.Source: Figure 1 · p. 1, §1 · p. 2
  • With one 8×A100 node, DataCompDR-12M yields 61.7% zero-shot ImageNet-val accuracy in approximately one day for ViT-B/16 CLIP training from scratch.DataCompDR-1B is reported to set state-of-the-art performance on several metrics using a fraction of the training compute budget of previous works.Source: §1 · p. 2
  • MobileCLIP-S0 is approximately 5× faster and 3× smaller than OpenAI ViT-B/16 CLIP, with the same average accuracy.MobileCLIP variants use hybrid CNN-transformer architectures and structural reparametrization in image and text encoders.Source: §1 · p. 2
  • 10×–1000× learning efficiency is reported for DataCompDR compared with DataComp.DataCompDR-12M supports rapid design iteration, while DataCompDR-1B targets best large-scale training performance.Source: §1 · p. 2

2. Related Work

Related work addresses efficient CLIP learning through enhanced objectives, dataset improvements, distillation, and more efficient architectures. The paper positions its multi-modal reinforced training and hybrid architecture as extensions within these areas.

2. Related Work

  • Enhanced objectives—including masking, unimodal self-supervision, fine-grained alignment, image-text-label contrastive learning, pairwise Sigmoid loss, and multi-resolution training—are described as complementary to this method.The passage attributes cost-effective multi-resolution training to CLIPA [34].Source: §2 · p. 2
  • Synthetic captions from pretrained captioning models, combined with real captions, have been reported to improve CLIP model quality; the paper says synthetic captions are crucial for its learning-efficiency gains.This work frames caption reinforcement as complementary to dataset collection and filtering.Source: §2 · p. 2
  • Prior compact-model approaches include DIME-FM [56] for zero-shot classification and TinyCLIP using cross-modal affinity mimicking and weight inheritance.The paper also uses cross-modal affinity mimicking and stores targets from an ensemble of CLIP models.Source: §2 · p. 2
  • Offline knowledge distillation [14] is used to mitigate teacher-model training overhead; this work extends dataset reinforcement [14] to CLIP and reports accuracy gains without added training-time computation.Source: §2 · p. 2
  • Efficient image encoders span convolutional, transformer, and hybrid designs, while text encoders include transformer and hybrid designs; the paper introduces an improved hybrid for both modalities.The passage notes that token-reduction methods can further improve its models' efficiency.Source: §2 · pp. 2–3

3. Multi-Modal Reinforced Training

Multi-modal reinforced training stores captioning-model and CLIP-ensemble knowledge in a dataset, so efficient models can reuse it without teacher-related training-time computation. The approach combines synthetic captions, image augmentations, and image-text alignment distillation.

3. Multi-Modal Reinforced Training

  • The method uses an image-captioning model for synthetic captions and a strong CLIP ensemble for image-text alignment knowledge, avoiding additional teacher computation during training.The reinforced dataset stores this additional knowledge for repeated experimentation.Source: §3 · p. 3, §3.1 · p. 3

3.1. Dataset Reinforcement

  • Real captions are generally more specific but noisier than synthetic captions, and combining both is crucial for best zero-shot retrieval and classification performance.Synthetic captions are generated with CoCa; the reinforced dataset stores multiple captions and image augmentations for experimentation.Source: §3.1 · p. 3
  • An ensemble of pretrained CLIP models supplies teacher embeddings for augmented images and synthetic and ground-truth captions, which are stored with the reinforced dataset.Dataset reinforcement is a one-time cost amortized across efficient-model training and experimentation.Source: §3.1 · p. 3

3.2. Training

  • The training loss distills image-text pair affinities from multiple teacher encoders into student encoders, alongside the standard CLIP loss.The stored teacher embeddings let the student compute the total loss after its forward pass without extra teacher computations.Source: §3.2 · pp. 3–4

4. Architecture

MobileCLIP pairs a hybrid convolution-and-attention text encoder with an improved hybrid image encoder. The image encoder adjusts FFN expansion and depth or width to improve capacity while maintaining parameter efficiency.

4.1. Text Encoder

  • Text-RepMixer combines 1-D convolutions and self-attention; its inference-time architecture reparameterizes skip connections for efficient inference.With efficient ViT-S/16 backbones, the MCt text encoder is smaller and faster while achieving performance similar to CLIP’s Base text encoder.Source: §4.1 · p. 4

4.2. Image Encoder

  • MCi lowers the FFN expansion ratio from 4.0 to 3.0 and increases depth while retaining the same number of image-encoder parameters.MCi1 is deeper than MCi0, while MCi2 is wider than MCi1; the design has minimal latency impact and improves capacity reflected in downstream performance.Source: §4.2 · pp. 4–5
  • MCi outperforms similar-sized FastViT-MA36 on zero-shot ImageNet validation while running 16.3% faster.The comparison uses the image encoders in a CLIP model with the Base text encoder.Source: §4.2 · p. 5, Table 1 · p. 5

5. Experiments

The experiments assess MobileCLIP and DataCompDR across ablations, small-scale training, learning efficiency, state-of-the-art comparisons, and retrieval benchmarks. They report improved latency-accuracy tradeoffs and substantial learning efficiency, alongside gains in average benchmark and compositional performance.

5. Experiments

  • The experiments evaluate zero-shot classification, distribution shifts, image-text retrieval, 38-dataset average performance, and ARO, without fine-tuning.They use DataComp-1B and a fixed 12.8M-pair DataComp-12M subset; DataCompDR adds synthetic captions, image augmentations, and teacher embeddings.Source: §5 · p. 5

5.1. Ablation Studies

  • Strong image augmentations improve distillation by +4.8% on IN-val and +4.4% on Flickr30k; performance nearly saturates at 5 augmentations and 2 synthetic captions.The authors use 10 augmentations for DataCompDR-12M and 30 for DataCompDR-1B, with 5 synthetic captions for maximal performance.Source: §5.1 · p. 6
  • Without dataset reinforcement, training is 16× slower; partial synthetic-caption reinforcement is 3× slower than regular CLIP training.The timing comparison measures one epoch on DataCompDR-12M using a single node with 8× A100-80GB GPUs.Source: §5.1 · p. 6, Table 4 · p. 6
  • The hybrid text encoder with 2 Text-RepMixer blocks and 4 self-attention layers matches the pure transformer while being 5% smaller and 15.8% faster.Introducing even one self-attention layer substantially improves zero-shot performance over the purely convolutional encoder.Source: Table 5 · p. 6, §5.1 · p. 7

5.3. Learning Efficiency

  • DataCompDR training reaches 71.7% ImageNet-val zero-shot accuracy after 120 epochs, versus 55.7% at best for non-reinforced training.With 1.28M samples, DataCompDR exceeds 55.2% accuracy while DataComp-1B reaches about 6%; the paper reports over 100× data efficiency and 1000× on Flickr30k.Source: §5.3 · p. 7, Figure 6 · p. 7

5.2. Small Scale Regime

  • With fewer than 370M samples, MobileCLIP-B on DataCompDR-12M outperforms methods trained up to 4× longer; TinyCLIP-8M/16 scores 41.1% versus MobileCLIP-S0's 59.1%.The two models have close latency: 2.6 ms and 3.1 ms, respectively.Source: Table 6 · p. 7, §5.2 · p. 7

5.4. Comparison with State-of-the-art

  • MobileCLIP-S2 improves average performance on 38 datasets by 2.8% over ViT-B/32-256, while being 1.5× smaller and 1.4× faster.MobileCLIP-B improves average performance on 38 datasets by 2.9% over SigLIP-B/16 and is 26.3% smaller.Source: Table 7 · p. 8, §5.4 · p. 8

5.5. Retrieval Performance Analysis

  • MobileCLIP-B improves Visual Genome Relation and Attributes accuracy over SigLIP by 19.5% and 12.4%, and Flickr30k-Order and COCO-Order recall@1 by 69.7% and 50.3%.The cited comparisons concern the ARO benchmark.Source: Table 8 · p. 8, §5.5 · p. 8

6. Conclusion

The conclusion presents MobileCLIP as an on-device image-text model family and DataCompDR as its knowledge-reinforced training dataset. It reports improved learning efficiency, latency-accuracy tradeoffs, and ARO performance.

6. Conclusion

  • MobileCLIP aligned image-text backbones are designed for on-device CLIP inference with low latency and size.The conclusion characterizes the models as achieving a state-of-the-art latency-accuracy tradeoff against previous work.Source: §6 · p. 8
  • DataCompDR reinforces DataComp with knowledge from a pretrained image-captioning model and an ensemble of strong CLIP models.The conclusion reports 10×–1000× learning efficiency with the reinforced dataset.Source: §6 · p. 8
  • MobileCLIP models show better robustness and improved performance on the Attribute, Relation and Order benchmark.Source: §6 · p. 8

Supplementary Material

The supplement describes three FastViT stage configurations and points to comparisons of their image encoders in a supervised ImageNet-1k setting.

A. Image Encoder Configurations

B. Experimental Setup

The appendix specifies training and evaluation details, including ImageNet-1k settings and configurations used across MobileCLIP variants.

B. Experimental Setup

  • MobileCLIP-B uses the listed DataCompDR-1B hyperparameters; S0, S1, and S2 use the same settings except λ = 1.0.DataCompDR-12M experiments use global batch size 8192, and models use strong image augmentation unless stated otherwise.Source: B. Experimental Setup · p. 13
  • Ensemble-distillation ablations use 32 A100 GPUs, while ablations in Appendices C and D use a uniformly sampled DataComp-8M subset.The reported DataCompDR seen sample is one augmented image, one ground-truth caption, and one randomly selected synthetic caption.Source: B. Experimental Setup · p. 13
  • ImageNet-1k experiments use 300 epochs, AdamW, weight decay 0.05, peak learning rate 10^-3, and total batch size 1024.The models use five warmup epochs, cosine learning-rate decay, and RegNetY-16GF as the distillation teacher.Source: B. Experimental Setup · p. 13, Table 10 · p. 13
  • The MCi variants obtain the best trade-off among recent efficient architectures in Figure 7.Source: B. Experimental Setup · p. 13

C. Image Augmentation

The ablations find that ensembling improves zero-shot CLIP performance and that stronger image augmentation benefits distillation. They also identify a two-model CLIP ensemble as the selected teacher for dataset reinforcement.

C. Image Augmentation

  • Strong RandomResizedCrop improves both classification and retrieval, whereas the commonly used CLIP recipe uses weak cropping.The comparison feeds the same augmented image to teacher and student; lower crop-area bounds of 0.08, 0.4, and 0.9 represent decreasing augmentation strength.Source: C. Image Augmentation · p. 14, Table 13 · p. 14
  • RangeAugment adjusts brightness, contrast, and noise dynamically, so it is applied only to student images in dataset reinforcement.Applying the same augmentation to teacher and student yielded 56.6% versus 55.9% on ImageNet-val, but was not feasible for the reinforcement approach.Source: C. Image Augmentation · p. 14
  • Among RandomHorizontalFlip, RandomErasing, and RandAugment, only RandAugment benefits this setup.Reinforced datasets store RandomResizedCrop and RandAugment parameters; RangeAugment is applied to student images during training.Source: C. Image Augmentation · p. 14

D. CLIP Ensembles

  • 67.3% average performance is achieved by ensembling two ViT-L-14 CLIP models, versus 66.3% and 61.7% for the individual models.The models are from datacomp_xl_s13b_b90k and openai, respectively.Source: D. CLIP Ensembles · p. 14
  • 81.7% zero-shot classification on ImageNet-val is reached by an ensemble of four CLIP models, while individual models reach at most 80.1%.The two-model ViT-L-14 ensemble also matches the 80.1% ImageNet-val performance of a ViT-bigG-14 image encoder with fewer parameters.Source: D. CLIP Ensembles · pp. 14–15
  • In the teacher comparison, more accurate CLIP models were not necessarily better teachers, so the authors selected an ensemble of two ViT-L-14-based CLIP models for dataset reinforcement.The comparison trained a ViT-B/16 CLIP model with different individual-model and ensemble teachers, using a uniformly sampled 8M subset.Source: D. CLIP Ensembles · p. 15

E. Ablations on Lossy Compressions

The ablations examine storage reductions from fewer augmentations and lossy embedding compression, and trade off hybrid text-encoder kernel size against mobile latency. They identify BFloat16 as accuracy-preserving within training variability and kernel size 11 as the best accuracy-latency choice.

E. Ablations on Lossy Compressions

  • BFloat16 embedding training achieves accuracy within the standard deviation of training on DataComp-12M.The storage analysis considers reducing augmentation count and lossy embedding compression; the reported storage measurements use 12.8k DataCompDR samples.Source: E. Ablations on Lossy Compressions · p. 15

F. Hybrid Text Encoder

  • A 1.1% zero-shot ImageNet-val improvement comes with a 4.5× slowdown as hybrid text-encoder kernels grow; kernel size 11 has the best accuracy-latency trade-off.The ablation uses a six-layer fully convolutional text encoder, a ViT-B/16 image encoder, DataCompDR-12M, and 30k training iterations.Source: F. Hybrid Text Encoder · p. 15
  • The hybrid text encoder uses depth-wise 2D convolutions for token interactions and FFN layers for channel-dimension interactions.For CLIP, the sequence length is 77; the input tensor is reshaped before the convolutional layer.Source: F. Hybrid Text Encoder · p. 15

G. Performance of other models on DataCompDR-12M

DataCompDR-12M improves zero-shot ImageNet-val accuracy across the tested image encoders. The extended results cover classification and retrieval for the MobileCLIP family, alongside related works with partial evaluations.

G. Performance of other models on DataCompDR-12M

  • 10.6% higher zero-shot ImageNet-val performance is obtained by a 4.9M-parameter MobileNetV3-L trained on DataCompDR-12M.The reported accuracy improvement applies to all tested image-encoder sizes and requires no training overhead.Source: G. Performance of other models on… · p. 15

H. Extended Results

  • Extended zero-shot classification and retrieval results cover MobileCLIP-S0, S1, S2, and B, plus related works with only partial evaluations.The related-work results include methods whose full DataComp evaluation was unavailable.Source: H. Extended Results · p. 15

I. Long training

Continuing MobileCLIP-B training beyond 13B seen samples improves average performance across 38 datasets and zero-shot ImageNet-val accuracy. A shorter continuation reaches an average-performance gain with fewer total seen samples.

I. Long training

  • Continual runs restart training on DataCompDR-1B from cosine- or constant-schedule checkpoints; DataCompDR and DataComp training have the same wall-clock time.A seen sample is a triplet of one randomly augmented image, one ground-truth caption, and one randomly selected synthetic caption.Source: §I · p. 15
  • At 39B total seen samples, MobileCLIP-B improves average performance on 38 datasets by 0.6% and zero-shot ImageNet-val accuracy by 0.4% over its 13B-sample training.The table reports retrieval performance at 1 and average performance across 38 datasets.Source: §I · p. 15, Table 21 · p. 18
  • At 18B total seen samples, continuing the original 13B-sample training yields a 0.4% improvement in average performance on 38 datasets.This continuation uses Cosine(40k, 131k, 2k).Source: §I · p. 16, Table 21 · p. 18
Loading 2311.17049v2…