PaperlayerView exact revision on arXiv

Source-linked AI summary

LiT: Zero-Shot Transfer with Locked-image text Tuning

Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, Lucas Beyer

arXiv:2111.07991v3cs.CVcs.CLcs.LG

TL;DR

The paper addresses how to obtain zero-shot vision transfer while benefiting from both broad image-text data and strong pre-trained image representations. It proposes contrastive-tuning, particularly LiT, which trains the text tower against a locked image tower. LiT reaches 85.2% ImageNet and 82.5% ObjectNet accuracy, while improving transfer across varied models and datasets.

  • Problem

    Image-text data supports broad zero-shot transfer but may be too noisy for state-of-the-art image descriptors, while curated image data is limited to predefined categories.

  • Method

    LiT tunes an unlocked text tower with image-text data while keeping a strong pre-trained image tower locked.

  • Results

    85.2% zero-shot transfer accuracy is achieved on ImageNet and 82.5% on the challenging ObjectNet test set, with ObjectNet outperforming the previous state of the art by 10.2%.

  • Takeaways & Limitations

    LiT turns publicly available pre-trained vision models into zero-shot classifiers and halves the gap between from-scratch contrastive learning and supervised fine-tuning.

  • Takeaways & Limitations

    The work evaluates only classification and retrieval, leaving detection, segmentation, visual question answering, and image captioning for future work.

Abstract

from arXiv · show

This paper presents contrastive-tuning, a simple method employing contrastive training to align image and text models while still taking advantage of their pre-training. In our empirical study we find that locked pre-trained image models with unlocked text models work best. We call this instance of contrastive-tuning "Locked-image Tuning" (LiT), which just teaches a text model to read out good representations from a pre-trained image model for new tasks. A LiT model gains the capability of zero-shot transfer to new vision tasks, such as image classification or retrieval. The proposed LiT is widely applicable; it works reliably with multiple pre-training methods (supervised and unsupervised) and across diverse architectures (ResNet, Vision Transformers and MLP-Mixer) using three different image-text datasets. With the transformer-based pre-trained ViT-g/14 model, the LiT model achieves 85.2% zero-shot transfer accuracy on the ImageNet test set, and 82.5% on the challenging out-of-distribution ObjectNet test set.

1. Introduction

The paper introduces contrastive-tuning, which aligns text with a strong pre-trained image model, and identifies LiT as an effective locked-image configuration for zero-shot transfer.

  • Motivation: Zero-shot transfer uses no supervised examples during transfer, although relevant supervised information may have appeared during pretraining.It differs from classical zero-shot learning by allowing supervised information during pretraining.
  • Prior approach: Contrastive learning aligns paired image-text representations while separating non-paired representations, enabling classification through textual class descriptions.At test time, image embeddings are compared with embeddings of textual class descriptions.
  • Contribution: LiT tunes the text tower against a pre-trained image tower and finds that locking the image tower works best.LiT teaches a text model to read out suitable representations from a pre-trained image model.
  • Results: 85.2% zero-shot transfer accuracy is achieved on ImageNet with ViT-g/14, while ObjectNet accuracy reaches 82.5%, outperforming the previous state of the art by 10.2%.The ObjectNet result is reported on a challenging out-of-distribution test set.
  • Scope: LiT is evaluated across supervised and self-supervised pretraining, three image-text datasets, and ResNet, Vision Transformer, and MLP-Mixer architectures.The paper reports better performance than from-scratch contrastive learning for self-supervised pre-trained models.
  • Practical impact: LiT reuses publicly available pre-trained models and datasets to support high-performance zero-shot models with modest computational resources.The recipe amortizes the resources used to train image models.

2. Related work

Related work frames LiT within transfer learning, zero-shot transfer, contrastive learning, and image-text modeling, emphasizing the cost of task-specific fine-tuning.

  • Transfer learning: Transfer learning reuses pre-trained models to solve new tasks better and faster than training from scratch.The usual procedure pre-trains on large image data and then fine-tunes with supervised task data.
  • Motivation: Pre-trained models can provide strong low-data or few-shot capabilities, while collecting task-specific data and fine-tuning remains costly.These pressures motivate alternatives based on zero-shot transfer.
  • Zero-shot transfer: Zero-shot transfer avoids the fine-tuning stage and performs classification from descriptions of target classes.Image-text alignment is one approach for obtaining zero-shot models.
  • Image-text modeling: Web image-text collections support zero-shot classification and image/text retrieval because their descriptions are open-ended rather than restricted to predefined categories.Their flexibility comes from free-form natural language paired with images.
  • Contrastive learning: Contrastive learning trains models to select correct answers from finite candidate sets, encouraging attention to high-level image information.This framing motivates its use for representation learning.

3. Methods

The method uses contrastive image-text alignment while separating image representation learning from alignment by reusing pre-trained image models, especially in the LiT configuration.

  • Contrastive pre-training: Contrastive pre-training learns image and text embeddings of equal dimensionality, bringing paired embeddings together and separating non-paired embeddings.A global loss computed jointly across devices is used in the experiments.
  • Contrastive pre-training: Zero-shot classification embeds class descriptions with the text model and selects the label closest to the image embedding; the same approach supports retrieval.The trained towers can therefore be used for both image classification and image-text retrieval.
  • Contrastive-tuning: Contrastive pre-training jointly learns image embeddings and text alignment, although image-text data may be suboptimal for learning the best image embeddings.The method addresses this tension by decoupling the two objectives.
  • Contrastive-tuning: Contrastive-tuning initializes alignment with an image model trained on cleaner labeled data, preserving image embeddings while learning image-text alignment independently.This combines the advantages of curated image data and broad image-text data.
  • Design choices and LiT: The two-character notation records image-tower and text-tower settings: L is locked pretrained, U is trainable pretrained, and u is trainable randomly initialized.The notation distinguishes initialization and whether weights are locked.
  • Design choices and LiT: LiT is the Lu setting, pairing a locked pretrained image model with an unlocked randomly initialized text model.Experiments identify this setting as particularly effective.

4. Image-text datasets

The experiments use public and private image-text datasets, including filtered YFCC100m and CC12M data, plus a large privately collected corpus.

  • CC12M: CC12M contains 12 million image-text pairs, but expired URLs reduced the experimental dataset to 10 million pairs.The dataset extracts, filters, and transforms image and alt-text pairs from web pages.
  • YFCC100m: YFCC100m contains 100 million media objects, including 99.2 million photos with rich metadata.The experiments use a 15-million-image subset filtered for high-quality English text, called YFCC100m-CLIP.
  • Private dataset: The privately collected corpus contains 4 billion image and alt-text pairs and removes near-duplicate images from evaluated dataset splits.Its collection process follows ALIGN with simpler text filtering.

5. Experiments

Experiments show that LiT improves zero-shot transfer and robustness while using pretrained image representations efficiently. Across design studies, locking the image tower generally preserves broader representation quality, though some retrieval settings favor unlocking with sufficient compute.

  • 5.1. Comparison to the previous state-of-the-art: 9% and 8.8% improvement over CLIP and ALIGN, respectively, on ImageNet zero-shot classification narrows the gap with supervised fine-tuning.LiT is evaluated against prior state-of-the-art methods on ImageNet, five out-of-distribution variants, and seven VTAB-natural tasks.
  • 5.1. Comparison to the previous state-of-the-art: 82.5% accuracy on ObjectNet establishes a new state-of-the-art, while LiT consistently outperforms previous models across ImageNet out-of-distribution variants.The pretrained ViT-g/14 reaches 82.5% on ObjectNet, compared with 70.5% after ImageNet fine-tuning.
  • 5.1. Comparison to the previous state-of-the-art: 81.7% top-1 accuracy on zero-shot ImageNet transfer requires only 300M image-text pairs, versus 76.2% for CLIP after 12.8B pairs.LiT converges faster than from-scratch setups and reuses pretrained image models, amortizing image-model training resources.
  • 5.2. Evaluation of design choices: Locking a pretrained image tower works best in the design study, while pretraining the text tower provides only a small benefit.The study varies initialization, locking, learning rates, weight decay, and training duration on YFCC100m-CLIP.
  • 5.2. Evaluation of design choices: Locked-image tuning yields worse loss on the alignment dataset but better out-of-distribution loss and broader image representations than contrastively fine-tuning the image tower.Fine-tuning improves performance on the contrastive dataset while worsening generality elsewhere; LiT aligns text to a strong, general image representation.
  • 5.3. LiT works better for more generally pretrained models: Generic pretraining produces similarly good image-text models after LiT, whereas narrowly pretrained models can perform well on narrow tasks but fall behind on general retrieval.The authors evaluate supervised and unsupervised pretraining across multiple architectures, datasets, and downstream tasks.
  • 5.4. Which text model to use?: BERT text encoders improve smaller-dataset results, especially retrieval, but this advantage disappears at large scale and BERT is less stable to train.Text-model capacity consistently improves performance, with a more pronounced effect for the image tower.
  • 5.2. Evaluation of design choices: Locked image towers reduce training time and memory, permit one-time embedding precomputation without augmentation, and support enormous models with very large batch sizes.On cross-modal retrieval, Uu or UU can overtake Lu under very long schedules when zero-shot classification is not the primary goal.

6. Discussion

The paper identifies task and societal boundaries for LiT. Its benefits are less clear for retrieval under larger budgets, and easier attachment of text towers also creates misuse risks.

  • LiT is evaluated only on classification and retrieval, leaving detection, segmentation, visual question answering, and image captioning for future work.
  • On cross-modal retrieval, Lu has no clear advantage over Uu or UU, and those setups can overtake it under very long tuning schedules.Within a fixed budget, Lu may still save computational cost; with a large budget, Uu may merit consideration when classification is not the primary goal.
  • LiT can simplify creation of malicious, offensive, or obscene text-tower counterparts for existing image models.The authors call for further research on equipping open-world image-text models with desired behavior.

7. Conclusion

The paper concludes that contrastive-tuning can convert pre-trained vision models into zero-shot systems using public data. LiT improves zero-shot transfer and extends across model families, while motivating reuse of existing models.

  • Contrastive-tuning transfers any pre-trained vision model to zero-shot use, with LiT providing substantial quality improvements on zero-shot transfer tasks.The paper reports that LiT halves the gap between from-scratch contrastive learning and per-task supervised fine-tuning.
  • LiT can turn publicly available models and data into zero-shot classifiers that rival prior work using more proprietary data.
  • The authors hope the results motivate smarter reuse and adaptation of pre-trained models for different research problems.
  • LiT works across ViT, Mixer, and ResNet model families, supporting applicability beyond Transformer-based image models.The comparison reports zero-shot top-1 ImageNet accuracy and inference speed for different model families.

B. Larger model capacity yields better results

Larger image and text towers generally improve LiT performance, with image-tower capacity providing the stronger ImageNet gain. The section also examines data signals, batch size, pre-computation, and practical tuning choices.

  • Model capacity: Increasing pre-trained image-tower capacity improves zero-shot ImageNet accuracy more than increasing text-tower capacity.In the private-data setup, performance rises from 74.5% with B/32 and a base text tower to 81.2% with g/14 and a huge text tower.
  • Model capacity: 61.1% with a B/32 image tower and base text tower increases to 67.6% with an L/16 image tower and large text tower on public CC12M data.The B/16-to-L/16 increase with the large text tower is more modest, from 66.9% to 67.6%.
  • YFCC100m signals: Jointly using all three YFCC100m text signals consistently outperforms sampling one signal per image or minibatch, though it requires triple the text-tower passes.
  • YFCC100m signals: Using the curated YFCCCLIP subset gives the best overall LiT results in this investigation, while full YFCC100m can be better for retrieval with only titles or tags.The investigation used the small ViT-B/32 model, so larger models may perform better on the full dataset.
  • Batch size and pre-computation: Larger effective batch sizes improve performance consistently, with global contrastive loss helping by introducing more hard negatives.
  • Batch size and pre-computation: Pre-computing image embeddings accelerates multi-epoch tuning and increases feasible batch size, but image augmentations reduce its speed benefit.Memory benefits remain even when augmentations prevent substantial computational gains from pre-computation.

H. Learning rate schedules

Learning-rate schedules trade off zero-shot ImageNet and retrieval-oriented metrics. Delaying or reducing image-tower updates can improve retrieval, while the default schedules remain strongest or weakest depending on the evaluated metric.

  • Training the image tower with a smaller learning rate or delaying its updates improves retrieval metrics.
  • The default LU and UU schedules achieve the best and worst ImageNet zero-shot accuracies, respectively, among the tested schedules.
  • Delaying the image learning rate modestly improves ITR, VTAB, and ImageNet zero-shot accuracy relative to UU.
  • Using an image learning rate of 1e-4 improves ImageNet zero-shot accuracy more but reduces VTAB accuracy.
  • Sigmoid learning-rate scaling produces a similar result, while two-cycle freezing and unfreezing performs between the lr+dl and lr=1e-4 schedules.

I.2. VTAB Evaluation

VTAB evaluates zero-shot models across natural, specialized, and structured visual tasks, revealing strong difficulty on structured tasks despite prompt engineering. The evaluation uses validation-selected preprocessing, prompts, and class names before testing.

  • Benchmark: VTAB contains 19 diverse visual tasks spanning natural, specialized, and structured categories.Structured tasks include scene-structure assessments such as 3D depth estimation and counting.
  • Evaluation protocol: Validation-selected preprocessing, prompt templates, and class names are evaluated on the official VTAB test set.The search uses a validation set of 800 images, including task-specific settings such as satellite-image prompts.
  • Limitations: The authors note that selecting settings on validation data is arguably not zero-shot transfer, while presenting it as principled and reproducible.This caveat concerns the evaluation protocol rather than the model’s training procedure.
  • Results: Structured VTAB performance does not significantly deviate from random guessing despite extensive prompt engineering.The paper leaves improving counting and distance assessment as an open research direction.

J. Multilingual details and limitations

The multilingual evaluation examines English performance, tokenization, translation quality, and backtranslation. It finds that multilingual training preserves English performance and that backtranslation can improve metrics, while several dataset and translation artifacts limit interpretation.

  • Multilingual evaluation: Training on the larger, more diverse multilingual dataset does not come at the expense of English zero-shot ImageNet performance.The comparison covers English and multilingual T5 models trained with LiT on YFCCCLIP versus YFCC100m.
  • Benchmark limitations: Wiki-Image Text is an imperfect multilingual benchmark because proper nouns often remain unchanged across translations.This can allow monolingual models to score higher than their actual multilingual understanding would warrant.
  • Tokenization: SentencePiece byte encoding can produce sequences exceeding the 16-token maximum, preventing some inputs from using pre-training effectively.The paper nevertheless reports that SentencePiece allows T5 models to learn decent non-English representations despite an ill-suited English-only vocabulary.
  • Translation limitations: Imperfect prompt translations likely underestimate non-English performance.The authors identify translation quality as a limitation of their multilingual setup.
  • Translation limitations: English words embedded in some translations can support performance without requiring full understanding of the target language.The paper gives Vietnamese prompts containing the English word “airship” as an example.
  • Backtranslation: Backtranslation improves all metrics at a 10% probability, which the authors describe as a good trade-off.The augmentation translates CC12M texts through nine languages and back, with the probability shared across candidates.

K. More de-duplication results

Additional experiments test whether duplicate examples explain strong zero-shot transfer results. Larger-architecture runs reproduce earlier conclusions, providing further evidence against duplication as the root cause.

  • Experimental setup: Larger-architecture ablations use a pre-trained ViT-L/16 image model with a randomly initialized, unlocked large text model.The experiments follow the Lu setup to examine duplication effects at greater capacity.
  • Results: The larger-architecture results are consistent with runs using the pre-trained ViT-B/32 image model.This consistency is reported across the three upstream de-duplication setups summarized in Table 9.
  • Conclusion: The results provide further evidence that duplications are not the root cause of good zero-shot transfer results.The conclusion is based on the consistency of findings across model scales and de-duplication experiments.

L. Image-text dataset comparison

The paper compares a larger, more simply filtered image-text dataset with ALIGN-style data and examines broader zero-shot capabilities. Dataset size helps more with longer training, while qualitative evaluations cover open-vocabulary, multilingual, and failure behaviors.

  • Dataset comparison: The authors’ dataset contains 3.6B image-text pairs, compared with 1.8B pairs in ALIGN.The larger dataset uses simpler text filters.
  • Dataset comparison: Dataset-size differences are small below one epoch but become more visible as training continues.Training schedules range from 900M to 18B seen images.
  • Dataset comparison: The larger dataset with simpler filters slightly outperforms the smaller, more heavily filtered dataset.The paper leaves thorough exploration of this comparison to future work.
  • Evaluation scope: The authors caution that classification and retrieval may probe simple concepts and call for benchmarks beyond fine-grained visual classification.They state that the presented qualitative qualities are not representative.
  • Open-vocabulary evaluation: LiT model predictions support classification and retrieval with open vocabularies rather than a fixed label set.The qualitative evaluation uses manually constructed image-text pairs.
  • Multilingual capabilities: Multilingual LiT recognizes concepts in Thai and Chinese, while also ranking English translations or transliterations highly.The authors connect multilingual capability with greater inclusivity and accessibility for non-English speakers.
  • Qualitative failures: Model outputs can be manipulated by changing candidate texts so that alternatives rank lower.Figure 18 presents qualitative failure examples illustrating this sensitivity.
Loading 2111.07991v3…