Source-linked AI summary
AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities
Zhongzhi Chen, Guang Liu, Bo-Wen Zhang, Fulong Ye, Qinghong Yang, Ledell Wu
TL;DR
Cross-lingual vision-language models have often emphasized retrieval, while retrieval tests can be small and distribution-sensitive and some methods perform poorly on ImageNet classification. AltCLIP replaces CLIP's text encoder with XLM-R and uses teacher learning followed by contrastive learning. The paper reports strong Chinese classification and retrieval results, close English performance to CLIP, and state-of-the-art multilingual retrieval on XTD.
Problem
Cross-lingual methods often focus on retrieval, whose small test sets and sensitivity to training-data distribution leave generalization under-addressed; current methods may also perform poorly on ImageNet classification.
Source: §1 · p. 1Method
AltCLIP replaces CLIP's text encoder with XLM-R, first distilling alignment from CLIP using parallel text and then fine-tuning with contrastive learning on text-image pairs.
Source: §1 · pp. 1–2, §3.1 · p. 3Results
The method reports new state-of-the-art results on multiple Chinese zero-shot classification and retrieval tasks, close performance to CLIP on almost all tasks, and state-of-the-art zero-shot results on XTD for AltCLIPM9.
Source: §1 · pp. 1–2, §6 · p. 9Takeaways & Limitations
The paper shows that altering CLIP's text encoder can yield bilingual and multilingual multimodal models while using two million bilingual text-image pairs for training.
Source: §6 · p. 9The authors leave for future work a proper analysis of whether AltCLIP's retrieval gains stem from the different-distribution LAION subset or XLM-R's language understanding.
Source: §5.2 · p. 6
Abstract
from arXiv · showhide
In this work, we present a conceptually simple and effective method to train a strong bilingual/multilingual multimodal representation model. Starting from the pre-trained multimodal representation model CLIP released by OpenAI, we altered its text encoder with a pre-trained multilingual text encoder XLM-R, and aligned both languages and image representations by a two-stage training schema consisting of teacher learning and contrastive learning. We validate our method through evaluations of a wide range of tasks. We set new state-of-the-art performances on a bunch of tasks including ImageNet-CN, Flicker30k-CN, COCO-CN and XTD. Further, we obtain very close performances with CLIP on almost all tasks, suggesting that one can simply alter the text encoder in CLIP for extended capabilities such as multilingual understanding. Our models and code are available at https://github.com/FlagAI-Open/FlagAI.
1 Introduction
The introduction frames multilingual vision-language modeling as a way to extend CLIP, while noting that prior cross-lingual work often emphasizes retrieval despite its limited and distribution-sensitive evaluation. AltCLIP combines teacher learning and contrastive learning to target strong English and Chinese performance across classification and retrieval tasks.
1 Introduction
- Training vision-language models can require hundreds of millions of text-image pairs; CLIP used 400M, while Taiyi used 123M.The introduction notes efforts to reuse pretrained models and train only part of a network.Source: §1 · p. 1
- Existing cross-lingual methods often focus on retrieval, whose small test sets and sensitivity to training-data distribution leave generalization insufficiently addressed.The introduction points to ImageNet classification over 1,000 classes as an indicator of generalization.Source: §1 · p. 1
- AltCLIP targets strong performance on ImageNet and multimodal retrieval in both English and Chinese.It uses teacher learning followed by contrastive learning on a relatively small amount of Chinese and English text-image pairs.Source: §1 · pp. 1–2
- The multilingual extension, AltCLIPM9, achieves state-of-the-art zero-shot results on the multilingual text-image retrieval dataset XTD.The paper also reports state-of-the-art results on multiple Chinese image-classification and retrieval tasks.Source: §1 · p. 2
2 Related Work
Prior work extends CLIP or trains Chinese vision-language models, but the introduction describes limitations in English-task performance and generalization. AltCLIP retains CLIP as a starting point while replacing its text encoder and adding text-image fine-tuning.
2 Related Work
- Prior Chinese text-image models often require large Chinese text-image datasets and show substantial performance declines on English tasks.The passage names Taiyi, CN-CLIP, Wukong, R2D2, and BriVL as examples.Source: §2 · p. 2
- Carlsson et al. (2022) distill CLIP's text encoder using machine-translated parallel data without text-image pairs, but report a significant drop in English zero-shot classification.AltCLIP adds English text pairs, human-curated translations, and text-image fine-tuning.Source: §2 · p. 2
- AltCLIP uses multilingual XLM-R as its text encoder and aligns it with CLIP's image encoder for cross-lingual and cross-modal tasks.The related-work passage describes XLM-R as a multilingual language model with strong cross-lingual task performance.Source: §2 · p. 2
3 Methodology
AltCLIP first transfers CLIP text-image alignment to an XLM-R student using parallel text, then further aligns text and image representations with contrastive learning on text-image pairs. The multilingual version applies the same approach across nine languages.
3.1 Teacher Learning Stage
- Teacher learning uses CLIP's text encoder as teacher and multilingual XLM-R as student, adding a fully connected layer to match the teacher's output dimension.Parallel English and Chinese text is used to distill text-image alignment; no images are needed in this stage.Source: §3.1 · pp. 2–3, §3 · p. 2
- The teacher encodes sent1 to provide a [TOS] embedding target, while the student encodes parallel sent2 and is trained to match it using mean squared error.After training, only the student encoder is used at inference; the teacher is training-only.Source: §3.1 · p. 3
- AltCLIPM9 extends the approach to English, Chinese, Spanish, French, Russian, Arabic, Japanese, Korean, and Italian..Source: §3.1 · p. 3
3.2 Contrastive Learning Stage
- Contrastive learning pairs the student text encoder with CLIP's Vision Transformer image encoder to further improve text-image alignment.The image encoder is frozen, and only text-encoder parameters are updated using contrastive loss.Source: §3.2 · p. 3
4 Model Training
Training uses parallel text for teacher learning and text-image pairs for contrastive learning, with separate data volumes for bilingual and multilingual models. The text encoder starts from XLM-RLarge, while CLIP ViT-L/14 supplies the teacher and image encoder.
4.1 Training Datasets
- Teacher learning uses machine-translated CC3M and Laion-400M text, plus human-translated TSL2019 English-to-Chinese data and multilingual parallel data from OPUS.The listed Laion-400M subsets are 28M for bilingual training and 10M per language for the multilingual model; TSL2019 contributes 5M English-to-Chinese translations.Source: §4.1 · pp. 3–4
- Contrastive learning uses 2M bilingual text-image pairs for AltCLIP and 100M multilingual pairs for the multilingual model.Sources include filtered Wudao MM Chinese data, LAION 5B English data, and LAION Multilingual 2B data.Source: §4.1 · p. 4
4.2 Implementation details
- The text encoder is initialized from XLM-RLarge, and CLIP ViT-L/14 provides both the teacher text encoder and image encoder.The passage directs readers to Appendix Table 6 for hyperparameters.Source: §4.2 · p. 4
5 Experiments
The experiments evaluate AltCLIP across English, Chinese, and multilingual classification and retrieval, then test training-stage and data choices and multilingual text-to-image generation.
5.1 Evaluation Datasets and Metrics
- Evaluation spans zero-shot classification on ImageNet and four out-of-distribution variants, retrieval on English and Chinese Flickr30k and MSCOCO, and multilingual retrieval on XTD.XTD includes English and ten other languages; the experiments also cover a wider set of English classification benchmarks.Source: §5.1 · pp. 4–5
5.2 Zero-shot performance
- AltCLIP is close to CLIP in English classification and sets new state-of-the-art results on Chinese ImageNet, ImageNet-A, ImageNet-R, and ImageNet V2.Compared with Chinese baselines using hundreds of millions of text-image pairs, AltCLIP uses 36M parallel text data and 2M text-image pairs.Source: §5.2 · pp. 5–6
- AltCLIP outperforms all Chinese retrieval baselines on most datasets and tasks and also outperforms CLIP in both text-to-image and image-to-text retrieval.Table 2 covers English and Chinese Flickr30k and MSCOCO retrieval; the compared models use ViT-L image encoders.Source: Table 2 · p. 5, §5.2 · p. 6
- AltCLIPM9 achieves state-of-the-art results in seven languages on XTD, maintaining state-of-the-art English capability and outperforming CLIP on this dataset.Table 3 reports image-to-text Recall@10 on XTD.Source: §5.2 · p. 6, Table 3 · p. 6
- AltCLIP is close to CLIP across broad English zero-shot benchmarks; contrastive learning improves performance, especially retrieval, while AltCLIPM9 stays close to bilingual AltCLIP.Figure 2 compares the teacher-learning and contrastive-learning stages, bilingual and multilingual models, and CLIP using CLIP ViT-L/14 image encoders.Source: Figure 2 · p. 6, §5.2 · p. 7
5.3 Ablation study
- Removing English-English pairs drops English ImageNet accuracy from 53.8 to 15.47; machine-translated English-Chinese data also materially affects Chinese ImageNet and Flickr30kCN performance.The ablations vary English-English, machine-translated and human-translated parallel data, and use of contrastive learning.Source: §5.3 · p. 7, Table 4 · p. 7
5.4 Examples of text-to-image generation
- AltDiffusion generates high-quality images close to Stable Diffusion from English and Chinese prompts, with similar outputs for translated prompts; multilingual prompts can reflect cultural differences.AltDiffusion uses AltCLIP or AltCLIPM9 as its language encoder; Table 5 shows AltDiffusionM9 generations for a prompt translated into nine languages with a fixed seed.Source: §5.4 · p. 7, Table 5 · p. 8
6 Conclusion
The paper presents a two-stage teacher-learning and contrastive-learning approach for bilingual and multilingual multimodal representations. In Chinese, it reports state-of-the-art zero-shot classification and retrieval results with less training data than most prior work, and identifies two directions for future research.
6 Conclusion
- The method sets new state-of-the-art results on multiple Chinese zero-shot image-classification and retrieval tasks.The experiments cover a wide range of tasks in English and Chinese.Source: §6 · p. 9
- Training uses tens of millions of text data and 2 million text-image pairs, compared with hundreds of millions of text-image pairs for most prior work.Source: §6 · p. 9
- The proposed two-stage schema combines teacher learning and contrastive learning to train bilingual and multilingual multimodal representation models.Source: §6 · p. 9
- Future work includes altering the image encoder to combine vision signals from different data distributions and possibly eliminating machine-translated data.Source: §6 · p. 9
A Hyper-parameters Optimization
Table 6 lists hyperparameter settings for the Teacher Learning and Contrastive Learning stages.
A Hyper-parameters Optimization
- Table 6 presents hyperparameter settings for the Teacher Learning and Contrastive Learning stages.Source: Table 6 · p. 12