PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages

Jinyi Hu, Yuan Yao, Chongyi Wang, Shan Wang, Yinxu Pan, Qianyu Chen, Tianyu Yu, Hanghao Wu, Yue Zhao, Haoye Zhang, Xu Han, Yankai Lin, Jiao Xue, Dahai Li, Zhiyuan Liu, Maosong Sun

arXiv:2308.12038v3cs.CLcs.CV

TL;DR

Non-English multimodal models are difficult to train because high-quality image-text data is scarce, despite advances in English. MPM uses a multilingual LLM to pivot visual alignment from English to other languages; VisCPM reports strong Chinese results using English-only multimodal pretraining, with transfer also demonstrated across six languages.

  • Problem

    Large multimodal models have mainly succeeded in English, while non-English development is constrained by scarce high-quality image-text data.

    Source: §1 · p. 1

  • Method

    MPM uses a multilingual LLM to connect visual signals and target languages, training multimodal models on English image-text data for cross-language transfer.

    Source: §1 · pp. 1–2

  • Results

    VisCPM achieves state-of-the-art performance among Chinese open-source multimodal models using English-only multimodal data, and the work extends the approach to a six-language chatbot.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The reported Chinese results show that English-only multimodal pretraining can support strong Chinese image-to-text and text-to-image models.

    Source: §6 · p. 9

Abstract

from arXiv · show

Recently there has been a significant surge in multimodal learning in terms of both image-to-text and text-to-image generation. However, the success is typically limited to English, leaving other languages largely behind. Building a competitive counterpart in other languages is highly challenging due to the low-resource nature of non-English multimodal data (i.e., lack of large-scale, high-quality image-text data). In this work, we propose MPM, an effective training paradigm for training large multimodal models in non-English languages. MPM demonstrates that Multilingual language models can Pivot zero-shot Multimodal learning across languages. Specifically, based on a strong multilingual large language model, multimodal models pretrained on English-only image-text data can well generalize to other languages in a (quasi)-zero-shot manner, even surpassing models trained on image-text data in native languages. Taking Chinese as a practice of MPM, we build large multimodal models VisCPM in image-to-text and text-to-image generation, which achieve state-of-the-art (open-source) performance in Chinese. To facilitate future research, we open-source codes and model weights at https://github.com/OpenBMB/VisCPM.git.

1 INTRODUCTION

Large multimodal models have advanced in image-to-text and text-to-image generation, but their success has largely been limited to English, where non-English development is constrained by scarce high-quality image-text data. MPM uses a multilingual language model to transfer visual alignment from English to other languages; VisCPM demonstrates this approach in Chinese and across six supported languages.

Motivation

  • Non-English multimodal research lags as training competitive models requires large quantities of high-quality image-text pairs that are scarce in those languages.The cited examples use more than 100M pairs for BLIP-2 pretraining and more than 2B for Stable Diffusion.Source: §1 · p. 1

MPM

  • MPM first establishes multilingual alignment with a multilingual LLM, then trains visual modules on English image-text pairs to transfer multimodal alignment to other languages.English serves as a pivot between visual signals and languages with limited multimodal data.Source: §1 · pp. 1–2

VisCPM

  • VisCPM, pretrained only on English image-text pairs, surpasses existing Chinese multimodal models trained on native-language pairs; the work also reports open-source state-of-the-art Chinese models and a six-language chatbot.The chatbot supports English, German, French, Spanish, Italian, and Portuguese.Source: §1 · p. 2

2 RELATED WORK

Prior multilingual vision-language work includes extending image-text models, building multilingual frameworks, and testing cross-language transfer. This work presents MPM as a systematic training formulation and reports zero-shot transfer that can surpass models trained on native-language multimodal data.

Image-to-text models

  • Prior image-to-text work connects visual modules to LLMs with perceivers or projectors, and develops multimodal instruction-following datasets.Examples include BLIP-2 and InstructBLIP with perceivers, and LLaVA and PandaGPT with projectors.Source: §2 · pp. 2–3

Text-to-image models

  • Text-to-image models progressed from GANs and autoregressive generation to large-scale diffusion systems such as DALLE-2, Imagen, and Stable Diffusion.Source: §2 · p. 3

Multilingual multimodal models

  • Multilingual vision-language research has used distillation, contrastive learning, multilingual pretraining, and English instruction tuning to extend cross-language capabilities.PaLI uses 10B image-text pairs spanning 100 languages, while Ying-VLM reports generalization from English instruction tuning.Source: §2 · p. 3
  • MPM offers a more systematic training formulation and demonstrates zero-shot transfer that can surpass models trained on native-language multimodal data.Source: §2 · p. 3

3 MPM TRAINING PARADIGM

MPM transfers visual concepts from a resource-rich pivot language to a target language through multilingual alignment followed by multimodal alignment. Its image-to-text and text-to-image procedures use the multilingual LLM to support target-language transfer from pivot-language training data.

3.1 PROBLEM FORMULATION AND OVERVIEW

  • MPM addresses scarce target-language image-text pairs by using a resource-rich pivot language and a pretrained multilingual LLM to align target and pivot representations before visual alignment.The paradigm is described as agnostic to model architecture and training method.Source: §3.1 · p. 3

3.2 IMAGE-TO-TEXT GENERATION

  • Image-to-text generation uses an image encoder and multilingual LLM, with visual-module pretraining followed by instruction tuning on pivot-language data blended with translated target-language examples.During visual pretraining, the LLM parameters are fixed; instruction tuning fine-tunes both the visual module and LLM.Source: §3.2 · p. 4
  • With instruction tuning only in the pivot language, the model can understand target-language questions but tends to answer in the pivot language; adding a small number of translated pairs calibrates response language.The reported transfer is quasi-zero-shot when target-language translated examples are excluded.Source: §3.2 · p. 4

3.3 TEXT-TO-IMAGE GENERATION

  • For text-to-image generation, the decoder is trained to denoise images conditioned on the frozen multilingual LLM, whose similar-semantic target-language representations enable zero-shot transfer from pivot-language prompts.The model uses a denoising network with cross-attention between the LLM and image decoder.Source: §3.3 · p. 4

4 VISCPM

VISCPM applies a Chinese-English language model to Chinese image-to-text and text-to-image generation. Its dataset analysis motivates using English image-text data, while the two model variants also test adding Chinese image-text pairs.

4 VISCPM

  • The VISCPM series has two Chinese-English model variants: VISCPM-Chat for image-to-text multimodal conversation and VISCPM-Paint for text-to-image synthesis.Table 1 reports GPT-4-assessed LLaVA Test Set results across conversation, detailed description, complex reasoning, and their average; the supplied passage gives no table values.Source: §4 · p. 5, Table 1 · p. 6

4.1 ARE CHINESE MULTIMODAL DATASETS ENOUGH TO TRAIN A MULTIMODAL MODEL?

  • Wukong contains 100M native-Chinese image-text pairs, but only a minor fraction are semantically matched; translation is costly and adds marginal improvement with a powerful multimodal LLM backbone.The authors argue that transferring multimodal alignment from existing English data is key to building a powerful Chinese model; Figure 2 shows the CLIP-score histogram and uses 0.18 as a moderate filtering threshold.Source: §4.1 · p. 5, Figure 2 · p. 5

4.2 | VISCPM-CHAT

  • VISCPM-Chat aligns its visual module with a frozen LLM using 100M image-text pairs for 180K steps, then instruction-tunes on bilingual LLaVA 150K and UniMM-Chat plus English M3IT for 80K steps.The instruction-tuning data include Chinese translations of LLaVA 150K and UniMM-Chat.Source: §4.2 · p. 5
  • VISCPM-Paint trains the UNet cross-attention layers and the LLM-to-UNet linear transblock on English Laion-2B pairs for 300K steps; VISCPM-Paint+ uses the same Chinese pairs as VISCPM-Chat+.The additional Chinese-pair set includes 20M Wukong pairs filtered at a CLIP-score threshold above 0.18, 20M Zero-Chinese pairs, and 136M Laion-COCO pairs translated into Chinese.Source: §4.2 · pp. 5–6

5 EXPERIMENTS

The experiments evaluate VISCPM across multilingual image-to-text and text-to-image tasks, including Chinese, English, and five additional languages. They report strong Chinese performance using English-only multimodal pretraining, alongside multilingual generalization and dataset-language ablations.

5.1.1 EVALUATION SETTING

  • The image-to-text evaluation uses LLaVA Test Set and UniMM-Bench in English and Chinese, with Chinese translations manually checked and GPT-4 rating generated responses.The baselines include English-only and Chinese-English bilingual models; the latter were trained on Chinese and English multimodal datasets.Source: §5.1.1 · p. 6

5.1.2 EXPERIMENTAL RESULTS

  • In Chinese, VISCPM-Chat outperforms all baselines on both benchmarks without Chinese multimodal pretraining; in English, it exceeds several baselines and is comparable to LLaVA.The Chinese evaluation covers LLaVA Test Set and UniMM-Bench; the English comparison is described as roughly on par with Ziya-Visual and Qwen-VL-Chat.Source: §5.1.2 · p. 7
  • VISCPM-Chat also relates a Chinese classical poem to an image’s full moon and pavilion, despite that culturally specific image-text pair being absent from training.The example concerns Su Shi’s “Water Melody.”Source: §5.1.2 · p. 7, Figure 3 · p. 7

5.1.3 GENERALIZATION TO MORE LANGUAGES

  • On LLaVA Test Set, the six-language chatbot averages above 88 in German, French, and Spanish, and above 85 in Italian and Portuguese.The five target languages are blended during instruction tuning.Source: §5.1.3 · p. 7, Table 3 · p. 7

5.2.1 EVALUATION SETTING

  • Text-to-image automatic evaluation uses zero-shot FID and CLIP score on 30K MSCOCO prompts, with Chinese prompts translated from English captions.VISCPM-Paint, AltDiffusion, and TaiyiDiffusion use the same sampling configuration; FID is grid-searched across eight classifier guidance scales.Source: §5.2.2 · p. 8, §5.2.1 · p. 8

5.2.2 AUTOMATIC EVALUATION

  • VISCPM-Paint achieves the best Chinese zero-shot FID on MSCOCO and is comparable to powerful text-to-image models in English.The Chinese comparison is against AltDiffusion and TaiyiDiffusion; the English comparison names Stable Diffusion and UniDiffuser.Source: §5.2.2 · p. 8, Table 4 · p. 8

5.2.3 HUMAN EVALUATION

  • In Chinese Drawbench human evaluation, VISCPM-Paint received the strongest preference across all three aspects, with more than 40% of its generated images favored in each aspect.For Overall and Alignment, VISCPM-Paint earned more than 20 percent 5/5 preference.Source: §5.2.3 · pp. 8–9, Figure 4 · p. 8

5.3 ABLATION STUDY

  • Native-Chinese-only training yields an 85.5 image-to-text average score and 15.1 text-to-image FID; filtered and translated data combinations produce different gains across tasks.Monolingual Chinese instruction tuning reduces image-to-text average performance from 88.0 to 86.3 versus bilingual tuning; unfiltered Chinese pairs worsen FID from 10.9 to 12.8, while mixing translated datasets improves FID from 10.7 to 9.6.Source: §5.3 · p. 9, Table 5 · p. 9

6 CONCLUSION

The conclusion presents MPM as a way to train non-English multimodal models through a multilingual LLM intermediary. It reports open-source Chinese VISCPM models with strong Chinese image-to-text and text-to-image performance using English multimodal data, and a chatbot supporting six languages.

6 CONCLUSION

  • MPM uses a multilingual LLM as an intermediary between vision signals and target languages, and VISCPM achieves state-of-the-art performance among Chinese open-source multimodal models using English multimodal data.The work also constructs a versatile chatbot supporting six languages and releases VISCPM models as open source.Source: §6 · p. 9

B.1 PRETRAINING DATASET

The appendices document the datasets and training setup for VISCPM, alongside case studies, baseline descriptions, and evaluation benchmarks.

B.1 PRETRAINING DATASET

  • The pretraining data include COCO with 118,287 images and five captions each, CC3M with about 2.8M downloaded pairs, and Wukong with 100M Chinese pairs.COCO yields 591,435 image-text pairs by randomly selecting one caption per image on each pass.Source: §B.1 · p. 16

B.2 INSTRUCTION TUNING DATASET

  • Instruction-tuning sources include GPT-4-generated LLaVA-Instruct-150K, UniMM-Chat, and M3IT, which combines vision-language tasks and includes some instances translated into 100 languages.The VISCPM-Chat training mix uses bilingual versions of LLaVA-Instruct-150K and UniMM-Chat.Source: §B.2 · p. 17, §D.2 · p. 19

B.3 CASE STUDY

  • VISCPM-Chat can recognize adapted artwork and a stained New York map, converse in Chinese and English, and read text in images.The case studies include identifying the Mona Lisa in a surreal style and recognizing Starbucks and Avengers: Endgame text.Source: §B.3 · p. 18

C DETAILS OF BASELINES

  • The baselines span multimodal chat and text-to-image models, including VisualGLM, Ziya-Visual, AltDiffusion, and TaiyiDiffusion.VisualGLM uses Chinese and English image-text pretraining data, while TaiyiDiffusion uses 20M filtered Chinese image-text pairs.Source: C DETAILS OF BASELINES · p. 19

D.1 MULTIMODAL PRETRAINING

  • For visual feature alignment pretraining, VISCPM-Chat and VISCPM-Chat+ train on a mix of image-text pair datasets for 180K and 480K steps, respectively, using batch size 768 and learning rate 1e-5.Only parameters of the visual module are optimized in this stage.Source: §D.1 · p. 19

D.2 INSTRUCTION TUNING

  • Instruction tuning optimizes both the visual module and LLM for 80K steps, and VISCPM-Chat uses beam search with beam size 3 at inference.Figure 5 plots response-language accuracy and content quality across translated-Chinese-data percentages.Source: §D.2 · p. 19, Figure 5 · p. 20

D.3 EVALUATION BENCHMARK

  • The LLaVA Test Set has 90 instances rated by GPT-4 on a normalized 1–100 scale, while UniMM-Bench samples 400 instances across four VQA benchmarks.UniMM-Bench evaluates reasoning, commonsense, grounding, and world knowledge.Source: §D.3 · p. 20

D.4 THE TREND OF USING DIFFERENT NUMBER OF CHINESE TRANSLATED DATA

The section examines how translated Chinese data during instruction tuning relate to response language and content quality.

D.4 THE TREND OF USING DIFFERENT NUMBER OF CHINESE TRANSLATED DATA

  • 1% translated Chinese data makes all responses Chinese, while 25% to 50% yields an optimal balance for content quality.The authors leave identifying a more precise proportion that maximizes both response quality and language accuracy for future work.Source: §D.4 · p. 20
  • 0.1% translated Chinese data achieves over 80% response-language accuracy, indicating sensitivity to minimal language-specific input.Source: §D.4 · p. 20
  • The authors varied translated-Chinese-data percentages to study target-language responses beyond the quasi-zero-shot transfer threshold.Source: §D.4 · p. 20

D.5 ADDITIONAL RESULTS ON IGLUE

The additional results evaluate MPM on multilingual image understanding and describe VISCPM-Paint’s training and fidelity-alignment trade-off.

D.5 ADDITIONAL RESULTS ON IGLUE

  • 56.05 on xGQA for CMN is reported for MPM with CPM-Bee, alongside improvement over baseline models and strong multimodal capabilities in several other languages.The authors name Spanish, French, Russian, and German as languages where multilingual LLM proficiency can generalize to multimodal capability.Source: §D.5 · p. 21

E.1 TRAINING AND SAMPLING

  • VISCPM-Paint is pretrained on English Laion-2B data, then VISCPM-Paint+ continues training with native Chinese and translated image-text pairs.The initial 300K steps use 256×256 and 512×512 resolutions; the continued training uses 20M native Chinese and 136M translated pairs.Source: §E.1 · p. 21

E.2 TRADE OFF OF FIDELITY AND ALIGNMENT

  • VISCPM-Paint and VISCPM-Paint+ balance fidelity and alignment, while AltDiffusion has slightly higher CLIP scores but significantly worse FID.The authors note that lower generated-image quality can affect practical use.Source: §E.2 · p. 21, Figure 6 · p. 21

E.3 HUMAN EVALUATION

The section describes Chinese Drawbench and the human-evaluation protocol for comparing text-to-image models.

E.3 HUMAN EVALUATION

  • Chinese Drawbench contains 174 prompts across eight categories, created by seven annotators and checked and polished afterward.Twenty prompts were constrained to nine common classes, while five prompts had no such constraint.Source: §E.3 · p. 22
  • Five evaluators compare shuffled outputs from VISCPM-Paint, TaiyiDiffusion, and AltDiffusion for overall quality, alignment, and fidelity.Alignment measures prompt consistency; fidelity covers clarity, aesthetics, and object realism.Source: §E.3 · p. 22

F DETAIL OF ABLATION STUDY

The ablation study details its image-to-text and text-to-image data configurations, including pretraining datasets and instruction-tuning variants.

F DETAIL OF ABLATION STUDY

  • Image-to-text pretraining compares 20M native Chinese pairs, English data used in VISCPM-Chat, and 390M Chinese pairs translated from English Laion-COCO.The native Chinese pairs are filtered from Wukong and Zero.Source: F DETAIL OF ABLATION STUDY · p. 22
  • Image-to-text instruction tuning uses translated Chinese LLaVA data or a bilingual mix of English and translated Chinese LLaVA.The study limits SFT data to LLaVA and its translated Chinese version to reduce computation time.Source: F DETAIL OF ABLATION STUDY · p. 22
  • Text-to-image pretraining uses Laion-2B for English, with the other setting matching the image-to-text tasks.Source: F DETAIL OF ABLATION STUDY · p. 22

G DETAIL OF CLIP SCORE

This section describes the Chinese CLIP score and filtering criteria, reports caption and image problems found in Wukong, and presents multimodal examples and preference-rate figures.

G DETAIL OF CLIP SCORE

  • Chinese CLIP computes scores for Chinese image-text pairs; the filtering criteria follow Taiyidiffusion’s data-volume and filtering standards, including CLIP score.The authors manually checked Wukong captions and found their overall text quality unsatisfactory.Source: G DETAIL OF CLIP SCORE · p. 22
  • The reported low-score Wukong issues include broken or redirected image URLs, captions unrelated to images, and truncated captions or missing punctuation.Source: G DETAIL OF CLIP SCORE · p. 22
  • Figures 7–9 show Chinese and English VISCPM-Chat conversation examples and VISCPM-Paint outputs, while Figure 10 reports Chinese Drawbench human-preference rates by aspect.Source: Figure 7 · p. 23, Figure 8 · p. 24, Figure 9 · p. 25

Related papers

Loading 2308.12038v3…