IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages

Benchmark motivation and scope

• IGLUE is the first multilingual multitask V&L benchmark, covering 20 diverse languages, five datasets, and four structurally different tasks.Its tasks include cross-modal retrieval, visual question answering, natural language inference, and visual reasoning.Source: §1 · pp. 1–2

Benchmark construction

• The benchmark aggregates existing resources and adds cross-lingual visual entailment and image–text retrieval data to broaden task coverage.The authors also provide new multilingual training and evaluation data and a multilingual grounded NLI dataset.Source: §1 · pp. 1–2

Transfer settings

• IGLUE evaluates zero-shot transfer, standardised few-shot transfer, and translate-test transfer from English into target languages.It provides translated test sets and standardised splits for few-shot experiments.Source: §1 · pp. 1–2

Evaluation

• Systematic evaluations compare multilingual pretrained architectures with monolingual V&L models combined with machine translation across tasks and languages.The models are evaluated through a unified comparative framework.Source: §1 · p. 2

Main findings

• Transfer performance remains far below English performance, especially for some lower-resource or more distant languages, while few-shot gains often require thousands of examples.NLI has the smallest English-transfer gaps, QA benefits most from few-shot adaptation, and reasoning and retrieval are harder in both respects.Source: §1 · pp. 1–2

Motivation

• Multilingual multimodal datasets have historically concentrated on English or a few high-resource languages, limiting coverage of linguistic diversity and cross-cultural nuances.This scarcity has slowed progress relative to monolingual vision-and-language research.Source: §2 · p. 2

Related multilingual V&L work

• Recent work has expanded multilingual data and pretrained models, but multimodal systems have often been evaluated primarily on cross-modal retrieval.IGLUE merges and extends these research threads for broader comparative evaluation.Source: §2 · p. 3

Cross-lingual benchmark precedent

• Prior multilingual benchmarks show that diverse tasks and languages are important for evaluating cross-lingual generalisation beyond monolingual limitations.IGLUE follows this multi-task, multi-language benchmark tradition in the multimodal setting.Source: §2 · p. 3

IGLUE’s motivation

• IGLUE introduces multimodal cross-lingual evaluation that extends beyond accuracy to fine-tuning efficiency, sample efficiency, and low-data adaptation.Its design also aims to improve multilingual inclusivity and diversity.Source: §2 · p. 3

3. IGLUE Benchmark

• IGLUE follows principles of task diversity, language diversity, and research accessibility while selecting tasks under multilingual-data and computational constraints.The benchmark is designed to support development by practitioners with limited resources.Source: §3 · p. 3

3.1. Tasks and Datasets

• The benchmark covers inference, question answering, reasoning, and retrieval through XVNLI, xGQA, MaRVL, xFlickr&CO, and WIT.These tasks require models to infer, answer questions, compare images, and retrieve images or captions.Source: §3.1 · pp. 3–4
• IGLUE adds culturally grounded and multilingual data, including native-speaker MaRVL descriptions, manually translated xGQA evaluation data, and newly crowdsourced multilingual image descriptions.XVNLI introduces cross-lingual visual entailment with entailment, contradiction, and neutral labels.Source: §3.1 · pp. 3–4
• The benchmark identifies substantial dataset leakage risks in some translated Flickr30K, COCO, and XTD evaluations because images overlap across training and test data.The authors report 882/1,000 COCO test images used as training examples in other languages and 946/1,000 XTD test samples from the COCO training split.Source: §3.1 · p. 4

3.2. Languages

• IGLUE covers 20 languages spanning 11 families, three geographic macro-areas, and scripts including Arabic, Cyrillic, Greek, Hangul, Kanji, Latin, and Tamil.Ten languages have data for at least two tasks, potentially supporting cross-task transfer.Source: §3.2 · pp. 4–5
• The language selection includes different levels of available unlabelled data, enabling evaluation across data-paucity regimes during pretraining.Not every language appears in every task because the source datasets selected languages independently.Source: §3.2 · p. 4

3.3. From Zero-Shot to Few-Shot Setups

• IGLUE standardises zero-shot and few-shot transfer by releasing task- and language-specific few-shot samples and splits.A shot is defined through an annotation context such as an image-caption pair or an image with all its questions.Source: §3.3 · p. 5

Framework

• IGLUE re-implements multilingual V&L pretrained models in the VOLTA framework, alongside five English V&L models and twelve downstream tasks.The implementation is released to facilitate multilingual V&L research and reproducibility.Source: §4 · p. 5

Pretrained Models

• Multilingual encoders use BERT-like Transformer architectures that combine image-region features with textual tokens and train on masked-language, masked-region, and image-text-matching objectives.The evaluated models differ in how multilingual and multimodal data are incorporated during pretraining.Source: §4 · p. 5
• The evaluated multilingual models are mUNITER, xUNITER, M3P, and UC2, which differ in initialization, code-switched training, translated captions, and alignment objectives.mUNITER and xUNITER use multilingual text-only data; M3P adds code-switched multimodal training; UC2 uses translated captions in five languages and additional alignment objectives.Source: §4 · pp. 5–6

Transfer Methods

• The benchmark also tests English encoders—LXMERT, UNITER, ViLBERT, VisualBERT, and VL-BERT—under translate-test transfer.Translations to English are obtained with Google Translate.Source: §4 · p. 6

Experimental Setup

• Experiments fine-tune models with AdamW under controlled hyperparameters and evaluate few-shot learning by adapting English-fine-tuned parameters to target-language task data.Few-shot experiments search three learning rates and train for 20 epochs per dataset-language-shot setting.Source: §4 · p. 6

Evaluation

• Table 4 reports average zero-shot and translate-test performance of multimodal models trained in English and evaluated on target languages in IGLUE, with the best model for each method in bold.Source: Table 4 · p. 6

5.1. Zero-shot Learning

• Translate-test transfer outperforms multilingual zero-shot transfer by 6.5–21.6 points across tasks, nearly doubling grounded question answering and retrieval performance.The largest differences are 21.6 points for xGQA, 21.3 for xFlickr&CO image retrieval, and 19.6 for text retrieval.Source: §5.1 · p. 7
• UC2 is consistently strongest among multilingual encoders, while task performance varies with label-space size, language diversity, and caption distribution.Retrieval on WIT is lower than on xFlickr&CO, plausibly because WIT captions describe entities and differ distributionally from standard captions.Source: §5.1 · p. 7
• Cross-lingual transfer leaves gaps of 10.7–26.5 points relative to English, with the largest deficits in xGQA and cross-modal retrieval.For the best multilingual encoders, the gaps are 14.8 for XVNLI, 26.5 for xGQA, 14.6 for MaRVL, 24.2/23.0 for xFlickr&CO, and 10.7/11.9 for WIT.Source: §5.1 · p. 7
• Wikipedia size partly correlates with target-language performance, whereas typological similarity to English has weaker and sometimes negative correlations.The typological correlation is even negative for MaRVL, indicating that similarity to English is not a consistent predictor.Source: §5.1 · p. 7, Figure 2 · p. 7

5.2. Few-shot Learning

• Few-shot learning produces flat or modest gains on XVNLI, MaRVL, and xFlickr&CO, unlike the large improvements observed in text-only multilingual tasks.Figure 3 summarizes performance against target-language data size, while the maximum-shot results are reported in Table 5.Source: §5.2 · p. 8, Figure 3 · p. 8, Table 5 · p. 8
• xGQA improves dramatically from 27 examples, especially for Choose and Query questions with larger label spaces, whereas Verify questions change little.The findings suggest that target-language fine-tuning can mitigate prediction-space misalignment when many answer labels are possible.Source: §5.2 · p. 8, Figure 4 · p. 8
• Retrieval may require examples in the thousands for significant few-shot gains, leaving sample efficiency an open challenge in resource-poor settings.The extended experiment evaluated German and Japanese xFlickr&CO data beyond the standard k-shot range.Source: §5.2 · p. 8, Figure 5 · p. 9
• Few-shot adaptation can be brittle: M3P sometimes falls to chance-level performance on MaRVL target languages, whereas UC2 remains strongest overall.This instability appears when fine-tuning M3P on Indonesian, Mandarin, and Turkish MaRVL data.Source: §5.2 · p. 9

6. Conclusion and Outlook

• IGLUE combines five datasets and four structurally different tasks across 20 languages, with standardized zero-shot and few-shot evaluation protocols.The benchmark is designed to support multilingual vision-and-language modelling and evaluation.Source: §6 · p. 9
• Translate-test transfer outperforms multilingual zero-shot transfer, few-shot learning often needs thousands of examples, and typological distance weakly predicts performance.Transfer gaps are largest for multicultural visual reasoning and cross-modal retrieval, and IGLUE deliberately avoids a single average score.Source: §6 · p. 9
• Future work can extend IGLUE with single- versus multi-source transfer, multitask and staged transfer, additional languages and tasks, video, and speech.These directions go beyond the scope of the present image-focused benchmark.Source: §6 · p. 9

A. Datasets

• IGLUE datasets cover visual entailment, question answering, visual reasoning, and image–text retrieval with multilingual data and few-shot splits.The section documents dataset construction, annotation contexts, statistics, and reproducibility-oriented splits.Source: A. Datasets · p. 16

A.1. XVNLI

• XVNLI uses images as annotation contexts, with up to 48 sampled images yielding up to 168 hypotheses and answers for few-shot evaluation.Its label distribution is uniform across splits, while test sentence lengths are broadly similar across languages and often shorter for Arabic.Source: §A.1 · p. 16

A.2. xGQA

• xGQA reuses target-language development data and few-shot splits containing translations of up to 1,490 questions per language.The task evaluates structured questions about images originally sampled from Visual Genome.Source: §A.2 · p. 16

A.3. MaRVL

• MaRVL collects native-speaker captions from eight-image annotation contexts, expands them to 80 data points per language, and provides few-shot data for only Indonesian, Mandarin, and Turkish.The collection avoids overlap with test images and limits few-shot coverage because annotation is difficult in Swahili and Tamil.Source: §A.3 · p. 16

A.4. xFlickr&CO

• xFlickr&CO combines Flickr30K and COCO images with multilingual captions collected under length-controlled annotation guidelines and supplemented by independently sourced captions.Few-shot descriptions are annotated for 100 Flickr30K training images, while German, Mandarin, and Turkish captions come from existing multilingual resources.Source: §A.4 · p. 17

A.5. WIT

• WIT derives 500,000 English image captions for training and multilingual test splits from Wikipedia imagery, with image overlap and caption-length statistics reported across languages.Few-shot splits are provided only for xFlickr&CO because evaluating retrieval systems requires demanding computational resources.Source: §A.5 · p. 17, Table 7 · p. 18

A.6. Ethics Statement

• MaRVL and xFlickr&CO use native speakers from under-represented languages, recruited through Prolific and paid £10–20 per hour.The annotation effort is motivated by representing greater linguistic and cultural variety in vision-and-language datasets.Source: §A.6 · p. 18

B. Experimental Details

• All encoders are trained end-to-end with dataset-specific heads, common optimization conventions, and visual regions extracted by pretrained ResNet-101 or ResNeXt-101 backbones.Training uses AdamW, warmup followed by decay, weight decay, gradient clipping, and task-dependent sequence lengths.Source: B. Experimental Details · p. 18
• Zero-shot tasks use English fine-tuning and task-specific formulations for entailment, question answering, visual reasoning, and retrieval.The setups use binary or multiclass objectives, pooled Transformer representations, hard negatives for retrieval, and different epoch counts by dataset size.Source: B. Experimental Details · pp. 18–19
• Few-shot models start from English fine-tuned parameters, search three learning rates, train for 20 epochs, and select checkpoints by validation performance.The reported validation sets remain in their original form because translated validation sets showed no significant checkpoint-selection difference.Source: B. Experimental Details · p. 19

C. Reproducibility and Additional Results

• The paper supplements the main experiments with per-language and few-shot results and documents the setup to support reproducibility.Additional experimental details are provided in the reproducibility appendix.Source: C. Reproducibility and Additional… · p. 19

C.1. A Unified Implementation Framework

• M3P and UC2 are reimplemented in VOLTA, which already supports mUNITER, xUNITER, and five English pretrained encoders.The unified framework is intended to make IGLUE experiments lighter and more reproducible.Source: §C.1 · p. 19
• The reimplemented models reproduce Multi30K performance within a couple of points of the original reports.The validation uses zero-shot evaluation and supports the correctness of the reimplementation.Source: §C.1 · p. 19, Table 10 · p. 19
• For UC2, replacing the triplet loss with cross-entropy reduces mean Recall by approximately 2.5 points in each language.The appendix also reports that using triplet loss for M3P reduces performance by nearly 3 points.Source: §C.1 · p. 19
• UC2 gains over 10 accuracy points over Pfeiffer et al.’s best model on zero-shot xGQA, while M3P is 13.4% more accurate than their model.The authors attribute the gains mainly to using a nonlinear two-layer prediction head instead of a linear classification head.Source: Table 11 · p. 19, §C.1 · p. 20

C.2. Performance per Target Language

• WIT retrieval performance is relatively constant across languages, including UC2’s Japanese split despite its Japanese multimodal pretraining.The authors relate this pattern to differing caption and image distributions in the Wikipedia domain.Source: §C.2 · p. 20
• XVNLI and xGQA performance peaks in French and German, respectively, while multilingual data availability is also indicative of final performance.Models can perform better in Russian than Spanish or Portuguese, and UC2 is strongest in languages represented among its Conceptual Captions translations.Source: §C.2 · pp. 20–21, Table 14 · p. 20, Figure 11 · p. 21
• MaRVL performance is comparably low across languages and models because its out-of-distribution images limit translate-train effectiveness.Mandarin still benefits from large-scale target-language multimodal data, although UC2’s gains are smaller than on other datasets.Source: Table 15 · p. 20, §C.2 · p. 21
• UC2 largely outperforms other systems in German, Japanese, and Mandarin on xFlickr&CO, but is otherwise often comparable.The per-language results are reported separately for image retrieval and text retrieval.Source: §C.2 · p. 21, Table 16 · p. 21, Table 17 · p. 21
• The best-to-worst language gaps reach 13.5 points for XVNLI, 22.9 points for xGQA, and 22.5 points for xFlickr&CO.These differences occur despite language-independent visual inputs in the evaluation sets.Source: §C.2 · p. 21

• Few-shot AUC summarizes performance over increasing numbers of shots using a normalized area under the learning curve.The score is approximated with the trapezoidal rule for each model and dataset.Source: §C.3 · pp. 21–22, Table 18 · p. 22
• UC2 is by far the strongest model across all tasks, followed by xUNITER, M3P, and mUNITER.xUNITER outperforms M3P on reasoning and retrieval, while mUNITER is generally weakest except on reasoning.Source: §C.3 · p. 22, Table 18 · p. 22
• Few-shot learning produces no clear improvement as shot counts increase for most tasks, with xGQA the only exception.The reported pattern applies to XVNLI, MaRVL, and xFlickr&CO, where roughly 100 labelled examples are insufficient for meaningful progress.Source: §C.3 · p. 22, D. Further Analyses · p. 22, Table 19 · p. 22
• The findings stress the need for improved zero-shot transfer because current V&L models require substantial task-annotated data to benefit from non-zero-shot setups.Cost-efficiently hand-labelling about 100 target-language examples is insufficient for several multilingual multimodal tasks.Source: §C.3 · p. 22

Further Analyses

• The additional analyses cover translate-train transfer, xFlickr&CO domain-specific retrieval, and structurally different xGQA question types.These analyses extend the benchmark evaluation beyond aggregate few-shot results.Source: D. Further Analyses · p. 22

Translate Train Performance

• UC2 matches translate-train performance with 100 or fewer examples and surpasses it with 1,000 or more, whereas Japanese requires 500 examples to match it.xUNITER benefits from at least 500 examples but remains far below its translate-train counterpart; translate-train also remains below English performance.Source: D. Further Analyses · p. 23

Retrieval Performance by Evaluation Subset

• Multilingual encoders perform similarly on the Flickr and COCO xFlickr&CO subsets, unlike English models, which show a domain shift.For UC2, this subset behavior is language-independent, but the available performance is too poor to determine whether future tightly multilingual models will face the same challenge.Source: D. Further Analyses · p. 23, Figure 13 · p. 23, Table 21 · p. 23