Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset

• Multilingual evaluation datasets are costly to create, and translated references can weaken automatic metrics’ agreement with human evaluations.Extensive human evaluations are costly to replicate across research efforts and do not provide a fast mechanism for comparing research directions.Source: §1 · p. 1
• XM3600 contributes a cross-language annotation protocol, geographically diverse image sampling, and an empirical evaluation of its ability to rank model variations.Source: §1 · p. 1
• Figure 1 presents sample captions in three of 36 languages, showing annotations consistent in style across languages and free of directtranslation artefacts; the full caption list appears in Appendix A.Source: Figure 1 · p. 2
• XM3600 references show high agreement with human judgments when ranking image-captioning model variations, potentially alleviating the need for future human evaluations.Source: §1 · p. 2

2 The XM3600 Dataset

• XM3600’s construction covers language and image selection, caption annotation, caption statistics, quality checks, and annotator details.Source: §2 · p. 2

2.1 Language Selection

• The authors select 30 languages roughly by web-content share, add five to include low-resource languages or major languages from otherwise uncovered continents, and include English for 36 total.Source: §2.1 · p. 2

2.2 Image Selection

• For each of 36 languages, the authors target 100 geotagged Open Images images from areas where the language is spoken, prioritizing languages with the smallest candidate pools.When local images are insufficient, selection backs off from language-speaking areas to country, continent, and ultimately anywhere; Persian uses 14 continent-level images, while all 100 Hindi images are global.Source: §2.2 · pp. 2–3

2.3 Caption Annotation

• Bilingual annotators are trained to produce visible, style-consistent captions without translation artifacts, using English model captions to establish style before writing target-language captions without seeing the English text.The protocol first asks annotators to assess English captions and then presents the images again without those captions; the batch size is intended to overwrite memory of their exact wording.Source: §2.3 · pp. 3–4

2.4 Caption Statistics

• The dataset contains 261,375 captions across 36 languages, with replication and average word and character counts reported by language.Caption lengths vary across languages, including 5–6 words for some agglutinative languages and up to 18 for Vietnamese; character counts also vary with language and script.Source: Table 1 · p. 4, §2.4 · p. 4

2.5 Caption Quality

• Caption quality is refined through pilot runs and repeated guideline updates, then checked by ratings of captions for 600 randomly selected images per language.Table 2 reports quality statistics for the 36 languages using the median of three ratings per image-caption pair.Source: §2.5 · p. 5, Table 2 · p. 5

2.6 Annotator Details

• Annotators are paid professionals, native in the target language whenever possible, and required to be proficient in English.For low-resource languages, annotators are usually linguists with advanced-level knowledge of the language.Source: §2.6 · p. 5

3 Model Comparison using XM3600

• CIDEr with XM3600 references shows high correlation with human-based model rankings, supporting XM3600 as an evaluation benchmark.The experiments compare model rankings based on human evaluations and CIDEr scores using XM3600 annotations as gold references.Source: §3 · p. 5

3.1 Datasets

• CC3M-35L and COCO-35L are created by translating the source captions into 34 other languages; the API does not support Cusco Quechua.The datasets use the standard CC3M splits and the COCO Karpathy split.Source: §3.1 · p. 5
• “Gold” denotes human-level-quality references, while automatically translated references are termed “silver.”Source: §3.1 · p. 5

XM3600 COCO-DEV

• Table 4 reports CIDEr scores for the models over four LCORE languages on XM3600 and COCO-DEV, whose references are machine-translated.Source: Table 4 · p. 6

3.2 Models

• The model combines pooled ViT image-patch features and a language identifier with Transformer encoders and decoders, using pretrained ViT and mT5 components.The tested combinations vary the mT5 and ViT sizes and include models trained on COCO-35L or CC3M-35L.Source: §3.2 · p. 6

3.3 Human Evaluation

• Human comparisons use three raters per pair on 600 images, with full pairwise evaluations in four core languages and targeted sessions across 28 extended languages.A majority of at least two raters determines wins and losses; the full evaluation would otherwise require 210 sessions.Source: §3.3 · p. 7

3.4 Results

• CIDEr differences on XM3600 are highly correlated with human judgments across OALL, OCORE, and OEXT, while translated COCO-DEV CIDEr is strongly anti-correlated on OEXT.Source: Table 6 · p. 8, §3.4 · p. 8
• Correlations using translated XM3600 references from several languages were similar to, and sometimes slightly higher than, those using human references, leaving their relative value unresolved.The authors call for further work to understand translated references compared with human-generated references.Source: §3.4 · p. 8

4 Conclusions

• The benchmark covers all inhabited continents and a large fraction of the world population, with potential relevance to accessibility for speakers of low-resource languages.It is designed to evaluate multilingual image-captioning models.Source: §4 · p. 9
• XM3600 references for automatic metrics show significant improvements in correlation with human judgments, alleviating the need for extensive multilingual human evaluation.The authors note that human evaluation is difficult across multiple languages and hinders direct comparison between research results.Source: §4 · p. 9
• Empirical observations primarily cover English, Spanish, Hindi, and Chinese, with additional observations across 28 languages validating the expected broader pattern.The authors expect similar results for other languages based on similar data collection and quality control.Source: §4 · p. 9

5 Limitations

• The 36-language annotation effort omits many languages, and selection based on internet presence over-represents European languages despite inclusion of five low-resource languages.The authors warn that sharing the dataset could perpetuate Eurocentrism in computational linguistics and AI.Source: §5 · p. 9
• Sampling 100 images per target language limits the natural and cultural phenomena represented, and uneven access to cameras and internet may bias image coverage.The authors note that the images may still omit important aspects of natural and cultural life worldwide.Source: §5 · p. 9
• Confidence that annotations lack translation artifacts rests on the caption-generation process, rater controls, spot checks, and comparisons with translated annotations.The authors describe this as a limitation concerning the absence of translation artifacts.Source: §5 · p. 9
• Automated evaluation may miss small differences between highly tuned methods, human judgment may still be needed, and metric-optimizing methods may be favored.The authors report good results across a range of model differences but caution that these limits remain.Source: §5 · p. 9
• The authors could not release model outputs and human judgments, which may hamper reproducibility of the reported computations.The data would also support development and validation of automated metrics and independent calculation of CIDEr and ΔS×S.Source: §5 · pp. 9–10
• XM3600 uses layered privacy safeguards, but the authors characterize such measures as mitigating rather than eliminating risks related to personal information.Safeguards include in-house annotation and quality control, image vetting, automated scanning of machine translations, and spot checks.Source: §5 · p. 10
• Despite these limitations, the authors describe XM3600 as a significant step toward ameliorating language and geographic bias.They recommend its use to advance image-captioning research over a wider variety of images and languages.Source: §5 · p. 10

A Additional Caption Examples

• Figure 4 shows example captions in XM3600’s 36 languages for the same image as Figure 1.The examples illustrate the dataset’s multilingual coverage.Source: A Additional Caption Examples · p. 11, Figure 4 · p. 12

B Instructions for Rating Captions

• Raters judge how well a caption describes the image, using four quality levels and a separate option for insufficient information.The guidance flags missed topics, major grammatical errors, and definite hallucinations as reasons for a BAD rating.Source: B Instructions for Rating Captions · p. 11

C Instructions for Generating Captions

• Caption writers are asked to describe the whole image with specific objects, salient relations, activities, attributes, and context.The procedure illustrates combining these elements into a caption and asks writers to produce captions independently rather than copy earlier ones.Source: C Instructions for Generating Captions · pp. 11, 13

D Detailed Results for Model Comparison

• Table 7 reports m2-minus-m1 CIDEr differences on XM600, XM3600, and COCO-DEV for LEXT languages.COCO-DEV uses machine-translated references.Source: Table 7 · p. 13

XM3600 COCO-DEV

• Table 8 reports CIDEr for the best-performing model BB+CC on XM3600 and COCO-DEV across all 35 languages.COCO-DEV uses machine-translated references.Source: Table 8 · p. 14

XM3600 COCO-DEV

• Table 9 reports CIDEr for model Bg on XM3600 and COCO-DEV across all 35 languages.COCO-DEV uses machine-translated references.Source: Table 9 · p. 14

XM3600 COCO-DEV

• Table 10 reports CIDEr for model Lg on XM3600 and COCO-DEV across all 35 languages; COCO-DEV uses machine-translated references.Source: Table 10 · p. 15
• Table 11 reports CIDEr for model BB on XM3600 and COCO-DEV across all 35 languages; COCO-DEV uses machine-translated references.Source: Table 11 · p. 15