Source-linked AI summary
Multi30K: Multilingual English-German Image Descriptions
Desmond Elliott, Stella Frank, Khalil Sima'an, Lucia Specia
TL;DR
Image description research has focused mainly on English, motivating a multilingual multimodal dataset. Multi30K extends Flickr30K with professional German translations and independently crowdsourced German descriptions, supporting a broader range of tasks and revealing distinct corpus characteristics.
Problem
Image description has received attention mainly in English, although applications such as image search and alternative text are useful across languages.
Method
Multi30K extends Flickr30K with 31,014 professional German translations and 155,070 independently crowdsourced German descriptions paired with the same images.
Results
The translated and independent corpora differ substantially in sentence length and vocabulary, reflecting their aligned versus independently collected sentence relationships.
Takeaways & Limitations
Multi30K supports multilingual image description, image-sentence ranking, multimodal semantics, machine translation, and other multilingual multimodal tasks.
Abstract
from arXiv · showhide
We introduce the Multi30K dataset to stimulate multilingual multimodal research. Recent advances in image description have been demonstrated on English-language datasets almost exclusively, but image description should not be limited to English. This dataset extends the Flickr30K dataset with i) German translations created by professional translators over a subset of the English descriptions, and ii) descriptions crowdsourced independently of the original English descriptions. We outline how the data can be used for multilingual image description and multimodal machine translation, but we anticipate the data will be useful for a broader range of tasks.
1 Introduction
Multi30K addresses the English-only focus of image description by introducing English-German image-sentence data with both aligned translations and independently collected descriptions. The dataset supports multilingual image description, multimodal machine translation, and broader multilingual multimodal research.
- Image description research has focused largely on English, despite applications such as natural-language image search and alternative text for visually impaired users.
- Multi30K extends Flickr30K with English-German image-sentence data for studying multilingual multimodal research.
- The dataset contains 31,014 professional German translations and 155,070 independently collected German descriptions.
- Translated sentences have strong cross-language correspondence, whereas independent descriptions are only linked by their shared image.
- Multi30K broadens multilingual multimodal research toward image description, machine translation, and image-sentence ranking.
- Independent descriptions can describe the same image with different details across languages, while translation may misinterpret image-dependent wording when context is unavailable.
2 The Multi30K Dataset
The dataset combines professionally translated German descriptions with independently crowdsourced German descriptions, using distinct collection procedures. Corpus analyses show systematic differences in length and vocabulary between the two data types and between English and German.
- Dataset composition: Flickr30K provides 31,014 images with five English descriptions each, and Multi30K adds translated and independent German sentences.
- Translated descriptions: Professional translators produced one German translation per image, with source descriptions sampled evenly across short, medium, and long lengths.
- Independent descriptions: Crowdworkers produced five independent German descriptions per image using translated collection instructions modeled on the original Flickr30K task.
- Collection quality: Quality control combined manual inspection with automated checks for short sentences, repetition, and high character-level 6-gram language-model perplexity.
- Corpus comparison: German translations average 11.1 words versus 9.6 for independent German descriptions, while selected English sources average 11.9 versus 12.3 words for Flickr30K.
- Corpus comparison: German uses 6.6% fewer words but 17.1% more characters than English in the translation corpora, alongside richer German vocabulary and more singletons.
3 Discussion
Multi30K is designed as a reusable resource for multilingual multimodal research beyond image description. Its images paired with English and German text support ranking, multilingual semantics, and machine translation with visual context.
- Multi30K is immediately suitable for automatic image description, image-sentence ranking, multimodal and multilingual semantics, and machine translation.
- Prior multilingual image-description work used multimodal neural models to transfer features between monolingual English and German systems.
- Multi30K enables machine translation research that observes images or video alongside text rather than relying only on textual corpora.
- Existing translation models can incorporate multimodal information through reranking, log-linear features, or neural feature vectors.
4 Conclusions
Multi30K combines German professional translations and independently crowdsourced descriptions with Flickr30K, enabling study of multilingual multimodal data across distinct corpus contexts. The dataset’s structural differences create different research challenges, while motivating broader multilingual coverage.
- Multi30K extends Flickr30K with German descriptions and professional translations for multilingual multimodal research.
- The descriptions were crowdsourced, whereas the translations came from professionally contracted translators, deliberately representing different multilingual data contexts.
- The translated and independently collected corpora differ substantially: translations have approximately matched token counts and sentence lengths across languages, while descriptions vary in sentence lengths and vocabulary.
- Independent collection rather than sentence-level alignment is likely to create different engineering and scientific challenges.
- The authors aim to study multilingual multimodality across a wider range of languages, including languages beyond Indo-European families.