Source-linked AI summary
Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description
Desmond Elliott, Stella Frank, Loïc Barrault, Fethi Bougares, Lucia Specia
TL;DR
The shared task evaluates multimodal translation and multilingual image description amid limited multilingual resources and small, expensive training datasets. It compares diverse systems and finds multimodal approaches strong in human evaluation, while text-only systems remain competitive on automatic metrics. The authors call for further study of visual disambiguation and human-centered evaluation.
Problem
Multilingual image description remains less developed than English image description, while available training datasets are small and expensive to construct.
Method
The shared task compares 19 systems from nine groups across multimodal translation and multilingual image description, including image-centered and text-only approaches.
Results
Multimodal systems generally ranked highly in human evaluation, while text-only systems remained competitive on automatic metrics and some multimodal approaches surpassed initialization and double-attention methods.
Takeaways & Limitations
Human judgments are important for evaluating these tasks because text-similarity metrics may fail to reward multimodal systems preferred by humans.
Takeaways & Limitations
The dataset's translation ambiguity remains insufficiently analyzed, motivating further work on datasets where visual information can disambiguate text.
Abstract
from arXiv · showhide
We present the results from the second shared task on multimodal machine translation and multilingual image description. Nine teams submitted 19 systems to two tasks. The multimodal translation task, in which the source sentence is supplemented by an image, was extended with a new language (French) and two new test sets. The multilingual image description task was changed such that at test time, only the image is given. Compared to last year, multimodal systems improved, but text-only systems remain competitive.
1 Introduction
The shared task addresses multilingual image description and multimodal translation, expanding evaluation while testing image use when source text is unavailable. It also promotes unconstrained data settings because existing datasets are small and costly to construct.
- The shared task targets image descriptions in languages other than English, supporting applications such as native-language image search and multilingual ecommerce.
- Task 1 translates a source-language description supplemented by an image, whereas Task 2 generates a target-language description from an image without source-language information at test time.
- The translation task added French, making Multi30K triple aligned with English descriptions translated into German and French.
- The description-generation task changed so source descriptions are no longer observed for test images, making multilingual image description necessarily image-based.
- Two evaluation datasets were introduced: an extended Multi30K set and a teaser set containing source-language ambiguities.
- Participants were encouraged to use unconstrained data because current training datasets are small and expensive to construct.
2 Tasks & Datasets
The shared task combines multimodal translation with image-only multilingual description, using expanded Multi30K resources and an ambiguity-focused Ambiguous COCO evaluation set.
- 2.1 Tasks: Task 1 translates image-associated source descriptions, while Task 2 generates target-language descriptions from unseen images without source descriptions at test time.
- 2.2 Datasets: Multi30K contains 31K images originally described in English, with professionally translated and independently crowdsourced German descriptions.
- 2.2 Datasets: The dataset was extended with new evaluation data, French translation data, and an ambiguity-focused evaluation dataset expected to benefit from visual context.
- 2.2 Datasets: New images were filtered for duplicates, watermarks, and dubious content, yielding 2,071 images for the evaluation resources.
- 2.2 Datasets: The 2,071 images were split into 1,000 images for multimodal translation and 1,071 for multilingual image description, with English and German descriptions collected accordingly.
- 2.2 Datasets: Ambiguous COCO was constructed from VerSe by selecting 461 descriptions covering 56 ambiguous verbs, aiming for 1–3 instances per sense per verb.
3 Participants
Participants submitted diverse multimodal translation and image-description systems, ranging from image-conditioned neural models to retrieval and multitask architectures. Baselines remained text-only for translation and image-attentive for description generation.
- Nine groups submitted systems to the shared task, whose participants and submission identifiers are summarized in Table 3.
- AFRL-OHIOSTATE used the image as the catalyst, generating target-language candidate captions and selecting the nearest candidate to the encoded source caption.
- CMU extended the image-description baseline with multitask learning to generate both German and English captions, although evaluation generated only German captions.
- CUNI combined image-attentive neural translation with additional selected parallel data and back-translated sentences, while its Task 2 system translated generated English captions into German.
- LIUMCVC explored separate source and image attention and multiplicative interaction between visual features and word embeddings.
- NICT used retrieval and multimodal reranking around hierarchical phrase-based and neural translation systems; adding the NMT feature significantly improved the phrase-based system, but the multimodal feature added no further development-set improvement.
- The Task 1 baseline was a text-only Nematus neural machine translation system using byte-pair encoding for rare and out-of-vocabulary words.
4 Text-similarity Metric Results
Text-similarity evaluations covered English→German and English→French multimodal translation, Ambiguous COCO, and multilingual image description. Multimodal translation generally outperformed text-only baselines, while multilingual image description did not surpass the monolingual German baseline on Meteor.
- English→German: Metrics did not fully agree on system rankings for English→German, although four statistically indistinguishable systems were best across metrics.All but one submission outperformed the text-only NMT baseline; the best systems included both multimodal and text-only submissions.
- Ambiguous COCO: 461 Ambiguous COCO descriptions produced lower scores than the 1,000-sentence Multi30K test set, but dataset difficulty cannot be determined definitively.DCU-ADAPT MultiMT C and OREGONSTATE 1NeuralTranslation C ranked relatively higher on Ambiguous COCO, suggesting greater robustness to its ambiguities.
- English→French: 63.1 Meteor for English→French baseline translation exceeded the 41.9 English→German baseline, reflecting higher scores for the French direction.Eight of ten English→French submissions outperformed the baseline, and evaluation metrics agreed more closely on rankings.
- English→French: The hierarchical phrase-based NICT NMTrerank C system outperformed the text-only LIUMCVC NMT C system in English→French, despite being close in English→German.This pattern held on both Multi30K 2017 and Ambiguous COCO test data.
- Multilingual image description: Multilingual image description submissions did not outperform the monolingual German baseline according to Meteor.CMU NeuralEncoderDecoder C marginally exceeded the baseline on TER and equalled its BLEU score; metrics disagreed substantially on rankings.
5 Human Judgement Results
Human evaluation used bilingual direct assessments for English→German and English→French multimodal translation outputs. Automatic and human evaluations broadly agreed, with stronger agreement for German than French, but some system rankings diverged.
- Evaluation procedure: Bilingual assessors judged semantic relatedness between English source sentences and German or French candidate translations, using images when needed.References were withheld to avoid influencing assessors.
- Evaluation procedure: 2,521 French and 3,485 German direct assessments were collected from seven and 25 assessors, respectively.Each submission received between 319–368 French and 291–357 German assessments.
- Results: Human and automatic evaluations globally agreed, with better agreement for English→German than English→French.The comparison used multimodal translation outputs on Multi30K 2017 test data.
- Results: In English→French, CUNI NeuralMonkeyMultimodalMT C and DCU-ADAPT MultiMT C significantly outperformed LIUMCVC MNMT C in human scores despite lower automatic metric scores.This disagreement illustrates that metric rankings and human judgments were not identical.
6 Discussion
The discussion finds mixed evidence for visual features, while external data consistently benefits multimodal translation. Results also highlight domain and vocabulary effects, and leave substantial room for better visual-text integration.
- Visual Features: do they help?: Visual features did not reliably improve metric performance across language pairs and systems, although LIUMCVC improved with multimodal input.CUNI and OREGONSTATE favored text-only systems in German, while CUNI showed small multimodal gains in French; LIUMCVC improved only in French.
- Evaluation and broader conclusions: Multimodal systems ranked highly in human evaluation, but text-only NICT remained competitive across both languages and phrase-based MT retained a role on small homogeneous datasets.Metric rankings did not always agree with human judgments; LIUMCVC text-only NMT ranked highly on metrics but poorly in human evaluation.
- Multimodal architectures: Alternative integration methods achieved strong metric and human-evaluation performance, surpassing more common initialization and double-attention approaches.The discussion identifies LIUMCVC MNMT and UvA-TiCC IMAGINATION as particularly strong alternative approaches.
- Unconstrained systems: 2–3 Meteor points: unconstrained UvA-TiCC and CUNI systems outperformed their constrained variants and achieved higher standardized mean DA scores.The results support using external parallel text and external monolingual image-description data for multimodal translation.
- Domain effects: 49.9 Meteor and 29.0 BLEU: an out-of-domain news system ranked 10th and 11th, below nearly all in-domain text-only submissions.The result indicates that tuning to the target domain remains important even for relatively simple image captions.
- The effect of OOV words: 16% of English test sentences contained an OOV word, and on those sentences UvA-TiCC exceeded the second-best system by 2.5 Meteor and 2.2 BLEU.The comparison was 45.6 versus 43.1 Meteor and 24.0 versus 21.8 BLEU against LIUMCVC NMT.
7 Conclusions
The second shared task highlights continued room for improved multimodal modeling, the value of external resources, and the need for human evaluation alongside text-similarity metrics. It also outlines future task designs to clarify system differences and support multi-source multimodal models.
- 7 Conclusions: Nine groups submitted 19 systems across the two tasks, with most submissions targeting Multimodal Translation.Human judgments were collected for translation for the first time and broadly agreed with automatic-metric rankings.
- 7 Conclusions: Winning systems demonstrate scope for novel approaches to integrating visual and linguistic features in multilingual multimodal models.
- 7 Conclusions: External resources improve multimodal translation beyond what can be learned from limited training data.
- 7 Conclusions: Text-similarity metrics can obscure differences between text-only and multimodal systems because humans may prefer multimodal outputs without metrics rewarding them.The authors recommend human judgments for future evaluation of both multimodal translation and multilingual image description.
- 7 Conclusions: Future editions may request single-decoder outputs and introduce a multilingual multimodal translation challenge with two language inputs plus an image.
A Significance tests
The appendix reports Wilcoxon signed-rank tests used to cluster systems for English→German and English→French results at a significance threshold of p ≤0.05.
- A Significance tests: Wilcoxon signed-rank tests were used to create the clustering of systems.
- A Significance tests: Table 11 reports the English→German tests using p ≤0.05 as the significance threshold.A dash indicates a value higher than 0.05.
- A Significance tests: Table 12 reports the English→French tests using p ≤0.05 as the significance threshold.A dash indicates a value higher than 0.05.