Source-linked AI summary
Distilling Translations with Visual Awareness
Julia Ive, Pranava Madhyastha, Lucia Specia
TL;DR
Multimodal translation systems often ignore images because visual information is needed only for specific ambiguities or source errors. The paper jointly trains a text-based draft decoder and a refinement decoder that uses target-side context and visual information, achieving state-of-the-art results and recovering from erroneous or missing source words.
Problem
Visual information is often needed only for specific ambiguities or source-language errors, so multimodal translation models tend to ignore it.
Method
A jointly trained translate-and-refine model generates a draft translation, then refines it with left- and right-side target context and structured visual representations.
Results
The approach achieves state-of-the-art results and improves translation robustness to noisy input, including erroneous or missing source words.
Takeaways & Limitations
Using visual information during a second refinement stage is a promising direction for multimodal translation that is robust to noisy input.
Takeaways & Limitations
Detected objects can be inaccurate or too imprecise to help, and may harm otherwise correct translations.
Abstract
from arXiv · showhide
Previous work on multimodal machine translation has shown that visual information is only needed in very specific cases, for example in the presence of ambiguous words where the textual context is not sufficient. As a consequence, models tend to learn to ignore this information. We propose a translate-and-refine approach to this problem where images are only used by a second stage decoder. This approach is trained jointly to generate a good first draft translation and to improve over this draft by (i) making better use of the target language textual context (both left and right-side contexts) and (ii) making use of visual context. This approach leads to the state of the art results. Additionally, we show that it has the ability to recover from erroneous or missing words in the source language.
1 Introduction
Multimodal machine translation uses images to resolve ambiguities, gender marking, and inaccurate source descriptions that textual context alone may not resolve. The paper proposes jointly trained draft-and-refine decoding, using target-side context and visual information only when refinement is needed.
- Human translators revised 62-77% of translations for incorrect or ambiguous source words and gender-neutral words requiring target-language gender marking.
- Multi30K test data contain at least one multiply translated word in 358 German and 438 French instances out of 1,000, although textual context or frequency bias often resolves the ambiguity.
- Images can correct lexical ambiguity, gender marking, and inaccurate source descriptions, including football-versus-soccer, player gender, and shirt-versus-sweater errors.
- The proposed deliberation approach jointly generates a draft and refines it using left- and right-side target context plus structured visual information, refining only when needed.
- The approach reports state-of-the-art results and includes a frequency-bias-free experiment showing multimodal refinement outperforms text-only refinement in more complex linguistic setups.
2 Related work
Prior multimodal translation methods integrate global or spatial image representations into text models, while translation refinement uses draft translations and second-stage decoding. This work differs by reserving visual input for refinement and adding object-level representations.
- Earlier MMT systems integrated global image vectors by initializing encoders or decoders, combining image and source annotations, or projecting image and encoder representations jointly.
- Spatial attention methods use convolutional feature maps over image regions and combine visual and textual attention through learned mechanisms or gating.
- The closest prior work uses doubly attentive transformers and spatial visual information, whereas this approach adds a second decoding pass and object-level visual information.
- Input-degradation studies found multimodal systems exploit visual cues under limited textual context and can handle typographical source errors; this paper extends that investigation.
- Translation refinement spans iterative substitutions, non-autoregressive decoding, and separate automatic post-editing models, with some approaches requiring draft-reference training data.
- Deliberation networks jointly train first- and second-stage decoders, giving the second decoder left and right target context for refinement.
3 Model
The model uses a Transformer with a deliberation-based second decoder that refines first-pass translations using source, target-side, and visual context. Images enter only during refinement through spatial or object-based representations.
- 3 Model: The implementation is based on the Transformer architecture, with six-layer encoder and decoder blocks and a three-layer additional deliberation decoder.The deliberation network trains 345M parameters compared with 210M for the Transformer.
- 3.1 Deliberation networks: The deliberation architecture adds a second decoder conditioned on the source and sampled first-pass outputs, enabling translation refinement.The first-pass decoder supplies candidate translations and hidden representations, while the second pass attends to them alongside encoder context.
- 3.2 Multimodal transformer & deliberation: The model supports additive image conditioning and attention over image features as alternative multimodal integration mechanisms.Additive conditioning projects an image vector into encoder outputs, whereas attention adds a visual cross-attention sublayer.
- 3.2 Multimodal transformer & deliberation: Visual cross-attention is added only to the second-pass decoder because images are expected to help in a limited number of cases.This design preserves a text-only first draft while exposing visual information during refinement.
- 3.3 Image features: Spatial features preserve CNN activation locations, while object-based features represent detected objects either as a sparse 545-dimensional vector or as embeddings.The object-embedding variant uses 50-dimensional GLoVe vectors for object categories to make object similarity easier to exploit.
4 Experimental settings
Experiments evaluate multiple Transformer and deliberation variants on Multi30K French and German translation, using standard and degraded-source settings. The study compares text-only and multimodal models across ambiguity, missing-person-word, and random-content-word conditions.
- 4 Experimental settings: Multi30K provides 29,000 training, 1,014 development, and 1,000 test instances, with experiments conducted on German and French.Each instance pairs an English description with human translations and an image.
- 4.2 Degradation of source: Source degradation masks random content words, ambiguous words, or person words to test the contribution of textual and multimodal context.The procedure is applied to training, validation, and test sets, with models compared under text-only, additional-text, and multimodal conditions.
- 4.2 Degradation of source: RND and PERS use the same degraded data across language pairs, whereas AMB differs by target language because replaced words depend on the target language.Table 1 reports the resulting dataset statistics for the three degradation strategies.
- 4.4 Training: The experiments compare eight Transformer, image-conditioned, and deliberation variants on French and German Multi30K test sets using METEOR and BLEU.The set includes baseline, additive and attention-based image models, standard deliberation, and deliberation models with image information.
- 4.4 Training: Deliberation models are trained with cross-entropy loss after initializing the encoder and first-pass decoder from a converged Transformer.First-pass samples are obtained with beam search of size 10, and Adam is used for optimization.
5 Results
Deliberation models improve translation quality over the transformer baseline, while visual information provides targeted benefits, especially for adequacy and difficult degraded-source cases.
- 5.1 Standard setup: Average ΔMETEOR = 1 and ΔBLEU = 1 over the transformer baseline, while multimodal models achieve state-of-the-art performance for constrained English-German transformer networks.The improvement applies across test sets, whereas image-enriched first-pass models show no major gains over the base model.
- 5.1 Standard setup: Human rankings favor deliberation setups, with French annotators showing a more positive preference for adding visual information through del+obj over del.These human judgments are consistent with automatic results for deliberation models but reveal stronger perceived benefits from visual input in French.
- 5.1 Standard setup: Deliberation improves grammaticality and adequacy by making contextual substitutions, including German verb changes and French noun replacements.German test-2016 edits include 15% adjective and 12% verb substitutions; French noun substitutions account for 15% of changes.
- 5.1 Standard setup: Object information resolves specific adequacy errors, changing Rennen to Auto and then Rennwagen for race car, and adding paddling in a French example.The examples show object features supplying lexical distinctions or content missing from the baseline translation.
- 5.2 Source degradation setup: Under source degradation, deliberation models generally outperform base models, while French gains over base average ΔMETEOR = 1 and ΔBLEU = 1.1.For French, adding image information to del significantly improves scores only for test-2018 RND; German benefits more from the complex del+obj architecture in RND and AMB settings.
- 5.2 Source degradation setup: Image information increases correctly resolved PERS blanks by 10% for del+obj versus del, while RND and AMB blanks remain harder and are often guessed from text.At most 40% of RND and AMB blanks are resolved, compared with 61% for PERS; visual cues help most when textual context is generic.
6 Conclusions
The approach combines deliberation over textual context with visual information in a second-stage refinement process. It improves translation quality and can help resolve blanks using textual or multimodal context, while visual information can also mislead.
- 6 Conclusions: Detected objects can also introduce errors: a model translates field as Reishut instead of Reisfeld despite the text-only system choosing the correct translation.The example illustrates that visual context is not uniformly beneficial when object detections conflict with textual evidence.
- 6 Conclusions: A textual-context example correctly translates biker as the masculine Geländemotorradfahrer, while a multimodal example resolves boy as l’enfant using detected objects.The multimodal example uses clothing, face, tree, boy, and jeans as detected objects; the textual example uses person, tree, bike, and helmet.
- 6 Conclusions: The approach uses a second decoding stage to exploit both textual and visual context, yielding better results than the previous state of the art.The authors specifically emphasize improved textual context through deliberation and visual information, particularly structural representations, for noisy input and substantial source-target restructuring.
- 6 Conclusions: The conclusions characterize multimodal deliberation as promising for machine translation that is robust to noisy input.This claim is stated for the combination of deliberation and additional modalities, with the visual component used during refinement.
A Appendices
The appendices provide qualitative examples of translations resolved by textual context, multimodal context, or deliberation with object information. They also include unresolved cases and examples where object cues help or harm translation.
- A Appendices: Figure 5 presents blanks correctly resolved by textual context, including gender and ambiguity examples handled by the base system.Examples include bride and groom translated as Frau and Mann, and runs translated as läuft.
- A Appendices: Figure 6 presents blanks correctly resolved by multimodal context, including a bench translated as Bank because the detected object identifies it.The appendix also shows a boy-related example where detected objects support the translation into l’enfant.
- A Appendices: Figure 7 collects unresolved blanks, including cases where listen and speaker receive inconsistent translations despite available object detections.The example lists alternatives such as schaut, rechts, and Wasser for the affected blanks.