Source-linked AI summary
Incorporating Global Visual Features into Attention-Based Neural Machine Translation
Iacer Calixto, Qun Liu, Nick Campbell
TL;DR
The paper addresses how visual information can improve attention-based NMT and whether different image-integration strategies and additional synthetic data help. It introduces multimodal models that place global image features in encoder or decoder initialization, finding improved performance over NMT, PBSMT, and a strong multimodal baseline, while repeated decoder conditioning overfits.
Problem
The paper investigates how visual features can be incorporated effectively into attention-based NMT and how additional multimodal, multilingual data affect performance.
Method
The authors use global image features in the source sentence, encoder initialization, or decoder initialization, and evaluate these models with synthetic back-translated training data.
Results
The best purely neural models significantly improve over comparable PBSMT and strong multimodal NMT baselines according to all evaluated metrics.
Takeaways & Limitations
Global image features are useful for visually grounding translations, but adding them directly to the decoder at every time step causes overfitting and prevents learning.
Takeaways & Limitations
Multimodal NMT is constrained by the small amount of publicly available training data, approximately 30k examples, limiting real-world applicability.
Abstract
from arXiv · showhide
We introduce multi-modal, attention-based neural machine translation (NMT) models which incorporate visual features into different parts of both the encoder and the decoder. We utilise global image features extracted using a pre-trained convolutional neural network and incorporate them (i) as words in the source sentence, (ii) to initialise the encoder hidden state, and (iii) as additional data to initialise the decoder hidden state. In our experiments, we evaluate how these different strategies to incorporate global image features compare and which ones perform best. We also study the impact that adding synthetic multi-modal, multilingual data brings and find that the additional data have a positive impact on multi-modal models. We report new state-of-the-art results and our best models also significantly improve on a comparable phrase-based Statistical MT (PBSMT) model trained on the Multi30k data set according to all metrics evaluated. To the best of our knowledge, it is the first time a purely neural model significantly improves over a PBSMT model on all metrics evaluated on this data set.
1 Introduction
The paper proposes attention-based multimodal NMT models that incorporate visual features into the encoder and decoder, and evaluates these strategies alongside synthetic multimodal data. Its models improve over strong NMT and PBSMT baselines across evaluated metrics, while decoder updates with image features at every time step overfit.
- The paper proposes attention-based multimodal NMT models that incorporate visual features into the encoder and decoder.
- Synthetic back-translated data are studied as additional multimodal and multilingual training data.
- Adding image features directly to the decoder at every time step causes overfitting and prevents learning.
- The models incorporate images as source-side words or separately for encoder or decoder initialization, unlike prior work using images only as words.
- The models outperform Huang et al. (2016)’s strong multimodal NMT baseline according to all evaluated metrics.
2 Attention-based NMT
The paper extends attention-based NMT with global visual features by integrating images into encoder inputs or hidden-state initialization, while retaining attention-based decoding. It evaluates several integration strategies, including a decoder-context variant that overfits and prevents learning.
- 2.1 Text-only attention-based NMT: Attention-based NMT encodes source words with a bidirectional GRU and uses attention-derived context vectors to compute each decoder hidden state.The decoder conditions on previously emitted words, its previous hidden state, and the time-dependent context vector.
- 2.2 Multi-modal NMT (MNMT): Global image features are extracted from the 4096-dimensional VGG19 FC7 representation for use in multimodal NMT.The features come from a pre-trained 19-layer VGG network and encode information about the entire image.
- 2.2 Multi-modal NMT (MNMT): The models incorporate images as source words, as encoder initialisation inputs, or as additional inputs for decoder initialisation.These strategies extend the attention-based framework with a visual component.
- 2.2 Multi-modal NMT (MNMT): Adding image features directly to the decoder’s recurrent context causes overfitting that ultimately prevents learning.This finding corroborates earlier results for using image features as an additional decoder context at every time step.
- 2.2.1 Images as source words: IMGW: Projected image features can be placed at the beginning or end of the source sequence, allowing attention to select the visual representation during decoding.IMG1W uses one projected image word, whereas IMG2W uses projected image words at both ends; the resulting sequence has N + 1 or N + 2 tokens.
- 2.2.2 Images for encoder initialisation: IMGE: Image features can initialise both directional encoder states, replacing zero initialisation with learned feed-forward projections into the forward and backward hidden-state spaces.The two projection networks separately compute the initial states of the encoder’s forward and backward RNNs.
3 Data set
The study uses translated and comparable multilingual expansions of Flickr30k, with image-aligned bilingual descriptions, standardized subword preprocessing, and back-translated synthetic training data.
- Data sources: Multi30k provides two multilingual Flickr30k expansions: M30kT contains professionally translated English–German sentence pairs, while M30kC contains independently collected German descriptions.M30kT uses one translated description per image; M30kC retains five German descriptions per image.
- Data splits: 29k, 1014, and 1k images form the M30kT training, validation, and test sets, respectively.Each image is accompanied by one English sentence and its German translation.
- Preprocessing: English and German descriptions are normalized, truecased, tokenized, and converted into subwords using Moses and the Sennrich et al. method.The shared vocabularies contain 83,093 English and 91,141 German subword tokens.
- Preprocessing and evaluation: Sentences longer than 80 tokens are discarded, and models are evaluated on cased, tokenized sentences with punctuation.Model selection uses BLEU on the M30kT validation set.
- Synthetic data: Back-translation converts 145k German M30kC descriptions into English and adds them as synthetic multilingual training data.A German→English baseline trained on textual M30kT data generates the back-translations.
4 Experimental setup
The experiments compare text-only, phrase-based, and multimodal NMT baselines on Multi30k, then test global-image integration strategies and additional back-translated data. Encoder- or decoder-initialized image features yield the strongest overall results, while synthetic data benefits these multimodal models.
- Evaluation: The evaluation uses BLEU4, METEOR, TER, and chrF3, with statistical significance reported for BLEU4, METEOR, and TER.BLEU4, METEOR, and chrF3 are higher-is-better metrics; TER is lower-is-better.
- Baselines and models: The benchmark compares text-only PBSMT and NMT baselines with Huang et al.’s multimodal systems and models using images as source words or encoder/decoder initialization.The multimodal variants are IMG1W, IMG2W, IMGE, and IMGD.
- Results: IMGE and IMGD improve on both text-only baselines across all evaluated metrics, while combining encoder and decoder image features does not improve results over using either location alone.The strongest models also outperform the comparable multimodal NMT baseline of Huang et al. (2016).
- Results: A purely neural model significantly improves over the PBSMT baseline on every evaluated metric on Multi30k, according to the authors.This is reported as the first such result on this data set to the authors’ knowledge.
- Additional data: The approximately 30k-example training set limits real-world applicability, motivating 145k additional synthetic triples created by back-translating German descriptions.The added triples contain a synthetic English description, the original German description, and an image.
- Additional data: With back-translated data, IMGE and IMGD improve significantly in BLEU, METEOR, and TER and also achieve better chrF3 scores, whereas IMG2W shows no significant change in the first three metrics.Adding synthetic sentence pairs to the PBSMT baseline deteriorated its results.
5 Conclusions
The paper finds that global image features improve attention-based NMT when incorporated through several encoder–decoder strategies, outperforming NMT, PBSMT, and a strong multimodal baseline across evaluated metrics. However, feeding image features directly into the decoder at every time step causes overfitting and prevents learning.
- Conclusions: Global image features improve attention-based NMT over NMT, PBSMT, and a strong multimodal baseline according to all evaluated metrics.The authors interpret this as evidence that images can visually ground translations and increase translation quality.
- Conclusions: The models incorporate images as source words, encoder-state initialisation, or additional decoder-state initialisation.These strategies extend multimodal input to different parts of the attention-based encoder–decoder framework.
- Conclusions: Directly supplying image features at every decoder time step causes overfitting and prevents learning.This corroborates previous findings concerning decoder-side image conditioning.
- Future work: Future work will systematically study synthetic back-translated data and investigate local, spatial-preserving image features.These directions extend beyond the global image features studied here.