PaperlayerView exact revision on arXiv

Source-linked AI summary

Doubly-Attentive Decoder for Multi-modal Neural Machine Translation

Iacer Calixto, Qun Liu, Nick Campbell

arXiv:1702.01287v1cs.CL

TL;DR

The paper addresses limited gains from spatial visual features in multi-modal translation. It introduces a decoder with separate source-word and image-region attention, and reports state-of-the-art Multi30k results while exploiting in-domain multi-modal and general-domain text-only data.

  • Problem

    Prior spatial-feature multi-modal NMT models did not significantly improve over comparable global-feature or text-only models.

  • Method

    The model uses a doubly-attentive decoder with independent attention over source-language words and spatial image features, pretrained with additional in-domain and general-domain data.

  • Results

    The model reports new state-of-the-art results on the M30kT test set and compares favourably with NMT and PBSMT baselines.

  • Takeaways & Limitations

    Images bring useful information when sentences describe illustrated objects, while the model can exploit additional data regardless of domain.

  • Takeaways & Limitations

    Future work includes incorporating coverage into the model and applying it to other Natural Language Processing tasks.

Abstract

from arXiv · show

We introduce a Multi-modal Neural Machine Translation model in which a doubly-attentive decoder naturally incorporates spatial visual features obtained using pre-trained convolutional neural networks, bridging the gap between image description and translation. Our decoder learns to attend to source-language words and parts of an image independently by means of two separate attention mechanisms as it generates words in the target language. We find that our model can efficiently exploit not just back-translated in-domain multi-modal data but also large general-domain text-only MT corpora. We also report state-of-the-art results on the Multi30k data set.

1 Introduction

The paper proposes an attention-based multi-modal translation model that independently attends to source words and image regions. It addresses limited gains from prior spatial-feature models and leverages both multi-modal and text-only data.

  • The model uses two independent attention mechanisms over source-language words and different image areas while generating translations.
  • The proposed MNMT architecture incorporates spatial visual features through a separate visual attention mechanism.
  • The models use medium-sized back-translated in-domain multi-modal data and large general-domain text-only MT corpora for pretraining.
  • Images provide useful information when sentences describe objects illustrated in the image.
  • Prior spatial-feature MNMT models had not significantly improved over comparable global-feature or text-only models, motivating this design.

2 Background and Notation

The section formulates attention-based NMT as an encoder–decoder model that learns p(Y | X), with source context dynamically computed during decoding. It specifies a bidirectional GRU encoder, conditional-GRU decoder, and independent attention over source words and image patches in the doubly-attentive extension.

  • NMT formulation: Attention-based NMT directly models p(Y | X) with an encoder, decoder, and attention mechanism conditioned on the source sequence and previously emitted words.The decoder computes target-word probabilities from its hidden state and a time-dependent source context vector.
  • Encoder: The encoder is a bidirectional GRU that reads the source sequence in both directions and concatenates forward and backward annotations.These annotations form the source representation set C used by the decoder.
  • Decoder: The conditional GRU initializes the decoder state and computes each step through REC1, source attention, and REC2.REC1 proposes a hidden state, ATTsrc computes the source context, and REC2 produces the final hidden state.
  • Source attention: Source attention computes normalized alignments between source annotations and the next target word, then forms a weighted source-context sum.The alignment mechanism uses a feed-forward network and learned parameters.
  • Multi-modal extension: The doubly-attentive decoder attends independently to source-language words and image patches while generating translations.This extends the text-only attention mechanism with a separate visual attention stream.

3 Multi-modal NMT

The MNMT model extends attention-based NMT with spatial visual features and a decoder that independently attends to source words and image regions. A gated image context is integrated into the decoder alongside the source context to predict each target word.

  • Visual features: Spatial features extracted from ResNet-50 encode images as a 14×14 grid of 196 region vectors, each with 1024 dimensions.The activations come from the res4f layer and are arranged as a 196×1024 matrix.
  • Doubly-attentive decoder: MNTSRC+IMG uses two separate attention mechanisms over source-language words and visual features within a single decoder RNN.The decoder conditions on its previous hidden state and emitted word, the source sentence, and the image.
  • Visual attention: The visual attention computes a time-dependent image context vector from a hidden-state proposal and spatial image annotations using soft attention.A feed-forward network scores image-region alignments, which are normalized across image patches.
  • Gated image context: A gating scalar β_t weights the expected importance of the image context relative to the next target word at each decoding step.The image context uses this gate, whereas the source context does not.
  • Decoder integration: The modified recurrent unit incorporates the time-dependent source and image context vectors into the decoder hidden state and next-word probabilities.The output distribution uses the multimodal hidden state, the previously emitted word, and both context vectors.

4 Data

The experiments use Multi30k’s translated and comparable image-description expansions, augmenting limited multimodal training data with back-translated descriptions and large English–German text-only corpora. Models are trained and evaluated on tokenized, cased English-to-German sentences with subword vocabularies.

  • Datasets: Multi30k provides translated and comparable multilingual expansions of Flickr30k, referred to as M30kT and M30kC.Flickr30k contains 30k images with five English descriptions per image.
  • Datasets: M30kT pairs one professionally translated German description with each English description and uses 29k, 1,014, and 1k images for training, validation, and testing.M30kC instead provides five independently collected German descriptions per image.
  • Training data: The models are trained on all M30kT training data, with validation BLEU used for model selection and the test set reserved for evaluation.A German-to-English text-only model back-translates 145k German M30kC descriptions into English for additional training data.
  • Training data: About 4.3M English–German sentence pairs from WMT 2015 are concatenated for text-only pre-training.The corpora include Europarl v7, News Commentary, and Common Crawl.
  • Preprocessing and evaluation: Models use Moses normalization and tokenization, subword conversion, shared vocabularies of 83,093 English and 91,141 German tokens, and an 80-token sentence limit.Evaluation uses cased, tokenized sentences with punctuation for English-to-German translation.

5 Experimental setup

The experiments compare text-only, multimodal, and pretrained systems on Multi30k using four translation metrics plus TER. The doubly-attentive model improves over comparable baselines, learns from both in-domain multimodal and general-domain text data, and selectively uses visual features with modest parameter growth.

  • Evaluation: The evaluation compares PBSMT, text-only NMT, Huang et al. (2016)’s multimodal models, and the proposed MNMT models on the M30kT test set.Results use BLEU4, METEOR, TER, chrF3, character-level precision, and character-level recall.
  • Training on M30kT: +1.4 BLEU and +2.7 METEOR are achieved over Huang et al. (2016)’s comparable model when training on M30kT.With more data available to Huang et al.’s model, the proposed model still gains +0.9 METEOR while maintaining the same BLEU4 scores.
  • Training on M30kT: The doubly-attentive model significantly outperforms the text-only NMT baseline on BLEU4, METEOR, and TER, and improves chrF3 by +2.1.Against PBSMT, it significantly improves BLEU4 and TER; METEOR rises by +0.7 but is not significant at p < 0.05.
  • Pre-training: The model learns from both the 145k-instance in-domain back-translated M30kC data and the ∼4.3M-sentence WMT 2015 text-only corpora.The WMT 2015 corpora comprise Europarl, Common Crawl, and News Commentary parallel sentences.
  • Textual and visual attention: The visual β gate exceeds 0.5 for 20% of target words and 0.8 for 8%, mainly on concrete nouns with strong visual appeal.In the illustrated example, image features are used mostly for the visually grounded words Mann and Hut rather than less visual words.
  • Model efficiency: The two-attention design uses ∼213M parameters versus ∼200M for attention-based NMT, an increase of ∼6.6%.The authors characterize this as a compromise between model compactness and flexibility.

6 Related work

Prior multi-modal work incorporated visual features into generation and translation, but this work introduces an end-to-end MNMT decoder with independent textual and visual attention mechanisms.

  • 6 Related work: Earlier multi-modal systems applied visual and textual representations to image description, ranking, video description, and translation-related tasks.These approaches included global image features, transferred visual representations, video frames, and image-region attention.
  • 6 Related work: Prior multi-modal MT approaches used global or spatial visual features in SMT re-ranking or NMT, but purely neural models had not significantly improved over both text-only NMT and SMT.The cited systems achieved some success, while the broader benchmark gap remained.
  • 6 Related work: Huang et al.’s purely MNMT model combined global VGG19 features with RCNN-based image regions and was comparable to an SMT model while improving over a strong text-only NMT baseline.Their models served as baselines for the present experiments.
  • 6 Related work: This work differs by jointly training a doubly-attentive decoder that independently attends to source-language words and image areas.The authors identify this as the first integration of multi-modal inputs in NMT through independent attention mechanisms.

7 Conclusions and Future Work

The paper presents an attention-based MNMT model that incorporates spatial visual information, reports state-of-the-art Multi30kT results, and efficiently exploits additional in-domain and general-domain data. Future work will add coverage and extend the model to other NLP tasks.

  • 7 Conclusions and Future Work: The model incorporates spatial visual information into NMT through a novel attention-based multi-modal architecture.It compares favorably with NMT and PBSMT baselines trained on the same data.
  • 7 Conclusions and Future Work: State-of-the-art results were reported on the M30kT test set, improving over previous multi-modal attention-based models.The paper also reports favorable comparisons with NMT and PBSMT baselines.
  • 7 Conclusions and Future Work: The model efficiently exploited one in-domain multi-modal data set and multiple general-domain text-only MT corpora regardless of domain.This conclusion concerns additional pre-training data used with the MNMT model.
  • 7 Conclusions and Future Work: Future work will incorporate coverage into the model and study its application to other Natural Language Processing tasks.These extensions are identified as directions rather than completed capabilities.
Loading 1702.01287v1…