PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Machine Translation with Recurrent Attention Modeling

Zichao Yang, Zhiting Hu, Yuntian Deng, Chris Dyer, Alex Smola

arXiv:1607.05108v1cs.NEcs.CL

TL;DR

Attention-based translation needs to use prior attention decisions when predicting future alignments, but existing approaches make limited use of this history. The paper adds recurrent dynamic memory for each source word and reports improved translation quality over independent attentive models on two MT datasets.

  • Problem

    Attention models provide an evolving input view but make limited use of historical attention masks, despite prior attention informing future alignment decisions.

    Source: §1 · p. 1

  • Method

    A recurrent network summarizes each source word’s local attention history and concatenates that dynamic memory with its static annotation for attention addressing.

    Source: §1 · p. 1, §2.2 · p. 3

  • Results

    The method improves over plain attentive neural models on two MT datasets, with larger attention windows producing better performance.

    Source: §1 · p. 1, §4 · p. 5

  • Takeaways & Limitations

    Explicitly modeling attention history captures coverage and ordering information useful for neural machine translation.

    Source: §1 · p. 1

    The authors identify extending the approach to fully dynamic source representations as future work.

    Source: §4 · p. 5

Abstract

from arXiv · show

Knowing which words have been attended to in previous time steps while generating a translation is a rich source of information for predicting what words will be attended to in the future. We improve upon the attention model of Bahdanau et al. (2014) by explicitly modeling the relationship between previous and subsequent attention levels for each word using one recurrent network per input word. This architecture easily captures informative features, such as fertility and regularities in relative distortion. In experiments, we show our parameterization of attention improves translation quality.

1 Introduction

The paper addresses attention models that underuse historical attention decisions and must handle variable-length source sentences. It models each source word’s attention history recurrently, capturing coverage and ordering patterns while improving translation quality.

  • The model addresses attention mechanisms’ limited use of historical attention masks and the need to model attention independently of content.Source: §1 · p. 1
  • Variable-length source sentences require models that handle different numbers of attention predictions.Source: §1 · p. 1
  • An RNN summarizes each source word’s preceding attention levels within a fixed window, learning coverage constraints and handling variable input sizes.Source: §1 · p. 1
  • The resulting summaries are concatenated with context vectors to predict attention, and experiments show improvements over plain attentive neural models on two MT datasets.Source: §1 · p. 1

2 Model

The model combines a bidirectional-LSTM source encoder and attention-based decoder with per-source-word dynamic memories. These memories summarize local attention history and augment addressing without changing the context vectors that are read.

2.1 Neural Machine Translation

2.1.1 Encoder

  • 2.1.1 Encoder: Bidirectional LSTMs produce annotations for each source word, forming static memory vectors used to generate the target sentence.Source: §2.1.1 · p. 2

2.1.2 Attention based decoder

  • 2.1.2 Attention based decoder: Content-based attention weights source annotations, whose weighted average is combined with the decoder state to predict each target word.Source: §2.1.2 · p. 2
  • 2.1.2 Attention based decoder: The baseline attention treats decisions independently, missing dependencies such as repeated coverage and neighboring-word selection.Source: §2.1.2 · p. 2

2.2 Dynamic Memory

  • 2.2 Dynamic Memory: The dynamic memory uses a window of neighboring attention levels, while the actual context memory excludes the dynamic vector.Source: §2.2 · pp. 2–3
  • 2.2 Dynamic Memory: Dynamic memory vectors are updated during decoding, summarize local attention history, and are concatenated with static source annotations for attention addressing.Source: §2.2 · pp. 2–3

3 Experiments & Results

Experiments evaluate the attention model on English–German and Chinese–English translation using BLEU and compare it with baselines and related systems. The model improves translation quality, with larger gains reported for Chinese–English.

3.1 Data sets

  • The experiments use WMT English–German and NIST Chinese–English data, with tokenized BLEU as the translation-quality metric.The English–German data contain about 4.5 million sentence pairs, while the Chinese–English data contain about 1.5 million.Source: §3.1 · p. 3

3.2 Experiments configuration

  • Training excludes sentences longer than 50 words, uses 50k and 30k vocabularies, and replaces out-of-vocabulary words with UNK.The English–German and Chinese–English models use different learning-rate schedules and training durations.Source: §3.2 · p. 3
  • The study compares attention windows of size 1 and 11 by evaluating k = 0 and k = 5.Source: §3.2 · p. 3

3.3 Results

  • 0.8 and 1.6 BLEU score improvements over the best related systems are reported for English–German and Chinese–English, respectively.The comparison includes systems using deeper LSTMs, local attention, and character-based encoding.Source: §3.3 · p. 4
  • 0.5 and 0.7 BLEU improvements over RNNSearch are reported on English–German newstest2014 and newstest2015.Using a window size of 1 provides little additional improvement, while UNK replacement yields BLEU scores of 22.1 and 25.0.Source: §3.3 · p. 4
  • Chinese–English gains are larger: coverage improves BLEU by 0.6, a window size of 11 by 1.5, and UNK replacement by a further 0.5.The text attributes the smaller UNK-replacement gain relative to English–German to less lexical overlap between Chinese and English.Source: §3.3 · p. 4

4 Conclusions & Future Work

The paper concludes that explicitly modeling attention history improves attention-based translation, and reports gains across two machine-translation datasets. It also identifies larger context windows as beneficial and proposes fully dynamic source-memory updates for future work.

4 Conclusions & Future Work

  • A recurrent network summarizes preceding attentions, and experiments on two translation datasets show better performance than previous independent attentive models.The method explicitly incorporates attention history when generating the current attention map.Source: §4 · pp. 4–5
  • Future work would extend the model to fully dynamic memory by directly updating source-word representations during target generation.Source: §4 · p. 5
Loading 1607.05108v1…