PaperlayerView exact revision on arXiv

Source-linked AI summary

What does Attention in Neural Machine Translation Pay Attention to?

Hamidreza Ghader, Christof Monz

arXiv:1710.03348v1cs.CL

TL;DR

Neural machine translation uses attention to select source-sentence information at each translation step, but how attention relates to traditional alignment and what it captures remain open questions. The paper compares attention with alignment and finds that attention agrees with alignment to varying degrees while also capturing other information, with patterns differing by word type and attention mechanism.

  • Problem

    Although attention is often treated as an alignment model, the paper asks whether it only models translational equivalents or captures other information.

    Source: §1 · pp. 1–2

  • Method

    The paper compares attention and traditional word alignment, examining attention patterns across word types and attention mechanisms.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Attention agrees with traditional alignment to varying degrees but also captures information beyond translational equivalents, with patterns differing by target-word type and attention mechanism.

    Source: §1 · p. 2, §6 · pp. 8–9

  • Takeaways & Limitations

    Alignment guidance is useful for generating nouns, but for verbs it may force attention to forget useful information already captured beyond alignment points.

    Source: §6 · pp. 8–9

    The analysis tests whether greater attention–alignment consistency is associated with better translation quality.

    Source: §4 · p. 3

Abstract

from arXiv · show

Attention in neural machine translation provides the possibility to encode relevant parts of the source sentence at each translation step. As a result, attention is considered to be an alignment model as well. However, there is no work that specifically studies attention and provides analysis of what is being learned by attention models. Thus, the question still remains that how attention is similar or different from the traditional alignment. In this paper, we provide detailed analysis of attention and compare it to traditional alignment. We answer the question of whether attention is only capable of modelling translational equivalent or it captures more information. We show that attention is different from alignment in some cases and is capturing useful information other than alignments.

1 Introduction

Attention in neural machine translation selects source-sentence parts at each translation step, but its relationship to traditional word alignment and its treatment of spread-out attention remain open questions. The paper finds that attention can track alignment in some cases while capturing other useful information in others.

1 Introduction

  • Attention is motivated by its ability to use relevant source-sentence parts at each translation step.The introduction places attentional NMT within the encoder-decoder approach to neural machine translation.Source: §1 · p. 1
  • The study compares attention with word alignment and examines what attention captures across syntactic phenomena.It also reports that attention mechanisms differ in alignment compliance, and that global compliance is not always helpful for word prediction.Source: §1 · p. 2
  • Attention agrees with traditional alignment for nouns but captures information beyond translational equivalents for verbs.The paper asks whether attention’s spread across multiple source words is an error or a desired behavior, and finds that its patterns vary by target-word type.Source: §1 · pp. 1–2

2 Related Work

Related work studies supervised alignment guidance, information encoded by neural machine translation systems, and how attention mass matches alignments. Mixed findings on alignment-guided attention motivate a more thorough analysis.

2 Related Work

  • Prior work reports that attention can affect encoded information, including more efficient learning of source-side POS tags in systems with attention.Other studies examined syntactic information in encoder states without an attentional model and morphological information with attention.Source: §2 · p. 2
  • Koehn and Knowles (2017) briefly measured attention’s match to automatic alignments using the probability mass assigned to alignment points.They also reported differences based on the most attended words.Source: §2 · p. 2
  • Prior alignment-guided attention studies report mixed results, motivating a more thorough analysis of attention models in neural machine translation.Reported benefits vary by domain, and approaches include feeding automatic alignments to attention or separating alignment and translation models.Source: §2 · p. 2

3 Attention Models

The paper investigates two attention models: non-recurrent attention, equivalent to Luong et al.’s global attention, and input-feeding attention. Both compute a context vector at each time step, but input-feeding incorporates the preceding context into its computation.

3 Attention Models

  • Both non-recurrent and input-feeding attention compute a context vector at each time step, which is concatenated with the decoder hidden state before the softmax output layer.The paper notes that Table 1 reports statistics for manual alignments from RWTH German–English data.Source: §3 · p. 3, Table 1 · p. 3
  • Non-recurrent attention compares the decoder hidden state with each encoder hidden state, often using a dot product, then applies softmax to produce attention weights.Those weights are used to compute a weighted sum over encoder hidden states, yielding the context vector.Source: §3 · p. 3
  • Input-feeding attention uses a hidden state informed by the previous context to compare against encoder hidden states.It feeds back its previous output rather than using the context-independent decoder state.Source: §3 · p. 3
  • The input-feeding network applies its stacked LSTM function to the preceding generated target word and its own prior output.The prior output is the result of the earlier input-feeding step.Source: §3 · p. 3

4 Comparing Attention with Alignment

The section tests whether greater consistency between attention and traditional word alignment corresponds to better translation. It compares attention with softened manual alignments and also measures how concentrated attention is.

4 Comparing Attention with Alignment

  • The analysis asks whether higher attention–alignment consistency corresponds to better translation quality.It compares attention loss with word prediction loss to examine this relationship.Source: §4 · p. 3, §4.1 · p. 4

4.1 Measuring Attention-Alignment Accuracy

  • Manual RWTH alignments are converted to soft alignments, and cross-entropy measures how closely attention matches them.Unaligned target words are assumed to align with all source words during conversion.Source: §4.1 · pp. 3–4
  • Spearman rank correlation compares attention loss with word prediction loss at each translation step.Word prediction loss is the negative log probability assigned to the reference target word.Source: §4.1 · p. 4
  • For “will” and “come,” attention is distributed across the manually aligned source word and other words.The figure outlines manual alignments and illustrates cases where attention deviates from them.Source: §4.1 · p. 4, Figure 2 · p. 4

4.2 Measuring Attention Concentration

  • Attention concentration is measured using the entropy of the attention distribution.The section contrasts this potentially broad distribution with alignments that typically involve one or a few words.Source: §4.2 · p. 4

5 Empirical Analysis of Attention Behaviour

The analysis compares global and input-feeding attention models trained on WMT15 German-to-English data, then examines how attention’s relation to word alignment varies by target-word type and distribution. It finds that closer alignment does not uniformly correspond to better translation.

5.1 Impact of Attention Mechanism

  • Input-feeding attention has higher BLEU and closer agreement with human alignments than non-recurrent attention in the reported comparison.Attention loss comparisons between the two systems follow the same pattern as AER.Source: §5.1 · p. 5, Table 4 · p. 5, Table 5 · p. 5

5.2 Alignment Quality Impact on Translation

  • Verbs have almost twice the average attention loss of nouns, yet lower average word-prediction loss; the loss relationship is low for verbs but higher for nouns.The authors interpret this as attention to source words beyond the aligned word being necessary for translating verbs, while alignment consistency is more desirable for nouns.Source: §5.2 · pp. 6–7, Figure 3 · p. 6, Figure 4 · p. 6

5.3 Attention Concentration

  • Noun attention is relatively concentrated and its entropy correlates highly with attention loss; verb attention is more distributed, while concentration is not required for translating verbs.Pronouns and particles also have low entropy–word-prediction-loss correlation; the authors suggest distributed attention may deliberately cover multiple relevant places, consistent with their relatively low word-prediction losses.Source: §5.3 · p. 7, Figure 5 · p. 7, Figure 6 · p. 7

5.4 Attention Distribution

  • Less than half of attention goes to alignment points for most POS tags; for the remaining attention, commonly attended roles include adjectives and determiners for nouns and auxiliaries, adverbs, subjects, and objects for verbs.The analysis measures attention mass on alignment points by POS tag, then examines the rest of the mass over source-side dependency roles.Source: §5.4 · p. 8, Table 8 · p. 8, Table 7 · p. 8

6 Conclusion

Attention’s agreement with word alignment varies by attention mechanism and target part of speech. Alignment supervision is useful for nouns, but for verbs it can suppress other relevant information captured by attention.

6 Conclusion

  • Attention agrees with traditional alignment to varying degrees depending on the attention mechanism and target part of speech.The conclusion distinguishes the concentrated attention pattern and relatively high alignment correlations for nouns from the broader attention paid to non-alignment words for verbs.Source: §6 · p. 8
  • Explicit alignment labels are useful for generating nouns, but not verbs, whose attention captures other relevant information.Forcing verb attention to follow alignment points would make the model forget that useful information.Source: §6 · pp. 8–9
  • This difference explains mixed results reported when attention is guided to comply with alignments.Source: §6 · p. 9

Related papers

Loading 1710.03348v1…