Fine-Grained Attention Mechanism for Neural Machine Translation

• Attention-based NMT became dominant and achieved state-of-the-art quality across several language pairs, amid advances in vocabulary handling, monolingual data, and attention (Bahdanau et al., 2015; Jean et al., 2015a; Sennrich et al., 2016).Source: §1 · p. 1
• Existing attention variants assign one scalar score per source-symbol context vector, treating all its dimensions equally.The introduction notes the same temporal-attention pattern in image captioning, where scores correspond to spatial locations (Xu et al., 2015).Source: §1 · p. 1
• Prior work found that embedding dimensions can play different contextual roles, with contextualization improving translation BLEU scores.Source: §1 · p. 2
• The proposed mechanism assigns a separate attention score to each context-vector dimension, enabling finer-grained focus on possible interpretations of a word.Unlike existing variants, it outputs as many scores as there are dimensions in a context vector (Choi et al., 2017; Van der Maaten and Hinton, 2012).Source: §1 · p. 2
• On En-De and En-Fi, fine-grained attention improves translation quality by up to +1.4 BLEU.Qualitative analysis found that it exploits the internal structure of each context vector.Source: §1 · p. 2

2 Background: Attention-based Neural Machine Translation

• The NMT architecture combines an encoder, decoder, and attention mechanism, with the decoder using source annotations to generate target words sequentially.A bidirectional encoder forms source annotations, and attention weights them to create a context used to update the decoder state.Source: §2 · pp. 2–3

2.1 Variants of Attention Mechanism

• In this variant, the score function receives information from the previous word—using the true target during teacher-forced training and the previously generated word at test time—yet gives a single score to context vector ht.Source: §2.1 · p. 3
• Figure 1 contrasts conventional attention, whose scores sum to 1 across source positions, with fine-grained attention, whose scores sum to 1 separately for each dimension.The proposed mechanism therefore gives each context-vector dimension its own attention distribution.Source: Figure 1 · p. 4

3 Fine-Grained Attention Mechanism

• Fine-grained attention assigns a separate score to each context-vector dimension rather than one scalar per vector.The motivation is that each dimension can represent a different perspective on the encoded representation’s internal structure.Source: §3 · p. 4
• A fully connected network outputs d dimension-specific scores for each context vector at decoder time t′.The scores are normalized dimension-wise across context vectors.Source: §3 · p. 4
• The context vectors are combined using element-wise multiplication with the dimension-specific scores.The passage identifies ⊙ as element-wise multiplication.Source: §3 · p. 4

4.1 Tasks and Corpora

• The tasks use WMT’15 corpora with 4.5M En-De and 2M En-Fi sentence pairs; En-De follows Jean et al. (2015a) preprocessing, while En-Fi uses tokenization only.En-De preprocessing yields 100M English-side words; both sides use 30k BPE subwords, and BLEU is computed on un-BPE’d but tokenized translations.Source: §4.1 · p. 5

4.2 Decoding and Evaluation

• Decoding uses forward beam search of width 12; En-De validation and test sets are newstest2013 and newstest2015, while En-Fi uses newsdev2015 and newstest2015.BLEU compares decoded translations against reference sentences and is computed over sentence sets.Source: §4.2 · p. 5

4.3 Models

• The Bahdanau et al. (2015) baseline replaces GRU with LSTM and uses 30K source and target vocabularies, 620-dimensional embeddings, 1K hidden nodes, and 2K alignment hidden nodes.Source: §4.3 · p. 5
• The model comparisons include AttY, which feeds the previously decoded symbol yt−1 into fAtt, and the proposed fine-grained model AttY2D.Source: §4.3 · p. 5
• The experiments also evaluate Context, which enriches word embeddings with contextual information, following Choi et al. (2017).Source: §4.3 · p. 5
• Models are trained with Adam (Kingma and Ba (2014)) until validation BLEU stops improving, using greedy validation search and sentences up to 50 words.Greedy search is used during training validation to reduce computational overhead.Source: §4.3 · p. 5

5.1 Quantitative Analysis

• Up to +1.4 BLEU is reported for fine-grained attention over conventional attention on both En-De and En-Fi.With contextualization, fine-grained attention adds up to +1.4 BLEU, while contextualization alone improves BLEU by up to +1.8 over the baseline.Source: §5.1 · pp. 5–6, Table 1 · p. 6
• Model size rises 3.5% from +AttY to +AttY2D and 3.4% from +C+AttY to +C+AttY2D; translation takes 4.5%–14% longer.The translation-time comparison is summarized in Table 2.Source: §5.1 · p. 6, Table 2 · p. 6

5.2 Alignment Analysis

• Averaging fine-grained attention over context dimensions yields source-target alignments comparable to conventional attention, with a clearer, more focused pattern.The fine-grained mechanism produces a 3-D tensor over source symbols, target symbols, and context dimensions, which in this case number 2000.Source: §5.2 · p. 6, Figure 2 · p. 7
• Averaging over targets reveals sparse attention patterns, with each source word receiving different patterns across context dimensions.This view is intended to show which context-vector dimensions matter for translating each source symbol, without specifying its aligned target symbols.Source: §5.2 · p. 7, Figure 3 · p. 7
• Example dimensions attend to different syntactic and semantic information, with syntactic patterns varying by word type and similar patterns observed in other examples.The examples include articles, prepositions, nouns, and adjectives; one dimension shows attention to “Republican,” “strategy,” “election,” and “Obama,” interpreted as politics-related.Source: §5.2 · p. 7

6 Conclusions

• Fine-grained attention significantly improves En-De and En-Fi translation quality, with further improvement when combined with the previous contextualization technique.The paper describes contextualization as based on a similar idea.Source: §6 · p. 7
• Alignment analysis shows that context-vector dimensions play different roles, with example dimensions attending to grammatical categories and politics-related semantic words.The examples include articles, prepositions, nouns, adjectives, and words representing politics.Source: §6 · p. 7, Figure 4 · p. 8
• The authors propose testing fine-grained attention with character-level or multilayer encoder-decoder NMT models and applying it to speech recognition.They cite Ling et al. (2015) and Chung et al. (2016a) for the NMT models.Source: §6 · p. 8