PaperlayerView exact revision on arXiv

Source-linked AI summary

Fine-Grained Attention Mechanism for Neural Machine Translation

Heeyoul Choi, Kyunghyun Cho, Yoshua Bengio

arXiv:1803.11407v2cs.CL

TL;DR

Existing attention mechanisms assign one scalar score to each context vector, treating its dimensions equally. The paper proposes fine-grained 2D attention with separate scores for each dimension, improving translation quality on En-De and En-Fi by up to +1.4 BLEU while exposing internal context-vector structure.

  • Problem

    Existing attention variants assign one scalar score to each context vector, treating all of its dimensions equally.

  • Method

    The paper extends attention to produce and dimension-wise normalize a separate score for every dimension of each context vector.

  • Results

    The fine-grained attention mechanism improves translation quality by up to +1.4 BLEU on En-De and En-Fi translation tasks.

  • Takeaways & Limitations

    Alignment analysis reveals that different context-vector dimensions play different roles in neural machine translation.

  • Takeaways & Limitations

    The improvements require larger models and additional translation time, with model-size increases of 3.5% and 3.4% and time increases from 4.5% to 14%.

Abstract

from arXiv · show

Neural machine translation (NMT) has been a new paradigm in machine translation, and the attention mechanism has become the dominant approach with the state-of-the-art records in many language pairs. While there are variants of the attention mechanism, all of them use only temporal attention where one scalar value is assigned to one context vector corresponding to a source word. In this paper, we propose a fine-grained (or 2D) attention mechanism where each dimension of a context vector will receive a separate attention score. In experiments with the task of En-De and En-Fi translation, the fine-grained attention method improves the translation quality in terms of BLEU score. In addition, our alignment analysis reveals how the fine-grained attention mechanism exploits the internal structure of context vectors.

1 Introduction

Attention-based NMT is dominant, but existing attention variants assign one scalar score to each context vector and treat its dimensions equally. This paper proposes dimension-specific attention and reports improved translation quality on En-De and En-Fi.

  • Attention-based NMT has achieved state-of-the-art translation quality across several language pairs.
  • Existing attention variants assign one scalar score to each source-associated context vector, treating all vector dimensions equally.
  • Contextualized word embeddings suggest that different representation dimensions can play different roles and improve BLEU scores.
  • The proposed fine-grained attention assigns a separate score to every context-vector dimension instead of one scalar per vector.This allows attention to focus on different interpretations encoded within a single context vector.
  • +1.4 BLEU is the reported maximum improvement on En-De and En-Fi translation tasks.Qualitative alignment analysis also found that fine-grained attention exploits the internal structure of context vectors.

2 Background: Attention-based Neural Machine Translation

Attention-based NMT uses an encoder, decoder, and attention mechanism to compute target translations from source annotations. Conventional variants score each context vector with a single scalar, normalize those scores, and combine the vectors for decoding.

  • The NMT architecture consists of an encoder, decoder, and attention mechanism.
  • The encoder reads source words with a bidirectional recurrent network and forms annotation vectors by concatenating forward and reverse hidden states.
  • The decoder recurrent network uses attention to weight annotation vectors before updating its hidden state and predicting target words.
  • The attention network takes the previous decoder hidden state and an annotation vector as input, then returns a relevance score.The scores are normalized to be positive and sum to 1 before computing the weighted annotation sum.
  • Existing attention variants, including content-, location-, and target-embedding-based variants, still assign a single score to each context vector.
  • The proposed mechanism instead assigns dimension-specific scores, with normalization performed separately for each dimension.

3 Fine-Grained Attention Mechanism

The proposed fine-grained attention assigns separate scores to context-vector dimensions rather than one scalar per vector, then normalizes and combines them dimension-wise.

  • Existing attention assigns one scalar score to each context vector, although its dimensions may represent different perspectives.
  • The proposed score function returns a set of scores corresponding to the dimensions of each context vector.
  • A fully connected network produces dimension-specific scores, with one output node for each context-vector dimension.
  • The dimension-specific scores are normalized dimension-wise before the context vectors are combined.
  • Figure 1 contrasts conventional attention with the proposed fine-grained attention mechanism.

4 Experimental Settings

The experiments evaluate fine-grained attention on large En-De and En-Fi translation datasets using BPE preprocessing, beam-search decoding, BLEU evaluation, and several model variants.

  • The study evaluates En-De and En-Fi translation using 4.5M and 2M WMT’15 sentence pairs, respectively.
  • The translation data use 30k BPE subwords, with translations un-BPE’d before BLEU computation.
  • Beam search uses width 12, and BLEU is measured on newstest2015 alongside the stated validation sets.
  • The baseline is an attention-based Bahdanau model with an LSTM replacing the original GRU and specified 30K vocabularies and hidden dimensions.
  • The experiments compare baseline attention, attention using the previous decoded symbol, and the proposed AttY2D model.
  • They also evaluate contextualized word embeddings from Choi et al. (2017) as a related representation-based comparison.

5 Experiments

Across En-De and En-Fi, fine-grained attention improves BLEU while adding modest model size and translation-time costs. Alignment analyses show that its dimension-specific scores produce focused, sparse patterns that capture differing syntactic and semantic information.

  • Quantitative Analysis: +1.4 BLEU is achieved by the fine-grained model compared with conventional attention across En-De and En-Fi.The comparison includes Baseline, Baseline+AttY, and Baseline+AttY2D models.
  • Quantitative Analysis: Contextualization alone improves BLEU by up to +1.8, while fine-grained attention adds up to +1.4 BLEU.The same improvement pattern is observed when contextualization is included.
  • Quantitative Analysis: The proposed model increases size by 3.5% or 3.4% and requires 4.5% to 14% more translation time.The size comparisons are from +AttY to +AttY2D and from +C+AttY to +C+AttY2D.
  • Alignment Analysis: Fine-grained attention assigns separate scores to context-vector dimensions, whose slices reveal different syntactic and semantic focus patterns.Examples distinguish word types such as articles, prepositions, nouns, and adjectives, while another dimension attends to politically related words.
  • Alignment Analysis: Averaged alignments remain similar across models, but the proposed model produces clearer, more focused alignment.The comparison covers vanilla attention, AttY, and AttY2D in En-De.
  • Alignment Analysis: Across the first 50 displayed dimensions, each source word receives a different, very sparse attention pattern.Figure 3 shows source sub-words vertically and context-vector dimensions horizontally.

6 Conclusions

The paper proposes fine-grained attention for NMT and reports improved translation quality on En-De and En-Fi. Alignment analysis shows that context-vector dimensions play different roles, while broader applications remain future work.

  • The proposed fine-grained, or 2D, attention mechanism improves NMT translation quality on En-De and En-Fi.
  • Alignment analysis shows that different context dimensions play different roles in neural machine translation.
  • The paper identifies testing fine-grained attention with other NMT architectures and tasks such as speech recognition as future work.
Loading 1803.11407v2…