PaperlayerView exact revision on arXiv

Source-linked AI summary

Monotonic Multihead Attention

Xutai Ma, Juan Pino, James Cross, Liezl Puzon, Jiatao Gu

arXiv:1909.12406v1cs.CL

TL;DR

Simultaneous machine translation seeks to generate translations before reading the full source, while prior learned monotonic methods relied on weaker RNN architectures. The paper introduces Monotonic Multihead Attention with two latency-control regularizers for Transformer models and reports better latency-quality tradeoffs than MILk. It further analyzes attention-span control and how decoder layers and heads affect quality and latency.

  • Problem

    Prior simultaneous translation approaches either apply fixed policies to Transformers or use learned monotonic attention with weaker RNN-based models.

  • Method

    The paper extends monotonic attention to multihead Transformer attention through MMA-H and MMA-IL, adding regularizers for average latency and cross-head positional consistency.

  • Results

    MMA achieves better latency-quality tradeoffs than MILk, the previous state-of-the-art approach, on two standard translation benchmarks.

  • Takeaways & Limitations

    MMA combines Transformer architecture with monotonic attention for stronger simultaneous sequence-to-sequence models and interpretable control of latency and attention span.

  • Takeaways & Limitations

    MMA-H may provide insufficient translation information because each head attends to one encoder state, while a head that remains at the beginning can degrade quality and is unsuitable for streaming.

Abstract

from arXiv · show

Simultaneous machine translation models start generating a target sequence before they have encoded or read the source sequence. Recent approaches for this task either apply a fixed policy on a state-of-the art Transformer model, or a learnable monotonic attention on a weaker recurrent neural network-based structure. In this paper, we propose a new attention mechanism, Monotonic Multihead Attention (MMA), which extends the monotonic attention mechanism to multihead attention. We also introduce two novel and interpretable approaches for latency control that are specifically designed for multiple attentions heads. We apply MMA to the simultaneous machine translation task and demonstrate better latency-quality tradeoffs compared to MILk, the previous state-of-the-art approach. We also analyze how the latency controls affect the attention span and we motivate the introduction of our model by analyzing the effect of the number of decoder layers and heads on quality and latency.

1 INTRODUCTION

Simultaneous machine translation must generate translations before reading the complete source, but prior approaches either use fixed policies with Transformers or learned monotonic attention with weaker RNN models. MMA extends monotonic attention to Transformer multihead attention and achieves better latency-quality tradeoffs than MILk on two translation benchmarks.

  • 1 INTRODUCTION: Simultaneous translation generates target text before reading the entire source, supporting real-time applications such as live captions and multilingual conversations.Unlike usual neural translation, the encoder does not first consume the complete source sentence.
  • 1 INTRODUCTION: Prior monotonic attention methods, including MILk, use RNN-based models, although Transformer models with multilayer multihead attention have surpassed RNNs in machine translation.MILk had improved latency-quality tradeoffs over fixed policies and earlier monotonic variants, making it the relevant prior state of the art.
  • 1 INTRODUCTION: MMA extends monotonic attention to multihead Transformer attention, enabling online decoding while combining Transformer capacity with monotonic-attention efficiency.The model includes Hard MMA and Infinite Lookback MMA variants.
  • 1 INTRODUCTION: The paper introduces two latency regularizers: one directly minimizes average latency, while the other keeps attention heads at similar positions.The second regularizer prevents latency from being dominated by one or a few heads.
  • 1 INTRODUCTION: MMA achieves better latency-quality tradeoffs than MILk on IWSLT15 English-Vietnamese and WMT15 German-English benchmarks.The paper also analyzes attention-span control and the effects of decoder-layer and decoder-head counts on quality and latency.

2 MONOTONIC MULTIHEAD ATTENTION MODEL

MMA extends monotonic attention to Transformer multihead decoder-encoder attention, giving each head an independent monotonic schedule while retaining access to prior encoder states. Its hard and infinite-lookback variants, together with latency controls, balance streaming efficiency, translation information, and head-level synchronization.

  • MMA architecture: MMA assigns each decoder-encoder attention head an independent monotonic attention process, combining Transformer multihead expressiveness with monotonic attention's low latency.The selection process is independent for each head in each decoder layer.
  • MMA attention behavior: MMA lets different heads adjust reading speed and preserve source history by attending to different encoder positions, including past states in MMA-H.Some heads can read new inputs while others remain at earlier positions.
  • MMA variants: MMA-H hard-attends to one encoder state per head, whereas MMA-IL can attend to all previous encoder states for greater translation information.MMA-H is better suited to stricter streaming efficiency requirements, while MMA-IL leverages more source information.
  • MMA decoding: At inference, each head samples its own read or write position, and the decoder emits a target token only after all heads have finished reading.The per-head computations are performed in parallel, but token generation waits for every attention to decide to write.
  • Latency control: Latency control uses weighted average latency and head-position regularization to control overall speed, head divergence, attention span, and reading buffer.The average-loss term controls overall speed, while the variance term encourages heads to maintain similar positions; MMA-IL uses both, whereas MMA-H uses only the variance term.
  • Limitations: MMA-H can produce degenerate alignments, such as staying at the beginning or assigning the end-of-sentence token, which may degrade quality and streaming suitability.A head that remains at the beginning does not create latency but can leave the decoder without useful input information.

3 EXPERIMENTAL SETUP

The experiments evaluate MMA-H and MMA-IL on two standard translation datasets using BLEU for quality and multiple latency metrics, with comparisons to offline models and MILk.

  • 3.1 Evaluation Metrics: MMA-H and MMA-IL are evaluated on IWSLT15 En-Vi and WMT15 De-En, with BLEU measuring quality and AP, AL, and DAL measuring latency.The datasets use different preprocessing and tokenization conventions, including Moses tokenization for both and joint 32K BPE for WMT15 De-En.
  • 3.2 Datasets: The setup reports dataset split sizes and offline model performance as reference points for the simultaneous translation experiments.Table 1 covers sentence counts by split, while Table 2 reports offline performance with unidirectional encoders and greedy decoding.
  • 3.2 Datasets: The IWSLT15 En-Vi data uses tst2012 for validation and tst2013 for testing, while WMT15 De-En uses newstest2013 for validation and newstest2015 for testing.IWSLT15 replaces words occurring fewer than five times with <unk>; WMT15 uses jointly learned source-target BPE.
  • 3.3 Models: The evaluated MMA models replace encoder-decoder layers in the offline models while retaining other hyperparameter settings, and MILk uses a specified IWSLT15 implementation.All offline models use unidirectional encoders, whose self-attention attends only to previous states.
  • 3.3 Models: The reported BLEU comparison for IWSLT15 includes tokenized BLEU because the acquired data were already tokenized and the original tokenizer was unavailable.Latency metrics are computed on BPE tokens for WMT15 De-En and word tokens for IWSLT15 En-Vi.

4 RESULTS

MMA achieves better latency-quality tradeoffs than MILk, while its variance-based latency control reduces attention span and quality generally improves with more decoder heads and layers at increasing latency.

  • 4.1 LATENCY-QUALITY TRADEOFFS: For a given latency, MMA-H and MMA-IL obtain better translation quality than MILk on the evaluated translation benchmarks.The comparison uses BLEU and latency scores selected from validation checkpoints; even MMA-H outperforms MILk despite each head attending to one state.
  • 4.2 ATTENTION SPAN: Increasing Lvar reduces the average attention span, confirming that the variance loss controls how far apart the attention heads read.The average attention span is averaged over the IWSLT and WMT test sets and estimates the reading buffer needed for streaming translation.
  • 4.3 EFFECT ON NUMBER OF LAYERS AND NUMBER OF HEADS: Quality generally improves as the number of decoder attention heads and layers increases for both the offline baseline and MMA-H.The analysis reports detokenized BLEU and DAL on the WMT13 validation set with λavg = 0 and λvar = 0.2.
  • 4.3 EFFECT ON NUMBER OF LAYERS AND NUMBER OF HEADS: Latency also increases with more decoder heads and layers when loss weights remain fixed.The authors note that λvar should increase when more heads are involved to control latency more effectively.

5 RELATED WORK

Related work on simultaneous machine translation includes rule-based reading policies and reinforcement-learning approaches for learning read-write behavior.

  • Rule-based and learned policies: Rule-based approaches schedule reading and writing with policies such as Wait-If-*, wait-k, and incremental decoding.These methods enable simultaneous decoding by specifying when the model reads input and produces output.
  • Rule-based and learned policies: Reinforcement-learning approaches learn a read-write policy from model states, including methods based on Markov chains for phrase-based translation.The related-work discussion places these methods in a separate category from rule-based schedules.

6 CONCLUSION

The paper proposes stronger simultaneous sequence-to-sequence models by combining Transformer architecture with monotonic multihead attention and targeted losses for latency and attention-span control.

  • Two monotonic multihead attention variants and two targeted loss terms control latency and attention span while leveraging Transformer quality.The paper presents detailed ablation studies supporting the efficacy and rationale of the approach.

A.1 HYPERPARAMETERS

This appendix section identifies the hyperparameter settings used for offline and monotonic Transformer models.

  • Table 3 lists the hyperparameters used for offline and monotonic models.

A.2 LATENCY METRICS DEFINITIONS

This appendix section defines latency metrics from the delays incurred when generating target tokens.

  • Table 4 defines AP, AL, and DAL from the per-target-token generation delays g.
Loading 1909.12406v1…