PaperlayerView exact revision on arXiv

Source-linked AI summary

Synchronous Bidirectional Neural Machine Translation

Long Zhou, Jiajun Zhang, Chengqing Zong

arXiv:1905.04847v1cs.CLcs.AIcs.LG

TL;DR

Unidirectional NMT leaves future target-side context unused, contributing to unbalanced outputs. SB-NMT generates in both directions simultaneously and interactively with one decoder, and reports gains over Transformer across three tasks.

  • Problem

    Left-to-right NMT uses previously generated outputs but leaves future target-side context unused, and the paper identifies unbalanced outputs as a resulting problem.

    Source: §1 · p. 1

  • Method

    SB-NMT uses one decoder for simultaneous, interactive left-to-right and right-to-left generation, so each direction can use outputs predicted by the other.

    Source: §1 · p. 2

  • Results

    3.92, 1.49 and 1.04 BLEU points are the reported gains over Transformer on Chinese-English, English-German and Russian-English, respectively; the paper reports state-of-the-art performance on the first two tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    On Chinese-English test sets, joint modeling and optimization of both directions performs better at leveraging bidirectional decoding than Rerank-NMT and ABD-NMT.

    Source: §4.4 · p. 9

    The authors leave learning a vector-valued λ for future exploration.

    Source: §3.2.1 · p. 5

Abstract

from arXiv · show

Existing approaches to neural machine translation (NMT) generate the target language sequence token by token from left to right. However, this kind of unidirectional decoding framework cannot make full use of the target-side future contexts which can be produced in a right-to-left decoding direction, and thus suffers from the issue of unbalanced outputs. In this paper, we introduce a synchronous bidirectional neural machine translation (SB-NMT) that predicts its outputs using left-to-right and right-to-left decoding simultaneously and interactively, in order to leverage both of the history and future information at the same time. Specifically, we first propose a new algorithm that enables synchronous bidirectional decoding in a single model. Then, we present an interactive decoding model in which left-to-right (right-to-left) generation does not only depend on its previously generated outputs, but also relies on future contexts predicted by right-to-left (left-to-right) decoding. We extensively evaluate the proposed SB-NMT model on large-scale NIST Chinese-English, WMT14 English-German, and WMT18 Russian-English translation tasks. Experimental results demonstrate that our model achieves significant improvements over the strong Transformer model by 3.92, 1.49 and 1.04 BLEU points respectively, and obtains the state-of-the-art performance on Chinese-English and English-German translation tasks.

1 Introduction

The introduction identifies unbalanced outputs as a limitation of unidirectional decoding and proposes a single-model approach that generates in both directions interactively. It reports improvements over Transformer across three translation tasks.

1 Introduction

  • Left-to-right decoding favors prefixes over suffixes and leaves future target-side context unused, contributing to unbalanced outputs.The introduction says this is important for avoiding under-translation.Source: §1 · p. 1
  • Earlier bidirectional approaches used two models or decoders, or allowed only forward decoding to use backward information.The authors propose synchronous interaction between directions as a promising alternative.Source: §1 · p. 2
  • The single-model framework uses one encoder, one decoder and one beam-search algorithm for parallel bidirectional decoding.The authors describe it as end-to-end joint optimization and faster and more compact than two-phase decoding.Source: §1 · p. 2
  • SB-NMT uses one decoder to generate left-to-right and right-to-left outputs simultaneously and interactively.Each direction can use previously generated symbols from the other direction as well as its own history.Source: §1 · p. 2
  • 3.92, 1.49 and 1.04 BLEU points are the reported gains over Transformer on NIST Chinese-English, WMT14 English-German and WMT18 Russian-English, respectively.The paper also reports state-of-the-art BLEU scores of 51.11 and 29.21 on Chinese-English and English-German.Source: §1 · p. 2

2 Background

The background outlines the Transformer encoder-decoder and its attention mechanism, then describes beam search for finding translations.

2 Background

  • The Transformer encoder maps input symbols to representations, while the decoder generates the output sequence one element at a time.Decoder inter-attention uses decoder states as queries and encoder outputs as keys and values.Source: §2 · pp. 2–3
  • Multi-head attention projects queries, keys and values into separate representations, applies scaled dot-product attention per head, then concatenates and projects the results.Figure 2 depicts scaled dot-product attention on the left and multi-head attention on the right.Source: §2 · p. 3, Figure 2 · p. 3
  • Beam search extends only the top-N hypotheses at each step, with hypotheses ranked from most to least probable.A complete hypothesis ends with EOS; Figure 3 illustrates beam size 4 and marks ongoing expansions in black.Source: §2 · p. 3, Figure 3 · p. 3

3 Our Approach

SB-NMT uses one decoder to generate left-to-right and right-to-left translations simultaneously, with each direction using information from the other. Synchronous bidirectional attention connects the directions in beam search and replaces Transformer decoder intra-attention; training addresses indirect self-visibility with pseudo references.

3 Our Approach

  • The approach uses one model and one decoder for parallel L2R and R2L generation, with bidirectional attention enabling joint optimization and a more compact beam-search procedure.The paper frames the method as using target-side history and future information during translation.Source: §3 · p. 3, §3.4 · p. 7

3.1 Synchronous Bidirectional Beam Search

  • The synchronous bidirectional beam search splits its beam between L2R and R2L hypotheses, which exchange predicted future context through SBAtt and compete to provide the final translation.At each step, the best items from each half-beam expand in parallel; the highest-probability complete hypothesis is output, reversing tokens if an R2L hypothesis wins.Source: §3.1 · p. 4

3.2.1 Synchronous Bidirectional Dot-Product Attention

  • SBAtt combines each direction’s conventional history attention with attention to the other direction’s hidden states, allowing L2R and R2L states to be calculated in parallel.For L2R, the forward query attends to backward keys and values to incorporate future information; Fusion combines this with the history representation.Source: §3.2.1 · p. 5

3.2 Synchronous Bidirectional Attention

  • The dot-product attention offers linear interpolation, nonlinear interpolation, and a gate to combine history and future representations.The gate uses element-wise weights to control how much previous-context information is preserved and how much backward hidden-state information is captured.Source: §3.2.1 · p. 5

3.2.2 Synchronous Bidirectional Multi-Head Attention

  • The model’s synchronous bidirectional multi-head attention replaces decoder intra-attention, with forward and backward information interacting in that layer while the decoder streams run in parallel.Its inputs concatenate forward and backward hidden states, and each head attends to both through synchronous bidirectional dot-product attention.Source: §3.2.2 · p. 5, Figure 6 · p. 6

3.3 Integrating Synchronous Bidirectional Attention into NMT

  • SBAtt replaces Transformer decoder intra-attention with synchronous bidirectional attention, while the encoder remains identical to the standard Transformer.The decoder also stacks two sublayers to seek translation-relevant source semantics, then computes next-token probabilities from the final forward and backward states.Source: §3.3 · p. 6

3.4 Training

  • Training maximizes likelihood for both decoding directions and constructs pseudo references to address indirect self-visibility between the two directions.Separate L2R and R2L models generate pseudo translations, which are paired with gold references to form training triples.Source: §3.4 · pp. 6–7

4 Experiments

SB-NMT is evaluated on three translation tasks and outperforms Transformer on each. The experiments also analyze translation errors, sentence length, beam size, and data use.

4.1 Dataset

  • Evaluation covers NIST Chinese-English, WMT14 English-German, and WMT18 Russian-English, with about 2.0M, 4.5M, and 14M training pairs, respectively.Source: §4.1 · p. 7, §4 · p. 7

4.2 Setting

  • The Transformer-based models use six encoder and decoder layers, 1024-dimensional hidden states, and beam size 4; English-German and Russian-English average the last 20 checkpoints.Source: §4.2 · pp. 7–8

4.4–4.6 Results

  • SB-NMT outperforms Transformer by 3.92 BLEU on Chinese-English, 1.49 on WMT14 English-German, and 1.04 on WMT18 Russian-English test; the Russian-English development gain is 1.10.It reports a state-of-the-art BLEU score of 29.21 on WMT14 English-German.Source: §4.4 · p. 8, §4.5 · p. 9, §4.6 · p. 9

4.3 Baselines

  • Baselines include RNMT, left-to-right and right-to-left Transformers, reranking, and asynchronous bidirectional decoding.Rerank-NMT independently searches forward and reverse models; ABD-NMT uses a backward decoder before forward search.Source: §4.3 · p. 8

4.7 Analysis

  • SB-NMT quality remains stable as beam size grows, unlike L2R and R2L decoding, which degrade beyond narrow beams.The model adds no parameters beyond λ compared with standard Transformer.Source: §4.7 · pp. 9–10, Figure 8 · p. 10
  • SB-NMT improves BLEU across all source-length groups and alleviates the degradation of Transformer and Transformer (R2L) on long inputs.Source: §4.7 · p. 10, Figure 9 · p. 11
  • Using six data triples improves SB-NMT by 1.03 BLEU over two triples; decoding runs at 17.87 sentences per second, two to three times faster than Rerank-NMT and ABD-NMT.Source: Table 6 · p. 9, §4.7 · pp. 10–11, Table 7 · p. 11
  • SB-NMT reduces under-translation errors by 30.6%, while Transformer and Transformer (R2L) have 7.85% and 7.81% errors.Case examples report that Transformer favors prefixes and Transformer (R2L) favors suffixes.Source: §4.7 · p. 11, Table 8 · p. 11, Table 9 · p. 12

5 Related Work

Prior sequence decoders and bidirectional translation methods address future information and output imbalance in different ways. The paper positions synchronous bidirectional decoding as a single-model, jointly optimized alternative.

Future Modeling

  • Future-modeling approaches include predicted future properties, target foresight, deliberation networks, and recurrent layers for translated and untranslated content.Source: §5 · p. 12
  • Twin networks align forward and backward hidden states, but use two decoders and do not use the backward network during inference.Source: §5 · p. 12

Bidirectional Decoding

  • Earlier bidirectional NMT methods use agreement, reranking, or asynchronous backward decoding; SB-NMT instead combines both directions in one parallel, end-to-end jointly optimized model.Source: §5 · p. 12

6 Conclusions and Future Work

The paper concludes that SB-NMT performs simultaneous, interactive decoding in both directions to use history and future information. It reports state-of-the-art BLEU scores on two tasks and proposes further applications and training-cost reduction as future work.

Conclusions

  • SB-NMT uses simultaneous, interactive left-to-right and right-to-left decoding to leverage history and future information.Source: §6 · p. 13
  • The model establishes state-of-the-art BLEU scores of 51.11 on NIST Chinese-English and 29.21 on WMT14 English-German.Source: §6 · p. 13

Future Work

  • Future work proposes applying the framework to sequence labeling, abstractive summarization, and image captioning, and reducing training cost with target noise and fine-tuning.Source: §6 · p. 13
Loading 1905.04847v1…