Asynchronous Bidirectional Decoding for Neural Machine Translation

Introduction

• Conventional recurrent NMT decoders generate left to right and therefore cannot capture reverse target-side context, which can provide complementary signals.The introduction gives an example in which right-to-left decoding accurately translates a latter half misinterpreted by conventional NMT; Table 1 identifies L2R and R2L translations and marks incorrect and correct text.Source: Introduction · p. 1, Table 1 · p. 1
• Earlier bidirectional methods often reranked candidates, but limited beam-search outputs and independent decoders restricted the useful reverse-context information.The paper describes 1-best translations as often unsatisfactory and says independent decoders cannot fully exploit each other's target-side contexts.Source: Introduction · pp. 1–2
• The proposed model adds a right-to-left backward decoder whose hidden states encode reverse target-side contexts for a left-to-right forward decoder with two attention models.The forward decoder attends to both source-side and reverse target-side states during translation prediction.Source: Introduction · p. 2
• The paper presents its approach as an end-to-end attentional NMT model that introduces a backward decoder to supplement conventional target-side context modeling.It reports experiments on Chinese-English and English-German translation showing significant improvements over conventional NMT.Source: Introduction · p. 2

Our Model

• The model has three parameterized components: an encoder, a backward decoder, and a forward decoder.Source: Our Model · p. 2

The Neural Encoder

• The encoder uses a bidirectional RNN whose concatenated forward and backward hidden states form source annotations for both decoders.Each annotation represents a source word with respect to its surrounding words.Source: Our Model · p. 2

The Neural Backward Decoder

• The backward decoder predicts target words right-to-left from source-side hidden states and previously generated words, producing reverse target-side hidden states for the forward decoder.Its attention forms a weighted sum of source annotations, allowing it to select source words while predicting in reverse.Source: Our Model · pp. 2–3

The Neural Forward Decoder

• At each step, the forward decoder predicts the next target word from previous target words and source-side and reverse target-side context vectors.The model uses GRUs, though the authors state it is also applicable to other RNNs such as LSTMs.Source: Our Model · p. 2, The Neural Forward Decoder · p. 3
• The forward decoder attends separately to source annotations and backward-decoder hidden states, using the hidden states rather than the generated reverse word sequence.It also uses the first reverse encoder hidden state to initialize its first hidden state.Source: The Neural Forward Decoder · p. 3, Figure 1 · p. 3

Training and Testing

• Training balances forward-translation likelihood with reverse-translation likelihood using a hyperparameter, while inference first generates reverse hidden states greedily and then beam-searches the forward decoder.During training, beam search generates reverse hidden states when optimizing forward likelihood, and a reverse likelihood term is also optimized; the authors use greedy right-to-left decoding because beam search is costly.Source: The Neural Forward Decoder · p. 4

Results on Chinese-English Translation

• The proposed model has about 130.0M parameters and is described as relatively slow, processing about 1,758 mini-batches in one hour during training.Source: Results on Chinese-English Translation · p. 5
• 7.38, 3.14, 3.26, 1.86, 2.34, and 1.92 BLEU points are the respective gains over Moses, RNNSearch, RNNSearch(R2L), ATNMT, NSC(RT), and NSC(HS) on Chinese–English.The evaluation uses case-insensitive BLEU scores with λ=0.7.Source: Results on Chinese-English Translation · p. 5, Table 2 · p. 5
• The model outperforms ATNMT on all test sets; the authors attribute this to richer reverse target-side contexts and finer-grained refinement through attention.ATNMT uses k-best hypothesis rescoring, while this model attends to reverse hidden states.Source: Results on Chinese-English Translation · p. 6
• The model also outperforms its RR variant, supporting the authors’ claim that greedy-search translations make training and testing more consistent.The paper reports this comparison as evidence for the consistency of the training and testing procedure.Source: Results on Chinese-English Translation · p. 6
• Across every source-sentence length group, the proposed model achieves the best performance, while all systems’ performance drops as sentence length increases.Figure 3 presents BLEU scores for groups divided by source sentence length.Source: Results on Chinese-English Translation · p. 6, Figure 3 · p. 6

Results on English-German Translation

• The proposed model significantly outperforms the compared systems on English–German, where the BLEU gaps are smaller than on Chinese–English.The paper attributes the smaller gaps to a single reference translation for English–German and the closer relation of German to English than Chinese to English.Source: Results on Chinese-English Translation · p. 6, Table 4 · p. 6

• The paper frames exploiting bidirectional decoding to refine translation as its focus.It contrasts this aim with prior work on combining NMT and SMT or neural post-editing.Source: Related Work · pp. 6–7
• SMT research explored right-to-left decoding, merging partial hypotheses from both directions, and backward language models.Finch and Sumita (2009) integrated both monodirectional approaches and a backward language model into reverse translation decoding.Source: Related Work · pp. 6–7
• Prior NMT work jointly trained directional LSTMs for testing-time search, reranked left-to-right outputs, or used continuous optimization for bidirectional inference.The discussion also relates this work to pre-translation and neural automatic post-editing because both involve two translation stages.Source: Related Work · p. 7
• The model attends to right-to-left hidden states within end-to-end NMT, unlike prior approaches using raw machine-translation outputs or independent bidirectional components.The authors also distinguish their goal—improving left-to-right decoding—from prior work using bidirectional scores to produce translations.Source: Related Work · p. 7

• The backward decoder generates reverse target-side hidden states, which the forward decoder attends to alongside encoder states during translation.The two attention mechanisms are independent in the presented model.Source: Conclusions and Future Work · p. 7
• Results on Chinese-English and English-German translation tasks demonstrate the model's effectiveness.Source: Conclusions and Future Work · p. 8
• The encoder and backward-decoder attention mechanisms are independent; closer coupling is proposed for future refinement.The authors also plan to investigate the approach on other tasks involving RNN-based decoder modeling, such as image captioning.Source: Conclusions and Future Work · p. 8