Source-linked AI summary
Learning to Parse and Translate Improves Neural Machine Translation
Akiko Eriguchi, Yoshimasa Tsuruoka, Kyunghyun Cho
TL;DR
Neural machine translation has used relatively little explicit linguistic knowledge, while prior linguistic approaches have often been limited to the source side or required external inference-time tools. The paper introduces NMT+RNNG, which jointly learns translation and parsing by combining attention-based NMT with an RNNG; experiments across four language pairs show its effectiveness.
Problem
Explicit incorporation of linguistic prior into NMT has received limited attention, and earlier approaches can require external tools during inference.
Method
NMT+RNNG combines an attention-based neural translation decoder with an RNNG and jointly trains translation and parse-action probabilities.
Results
Significant BLEU improvements occur on three of four language pairs, while RIBES improves significantly on all four without added inference-time parameters or computational overhead.
Takeaways & Limitations
The hybrid model incorporates linguistic priors during training while allowing translation without an external parser at test time.
Abstract
from arXiv · showhide
There has been relatively little attention to incorporating linguistic prior to neural machine translation. Much of the previous work was further constrained to considering linguistic prior on the source side. In this paper, we propose a hybrid model, called NMT+RNNG, that learns to parse and translate by combining the recurrent neural network grammar into the attention-based neural machine translation. Our approach encourages the neural machine translation model to incorporate linguistic prior during training, and lets it translate on its own afterward. Extensive experiments with four language pairs show the effectiveness of the proposed NMT+RNNG.
1 Introduction
NMT often succeeds with little explicit linguistic knowledge, motivating methods that incorporate linguistic priors without requiring external tools during inference. NMT+RNNG jointly learns translation and parsing, improving evaluation scores across four language pairs.
- 1 Introduction: NMT has achieved strong results using flat character sequences or statistically motivated subword units rather than extensive linguistic knowledge.The encoder has also been observed to capture syntactic properties of source sentences.
- 1 Introduction: Earlier explicit approaches incorporate source-side linguistic features or structures but require external tools during inference.These approaches include word annotations, tree-based encoding, or lattice-guided decoding.
- 1 Introduction: NMT+RNNG jointly learns to translate and parse by combining an attention-based translation decoder with an RNNG.The model shares parameters and is trained using both target-sentence and parse-action probabilities.
- 1 Introduction: Significant improvements occur in BLEU for three of four language pairs and in RIBES for all four evaluated pairs.The evaluated pairs are Japanese–English, Czech–English, German–English, and Russian–English.
2 Neural Machine Translation
Attention-based NMT encodes a source sequence into hidden states and generates the target sentence with a conditional recurrent decoder. Attention computes a time-dependent context vector from weighted encoder states.
- 2 Neural Machine Translation: The bidirectional encoder maps source words to a sequence of hidden states used by the decoder.Each hidden state concatenates outputs from forward and backward recurrent networks.
- 2 Neural Machine Translation: The source sentence is represented as a word sequence x = (x1, x2, . . . , xN), which the encoder reads recurrently.The encoder returns one hidden state for each source position.
- 2 Neural Machine Translation: The decoder models the translation as a conditional recurrent language model over target words.Its recurrent activation can use architectures such as LSTM or GRU.
- 2 Neural Machine Translation: At each target step, attention compares the decoder state with encoder states, normalizes the scores, and forms a weighted-sum context vector.The resulting context vector is time-dependent and conditions target-word prediction.
3 Recurrent Neural Network Grammars
RNNG is a syntax-based language model that jointly represents words and tree composition through buffer, stack, and action-history state. In this paper, replacing its buffer with a recurrent language model enables generation while parsing actions control the process.
- 3 Recurrent Neural Network Grammars: RNNG predicts parsing actions from recurrent states of a buffer, stack, and action-history stack LSTM.Its actions jointly model tokens and tree-based composition.
- 3 Recurrent Neural Network Grammars: Shift moves a buffered word to the stack, reduce combines the top two stack words, and non-terminal actions push symbols onto the stack.These transitions construct the parse incrementally.
- 3 Recurrent Neural Network Grammars: The stack representation is updated recursively from parent and dependent phrase vectors until a complete parse tree is built.The paper uses dependency trees represented through transition-based parsing actions.
- 3 Recurrent Neural Network Grammars: Replacing the RNNG buffer with a recurrent language model lets the system summarize shifted words and generate future words when shift is selected.This buffer replacement is the basis for the proposed hybrid decoder.
4 Learning to Parse and Translate
NMT+RNNG hybridizes an attention-based translation decoder with an RNNG so the model jointly learns translation and target-side parsing, using automated parse annotations for training.
- Construction: The model replaces the RNNG buffer with the NMT decoder and uses the decoder's word prediction as the RNNG generator.The decoder is controlled by RNNG actions, while the action model conditions on the decoder's hidden state.
- Learning and Inference: Joint training maximizes the conditional probability of translation and parse actions, providing decoder and encoder supervision from target syntax.At inference, the RNNG stack and action LSTMs are discarded, allowing translation without the parsing components.
- Knowledge Distillation for Parsing: Because parallel corpora lack gold target parses, the authors automatically annotate target sentences with an external parser.SyntaxNet parses are converted into SHIFT, REDUCE-L, and REDUCE-R actions, with dependency labels attached to REDUCE actions.
- Knowledge Distillation for Parsing: The automated annotations provide noisy supervision through a teacher-like external parser, framing the procedure as knowledge distillation or distant supervision.The external parser supplies annotations for otherwise unlabeled target-side syntax.
5 Experiments
Experiments compare NMT+RNNG with a baseline across four language pairs using BLEU and RIBES, including component ablations and an example parsed translation.
- Experimental Setup: The evaluation covers Jp-En, Cs-En, De-En, and Ru-En, with test quality measured using BLEU and RIBES.The corpora include ASPEC for Japanese-English and News Commentary v8 for the other language pairs.
- Experimental Setup: Training the proposed JP-En model takes about 20 minutes per epoch versus about 15 minutes for the baseline in the reported implementation.
- Results and Analysis: NMT+RNNG significantly improves over the baseline on all reported cases except De-En measured by BLEU, without added inference parameters or computational overhead.Table 2 reports bootstrap significance testing with † marking p < 0.005.
- Ablation: The complete RNNG configuration achieves the best ablation performance, while removing the stack causes the largest degradation.The stack's importance also matches its reported role in parsing experiments.
- Generated Sentences with Parsed Actions: The proposed decoder generates a translation while the RNNG decoder predicts parsing actions, producing a dependency structure alongside the sentence.The example is mostly correct but includes an erroneous dependency relation between “The” and “transition.”
6 Conclusion
NMT+RNNG combines attention-based neural translation with an RNNG so the model learns to parse and translate simultaneously. Experiments across four language pairs confirmed its effectiveness, while parser-free training remains future work.
- NMT+RNNG combines an attention-based neural translation decoder with an RNNG to learn parsing and translation simultaneously.
- Experiments on four language pairs, translating from Japanese, Czech, German, and Russian into English, confirmed the model’s effectiveness.
- Training the RNNG without ground-truth parses could eliminate external parsers completely, but the paper leaves this possibility for future research.