PaperlayerView exact revision on arXiv

Source-linked AI summary

Tree-to-Sequence Attentional Neural Machine Translation

Akiko Eriguchi, Kazuma Hashimoto, Yoshimasa Tsuruoka

arXiv:1603.06075v3cs.CL

TL;DR

Existing NMT models primarily process sequential data and did not directly use source syntax, a challenge for structurally distant English-to-Japanese translation. The paper extends attentional sequence-to-sequence NMT with a bottom-up tree-based encoder and phrase-aware attention. On WAT’15 English-to-Japanese, the model achieved state-of-the-art accuracy, the best RIBES score, and outperformed sequential attentional NMT.

  • Problem

    Existing NMT models focus on sequential data and do not directly incorporate syntactic information, while competitiveness with SMT for structurally distant English-to-Japanese remained unresolved.

  • Method

    The model recursively encodes source phrase structure bottom-up and uses attention to align target words with both source phrases and words.

  • Results

    The model achieved state-of-the-art translation accuracy, the best RIBES score, and outperformed the sequential attentional NMT model on WAT’15 English-to-Japanese.

  • Takeaways & Limitations

    Source-side phrase structure can be integrated into end-to-end attentional NMT for phrase- and word-level alignment in English-to-Japanese translation.

  • Takeaways & Limitations

    The model’s reported scope is the WAT’15 English-to-Japanese translation task, and the paper notes that BLEU penalizes some synonym translations that may not be wrong.

Abstract

from arXiv · show

Most of the existing Neural Machine Translation (NMT) models focus on the conversion of sequential data and do not directly use syntactic information. We propose a novel end-to-end syntactic NMT model, extending a sequence-to-sequence model with the source-side phrase structure. Our model has an attention mechanism that enables the decoder to generate a translated word while softly aligning it with phrases as well as words of the source sentence. Experimental results on the WAT'15 English-to-Japanese dataset demonstrate that our proposed model considerably outperforms sequence-to-sequence attentional NMT models and compares favorably with the state-of-the-art tree-to-string SMT system.

1 Introduction

NMT had not directly incorporated source syntax, limiting its established attention-based approach for structurally distant English-to-Japanese translation. The paper proposes a tree-to-sequence attentional model that encodes phrase structure and aligns target words with source phrases and words.

  • 1 Introduction: Structurally distant English-to-Japanese translation remained an open test of whether NMT could compete with traditional SMT.Existing NMT results were strong for English-to-French and English-to-German, but competitiveness for English-to-Japanese was unresolved.
  • 1 Introduction: English and Japanese use different syntactic constructions and lexical units, creating word-level alignment mismatches.For example, Japanese “緑茶” aligns with “green” and “tea,” while “a cup of” aligns with a null symbol.
  • 1 Introduction: Existing NMT models did not support the phrase-level alignments that can address these mismatches.The paper motivates aligning “a cup of green tea” with “緑茶” rather than aligning only individual words.
  • 1 Introduction: The proposed model recursively encodes source sentences bottom-up according to phrase structure and decodes while aligning input phrases and words with output words.This extends attentional sequence-to-sequence translation with source-side syntactic information.
  • 1 Introduction: On the WAT’15 English-to-Japanese task, the proposed model achieved state-of-the-art translation accuracy.The supplied passage reports this as an experimental outcome without giving a metric value.

2 Neural Machine Translation

Neural machine translation encodes source sequences and generates target sequences with recurrent encoder-decoder models. Attention adds soft alignment between decoder states and encoder states, while training maximizes translation-pair likelihood.

  • 2.1 Encoder-Decoder Model: NMT directly estimates p(y|x) from parallel sentence pairs using separate encoder and decoder processes.The encoder embeds source words into d-dimensional vectors, and the decoder generates the target word sequence.
  • 2.1 Encoder-Decoder Model: The recurrent encoder updates each hidden unit from the current source word and previous hidden unit until the complete source sequence is represented.The final hidden unit is treated as a vector representing the input sequence’s meaning.
  • 2.1 Encoder-Decoder Model: The decoder is initialized with the encoded sentence vector and generates target words conditionally from prior target words and decoder states.The decoder uses nonlinear transformations to calculate hidden states and target-word probabilities.
  • 2.1 Encoder-Decoder Model: LSTM units replace vanilla RNN units, using gates and memory states to process sequential information.The passages identify input, forget, and output gates plus a memory-cell state.
  • 2.2 Attentional Encoder-Decoder Model: Attention softly aligns each decoder state with encoder states and quantifies each encoder state’s contribution to predicting a target word.The attention score uses the similarity between source hidden unit h_i and target hidden unit s_j.
  • 2.2 Attentional Encoder-Decoder Model: The attention-weighted context vector is combined with the decoder state before softmax prediction over the target vocabulary.The model concatenates the decoder state and context vector, applies an additional hidden layer, and predicts the next word.
  • 2.3 Objective Function of NMT Models: Training maximizes the sum of translation-pair log-likelihoods, with parameters learned using stochastic gradient descent.The objective is defined over a set of parallel sentence pairs.

3 Attentional Tree-to-Sequence Model

The proposed tree-to-sequence attentional NMT model incorporates source phrase structure alongside sequential representations. It uses phrase-aware attention and a sequential fallback, while BlackOut reduces softmax training cost for tree-structured data.

  • Tree-based Encoder + Sequential Encoder: The model addresses sequence-based NMT's neglect of sentence structure with a tree-based encoder over source phrase structure.Phrase representations are constructed bottom-up, producing a sentence vector that contains structural information.
  • Tree-based Encoder + Sequential Encoder: Each phrase node is computed from child representations using Tree-LSTM, with sequential LSTM units initializing the leaf nodes.Tree-LSTM represents each binary-tree node, while sequentially contextualized leaves support context-sensitive phrase representations.
  • Initial Decoder Setting: The encoder combines the final sequential and tree-based units through another Tree-LSTM to initialize the decoder with both information sources.This initialization is designed to capture sequential data and phrase structure together.
  • Attention Mechanism in Our Model: Attention aligns each decoded target word with both sequential hidden units and phrase hidden units.The context vector weights word-level and phrase-level representations using attention scores.
  • Attention Mechanism in Our Model: Input-feeding supplies the previous attentional decoder unit to the current decoder state.The previous unit is concatenated with the previous decoder state and is reported to improve BLEU scores in prior work and preliminary experiments.
  • Training: BlackOut approximates the vocabulary softmax with weighted sampling to reduce training cost for the tree-structured NMT model.Negative samples are drawn from a unigram distribution raised to β, addressing the vocabulary-dependent softmax bottleneck.

4 Experiments

Experiments use ASPEC English-to-Japanese data with binary source phrase structures, CPU-based implementations, and dataset-specific training configurations. Decoding uses beam search with source-target length statistics, and models are evaluated with RIBES and BLEU.

  • 4.1 Training Data: The experiments use the first 1.5 million ASPEC English-to-Japanese training pairs, with Enju providing binary English phrase structures and KyTea segmenting Japanese.Enju's HPSG-specific information is not used beyond the binary phrase structure.
  • 4.1 Training Data: Vocabulary entries must appear at least 2 times for the small dataset or 5 times for the large dataset; other words map to “unk”.An “eos” symbol is appended to every sentence in both languages.
  • 4.2 Experimental Settings: The proposed and sequential attentional models use 256-dimensional hidden units and embeddings on the small dataset, with BlackOut negative samples K of 500 or 2000.
  • 4.2 Experimental Settings: On the large dataset, the proposed model uses 512-dimensional embeddings, hidden size d ∈{512, 768, 1024}, and BlackOut K = 2500.
  • 4.2 Experimental Settings: Experiments run on multicore CPUs, and training the large dataset with d = 512 takes about one week.The implementation uses C++ and Eigen.
  • 4.3 Decoding process: Beam search scores target sentences by summed log-likelihoods, while length-aware decoding incorporates conditional target-length probabilities given source length.The length statistics are computed from the first one million training pairs, and decoding permits up to 100 generated words.
  • 4.4 Evaluation: Models are evaluated with RIBES and BLEU using the KyTea-based evaluation script, following WAT’15.RIBES uses rank correlation with word precision, whereas BLEU uses n-gram precision and a brevity penalty.

5 Results and Discussion

Experiments show that the proposed tree-based attentional NMT benefits from BlackOut, syntactic structure, sequential leaf LSTMs, and the proposed beam search. On the large dataset, it outperforms sequential NMT baselines and compares favorably with tree-to-string SMT, while attention examples show phrase-level alignments.

  • Small Training Dataset: BlackOut results improve as the number of negative samples K increases, while softmax achieves better results but takes about three times longer per epoch.These findings are reported on the small development dataset.
  • Small Training Dataset: Reversing input word order decreases English-to-Japanese scores, whereas incorporating syntactic information improves scores over sequential attention.This differs from improvements reported for reversed inputs in other language pairs.
  • Small Training Dataset: The proposed beam search outperforms simple beam search in both RIBES and BLEU by retaining long-sentence candidates with a large beam size.BLEU is sensitive to beam size because brevity penalty has a relatively large impact.
  • Small Training Dataset: Sequential LSTMs at tree-encoder leaf nodes contribute to context-aware construction of phrase representations.The comparison evaluates the proposed encoder against a tree-based encoder without sequential LSTMs.
  • Large Training Dataset: The ensemble of models with d = 512, 768, and 1024 achieves the best RIBES score among all systems.The proposed models show similar performance across these values of d.
  • Large Training Dataset: The single proposed model with d = 512 outperforms Zhu’s best end-to-end NMT model by +1.19 RIBES and +0.17 BLEU.The ensemble also exceeds Zhu’s hybrid system and Lee et al.’s ANMT system on both reported metrics.
  • Large Training Dataset: The proposed end-to-end NMT model compares favorably with the WAT’15 tree-to-string SMT system enhanced by ANMT reranking.The comparison uses systems reported in the WAT’15 evaluation.
  • Qualitative Analysis: The model softly aligns Japanese target words with English source words and phrases, including phrase-level relations in short and long sentences.Examples include “示” aligned more strongly with “showed excellent performance” than with “showed” alone.

6 Related Work

Earlier NMT systems primarily use sequential encoders and attention mechanisms, while this work addresses syntactic structure through a tree-based encoder. Related efforts include sequence reversal, jointly learned source structures, and refinements of sequential attention.

  • Sequential NMT: Earlier end-to-end NMT models use CNNs or RNNs as sequential source encoders with RNN target decoders.The Encoder-Decoder model replaced CNN encoders with GRU- or LSTM-based RNN encoders.
  • Sequential NMT: Reversing input sequences improved French-to-English and other European-language translation tasks, but these models remained based on sequential encoders.The paper contrasts this prior pattern with its structural approach.
  • Attentional NMT: Attention mechanisms enable target-source alignment, while later work refined attention to focus dynamically on local windows and improve attentional paths.Several subsequent ANMT models still build on sequential attentional models.
  • Syntactic NMT: The proposed work differs from prior attentional models by explicitly incorporating syntactic information through source-side phrase structure.Its tree-based encoder constructs representations from phrase structure rather than only sequential data.

7 Conclusion

The proposed syntactic attentional NMT model extends sequential NMT with a tree-based encoder that represents source phrase structure and aligns phrases as well as words. On WAT’15 English-to-Japanese translation, it achieves the best RIBES score and outperforms sequential attentional NMT.

  • The tree-based encoder follows the parsed source tree and combines tree-LSTM leaf units with the original sequential LSTM encoder.This extends the sequential encoder while incorporating source-side phrase structure.
  • Its attention mechanism aligns input phrases and words with output words.The decoder can use both word-level and phrase-level source representations during translation.
  • The model achieves the best RIBES score on the WAT’15 English-to-Japanese translation dataset.
  • The proposed model outperforms the sequential attentional NMT model.
Loading 1603.06075v3…