Source-linked AI summary
Document-Level Neural Machine Translation with Hierarchical Attention Networks
Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, James Henderson
TL;DR
Document-level NMT can benefit from contextual information that sentence-level translation omits. The paper integrates hierarchical attention into NMT to dynamically use source and target context, and reports improvements over strong baselines with complementary encoder and decoder contributions.
Problem
Sentence-level NMT ignores document context, potentially degrading the coherence and cohesion of translated documents.
Method
The model integrates hierarchical multi-head attention into NMT, reusing hidden representations from previous source and target sentences and dynamically attending over contextual words and sentences.
Results
The model significantly outperforms two competitive baselines and improves lexical cohesion and coherence as well as noun and pronoun translation.
Takeaways & Limitations
Source-side and target-side context are complementary, with the model identifying important previous sentences and words for correct prediction.
Abstract
from arXiv · showhide
Neural Machine Translation (NMT) can be improved by including document-level contextual information. For this purpose, we propose a hierarchical attention model to capture the context in a structured and dynamic manner. The model is integrated in the original NMT architecture as another level of abstraction, conditioning on the NMT model's own previous hidden states. Experiments show that hierarchical attention significantly improves the BLEU score over a strong NMT baseline with the state-of-the-art in context-aware methods, and that both the encoder and decoder benefit from context in complementary ways.
1 Introduction
The paper introduces hierarchical attention for document-level NMT, modeling context dynamically at word and sentence levels while integrating source and target-side representations into the translation architecture.
- NMT typically translates sentences independently, ignoring document context that can affect translated-document coherence and cohesion.
- The proposed HAN models contextual information through word-level and sentence-level abstractions, dynamically focusing on relevant words and sentences.
- Two HAN components provide complementary context: the encoder supports source-word disambiguation, while the decoder supports target-side lexical cohesion and coherence.
- The model reuses hidden representations from previous sentence translations and feeds them into the current encoder and decoder, enabling joint multi-sentence optimization.
- Experiments report significant improvement over two strong baselines across multiple datasets, with source and target contextual information contributing complementarily.
2 The Proposed Approach
The proposed approach adds source- and target-side dependencies to sentence-level NMT through hierarchical attention over previous sentences. Word- and sentence-level abstractions dynamically provide document context, while gating regulates its combination with the current hidden state.
- 2 The Proposed Approach: The model conditions translation on the previous k source and target sentences, whose contexts are modeled with HANs.Dxn and Dyn represent the previous k sentences on the source and target sides, respectively.
- 2.1 Hierarchical Attention Network: HAN uses word-level abstraction to summarize each previous sentence and sentence-level abstraction to summarize the required contextual information.Word attention produces sentence vectors, followed by sentence-level attention over those representations.
- 2.3 Integrated Model: The encoder and decoder use similar HAN integrations, with the encoder producing a context-aware hidden state for the current word.The decoder applies an analogous architecture during target-side processing.
- 2.2 Context Gating: A gate regulates sentence-level and document-level information because different words require different amounts of context.The resulting context-aware hidden representation replaces the original hidden state during final classification.
- 2.3 Integrated Model: Attention queries derive from the NMT hidden state, while values can come from previous encoded source states, decoded target states, or alignment vectors.The experiments include one encoding configuration, three decoding configurations, and one combined configuration.
3 Experimental Setup
The evaluation tests Chinese-to-English and Spanish-to-English translation across talks, subtitles, and news. It compares transformer NMT systems, including a cache-based context-aware baseline, using BLEU and paired bootstrap significance testing.
- 3.1 Datasets and Evaluation Metrics: Experiments cover Chinese-to-English and Spanish-to-English translation in talks, subtitles, and news domains.The corpora include TED Talks, subtitles, and News-Commentaries data.
- 3.1 Datasets and Evaluation Metrics: Evaluation uses BLEU on tokenized text, with significance measured by paired bootstrap resampling.The metric and significance procedure are applied to the translation results.
- 3.2 Model Configuration and Training: Baselines include a transformer NMT model and a cache-memory context-aware transformer modeled after Tu et al. (2018).The cache baseline uses a memory size of 25 words.
- 3.2 Model Configuration and Training: Models use the transformer base configuration and are trained first without HAN, then with the full network optimized.The reported setup uses k = 3 previous sentences, selected for best performance in the described configuration.
4 Experimental Results
Across translation, pronoun and noun accuracy, cohesion, coherence, and qualitative analysis, HAN improves document-level translation by exploiting source and target context. The strongest results come from combining encoder and decoder context, although the useful context size and component contributions vary by dataset and metric.
- Translation Performance: The combined HAN encoder and decoder achieves the best BLEU scores and significantly outperforms the cache method on all datasets without compromising training speed.Training speed is 2.3K versus 2.6K tok/sec.
- Translation Performance: HAN encoder context provides a substantial share of the BLEU improvement, while combining encoder and decoder context yields further gains from complementary information.The source side always contains correct information, whereas target-side context may include erroneous test-time predictions.
- Context Size: The best context size is three previous sentences for TED talks and news, while subtitles perform similarly with three or seven.This comparison uses the combined HAN encoder and decoder model.
- Pronoun and Noun Translation: The joint HAN achieves the best noun accuracy, and it also performs best for pronouns on TED talks and news.For subtitles, the HAN encoder alone is better because the decoder can repeat previously translated personal pronouns.
- Cohesion and Coherence: HAN improves lexical cohesion, but its scores remain far below the human reference, leaving room for improvement.Higher cohesion can sometimes result from more repetitions, which do not always improve translation quality.
- Cohesion and Coherence: The joint HAN consistently obtains the best coherence score, with most of the improvement coming from the HAN decoder.Its coherence scores are close to those of the other HAN models.
- Qualitative Analysis: A qualitative example shows HAN resolving Spanish “su” as English “his” by attending to a prior “his” and its source-side antecedent “Nathaniel”.The attention visualization indicates interpretable inter-sentence connections.
5 Related Work
Document-level NMT methods incorporate preceding context through concatenation, context encoders, recurrent summaries, or cache memories. The paper evaluates discourse phenomena and illustrates HAN’s context-based pronoun disambiguation.
- Evaluation: 37.15 BLEU is reported for HAN, compared with 36.52 for cache and 35.99 for the NMT transformer.These values are reported for NIST BLEU.
- Evaluation: Table 4 evaluates noun and pronoun translation accuracy, lexical cohesion, and coherence across discourse-focused dimensions.Lexical cohesion measures repeated or lexically similar content words, while coherence averages cosine similarity between consecutive sentences.
- Document-level NMT: Document-level NMT methods add preceding sentences or translated-word representations through concatenation, context encoders, recurrent summaries, and cache memories.Cache memories retain word representations without preserving the sentences where those words occurred.
6 Conclusion
The paper concludes that hierarchical multi-head HAN NMT captures inter-sentence connections by integrating source and target context into sentence translation. It reports gains across discourse-related measures and identifies annotated discourse modeling as future work.
- 6 Conclusion: The hierarchical multi-head HAN NMT model connects representations from previous source and target sentence translations to the current translation.This design captures inter-sentence connections through both contextual sides.
- 6 Conclusion: The model significantly outperforms two competitive baselines, while ablations show that target and source context are complementary.The conclusion attributes complementary contributions to the two context sources.
- 6 Conclusion: HAN improves lexical cohesion, coherence, and the translation of nouns and pronouns.The conclusion summarizes improvements across these discourse-related outcomes.
- 6 Conclusion: Future work will explicitly model discourse connections using annotated data to potentially improve translation quality.The proposed direction depends on annotated discourse information.