Source-linked AI summary
Evaluating Discourse Phenomena in Neural Machine Translation
Rachel Bawden, Rico Sennrich, Alexandra Birch, Barry Haddow
TL;DR
Discourse-dependent translation requires extra-sentential context, yet standard MT metrics are poorly suited to evaluating such phenomena. The paper introduces targeted discourse test sets and contextual NMT strategies, finding that its novel combination outperforms existing models under targeted evaluation and BLEU.
Problem
MT systems often translate sentences independently, while phenomena such as coreference and lexical cohesion require context beyond the current sentence; standard metrics may overlook relevant improvements.
Method
The paper evaluates contextual NMT using hand-crafted English-to-French test sets for coreference and coherence/cohesion, comparing single- and multi-encoder strategies with previous-sentence context.
Results
The novel combination of multiple encoders with decoding of both previous and current sentences outperforms existing contextual models under targeted evaluation and BLEU.
Takeaways & Limitations
Targeted discourse evaluation reveals limited capacity in multi-encoder architectures alone, while decoding previous and current sentences together is highly effective for exploiting context.
Takeaways & Limitations
The evaluation assumes that disambiguating context is in the previous sentence, although relevant context is not guaranteed to occur there.
Abstract
from arXiv · showhide
For machine translation to tackle discourse phenomena, models must have access to extra-sentential linguistic context. There has been recent interest in modelling context in neural machine translation (NMT), but models have been principally evaluated with standard automatic metrics, poorly adapted to evaluating discourse phenomena. In this article, we present hand-crafted, discourse test sets, designed to test the models' ability to exploit previous source and target sentences. We investigate the performance of recently proposed multi-encoder NMT models trained on subtitles for English to French. We also explore a novel way of exploiting context from the previous sentence. Despite gains using BLEU, multi-encoder models give limited improvement in the handling of discourse phenomena: 50% accuracy on our coreference test set and 53.5% for coherence/cohesion (compared to a non-contextual baseline of 50%). A simple strategy of decoding the concatenation of the previous and current sentence leads to good performance, and our novel strategy of multi-encoding and decoding of two sentences leads to the best performance (72.5% for coreference and 57% for coherence/cohesion), highlighting the importance of target-side context.
1 Introduction
Discourse-dependent translation choices can require source or target context beyond the current sentence, but standard MT evaluation often misses these phenomena. The paper introduces targeted discourse test sets and contextual NMT strategies for English-to-French translation.
- Motivation: Coreference, lexical cohesion, and lexical disambiguation can require linguistic context outside the current sentence.The relevant context may be source-side or target-side, and can determine French gender, repetition, alignment, or word choice.
- Evaluation gap: Standard automatic metrics are poorly adapted to evaluating discourse phenomena, motivating targeted evaluation.The paper focuses on hand-crafted tests that directly assess models’ use of discourse context.
- Contribution: The paper presents two hand-crafted discourse test sets for coreference and coherence/cohesion in English-to-French translation.The test sets assess whether models exploit previous source and target sentences.
- Contribution: The study reviews contextual NMT strategies trained on subtitles and proposes a novel contextual model.The models are evaluated using targeted discourse tests alongside BLEU.
- Headline result: The authors report that their new combination of strategies outperforms previous methods on targeted evaluation and BLEU.The supplied passage states the overall comparison but does not provide the corresponding numerical scores.
2 Evaluating contextual phenomena
The paper develops controlled contrastive tests for evaluating whether MT models use previous source and target context to resolve discourse-dependent translations. These tests target coreference and coherence/cohesion while addressing limitations of standard metrics and costly manual evaluation.
- Evaluation motivation: Standard metrics can overlook discourse improvements because relevant phenomena affect few word forms or depend on target-side coherence.A correct translation may differ from a reference while remaining coherent in context.
- Test-set design: The test sets rank correct and incorrect translations of ambiguous current sentences using previous source and/or target context.Examples are hand-crafted but inspired by OpenSubtitles2016.
- Novelty: The tests differ from prior suites by requiring previous context, excluding current-sentence antecedents, and enabling fully automatic evaluation.This design directly probes discourse-level context use rather than relying on fixed antecedents or one-off manual analysis.
- Scope: The authors first test whether models can use previous-sentence context before investigating longer-distance context.They acknowledge that real disambiguating context is not guaranteed to occur in the immediately preceding sentence.
- Coreference test set: The coreference set contains 50 blocks testing whether pronoun gender agrees with translated antecedents, including semi-correct antecedent translations.Each block has four contrastive translation pairs, with feminine and masculine antecedent translations.
- Evaluation setup: The test sets deliberately use controlled two-alternative cases, guaranteeing 50% accuracy for a non-contextual baseline.The coreference design restricts antecedents and pronouns to maintain this controlled comparison.
- Coherence/cohesion test set: The coherence/cohesion set contains 100 blocks with two contrastive pairs whose correct translations depend on previous-sentence context.The context can occur on the source side, target side, or both, and blocks require both pairs to be ranked correctly.
3 Contextual NMT Models
The paper compares single-encoder and multi-encoder NMT strategies for incorporating preceding linguistic context, including a novel model that encodes and decodes two sentences.
- Motivation: These models address discourse phenomena whose correct translation may depend on context outside the current sentence.Relevant phenomena include coreference, lexical cohesion, and lexical disambiguation.
- 3.1 Single-encoder models: 2-TO-2 translates concatenated previous and current source-target sentences, retaining only the current sentence’s output.2-TO-1 concatenates only source sentences and directly produces the current translation, enabling comparison of decoder-side contextualization.
- 3.2 Multi-encoder models: Multi-encoder models separately encode contextual inputs and combine their context vectors for decoding.The studied combination strategies are vector concatenation, an attention gate, and hierarchical attention.
- 3.2 Multi-encoder models: An attention gate learns differing importance for elements of the two context vectors.Hierarchical attention instead assigns a weight to each encoder’s context vector.
- 3.3 Novel strategy of hierarchical attention and context decoding: The novel strategy separately encodes previous and current sentences, combines them with hierarchical attention, and decodes both target sentences.Only the second, current-sentence portion of the concatenated target output is retained.
4 Experiments
Experiments test contextual inputs from previous source and target sentences using subtitle data and common NMT training settings.
- Contextual inputs: Multi-encoder strategies use previous source sentences, previous target sentences, or both as additional inputs.The prefixes S-, T-, and S-T- identify these contextual-input configurations.
- Data: Models are trained and tested on fan-produced parallel subtitles from OpenSubtitles2016.The data is corrected, tokenized, cleaned, truecased, and segmented into subword units with BPE.
- Data: The high-resource training set contains approximately 29M parallel sentences, with approximately 55k English and 60k French vocabulary items.Subtitles are retained when they contain no more than 80 tokens.
- Training: All models use attention-based sequence-to-sequence architectures implemented in Nematus and trained with Adam until convergence.Shared settings include a 0.0001 learning rate, 512-dimensional embeddings, and 1024-dimensional hidden layers.
- Evaluation: Models using previous target sentences are trained with reference translations but use baseline translations during translation.For targeted evaluation, the scored translations are given, so this distinction does not apply.
5 Results and Analysis
Targeted discourse evaluation reveals that contextual NMT strategies vary substantially: source-side and decoder-based target context help, while auxiliary target encoding often performs poorly. The novel S-HIER-TO-2 model achieves the strongest reported discourse results, although gains depend on the phenomenon and test set.
- Overall performance: BLEU scores vary dramatically across comedy, crime, fantasy, and horror, and the best model differs by genre.The models are ensembled and evaluated on four test sets, each containing three films from a genre.
- Overall performance: The S-HIER-TO-2 model exceeds the baseline BLEU by more than +1 on almost all test sets.No consistent second-best model emerges because performance depends strongly on the test set.
- Overall performance: Previous target-sentence input considerably degrades overall performance, and reference translations at decoding time do not significantly improve it.The result is therefore unlikely to reflect overfitting during training.
- Coreference: Multi-encoder models achieve at best random accuracy on coreference, matching the non-contextual baseline.Models using only the previous source sentence perform poorly, while previous-target models also perform poorly, possibly because target context is not encoded sufficiently.
- Coherence and cohesion: 53.5% coherence/cohesion accuracy is reached by S-CONCAT, while S-HIER-TO-2 raises accuracy to 57%.The coherence/cohesion examples show less score variation, suggesting greater difficulty and room for improvement.
- Coherence and cohesion: Previous target context performs near randomly, whereas decoding previous and current sentences with multi-encoding improves discourse handling.Adding previous target input as an auxiliary encoder degrades the result, while decoder-side target context is more effective.
- Context use: The results suggest that target context is exploited more effectively through the recurrent decoder than through auxiliary input encoding.Attention analysis does not show increased attention between translated pronouns and their source antecedents.
6 Conclusion
The conclusion finds that multi-encoder architectures alone have limited ability to use discourse context, while a novel strategy substantially outperforms existing models. Its effectiveness suggests stream decoding as a promising direction for future research.
- Multi-encoder architectures alone have limited capacity to exploit discourse-level context, with poor coreference results and more promising coherence/cohesion results.
- The novel combination of contextual strategies greatly outperforms existing models.
- The strategy uses the previous source sentence as an auxiliary input and decodes both the current and previous sentence.
- Its effectiveness suggests that stream decoding and maintaining a constant flow of contextual information in the decoder could be promising for future research.