Document-Level Machine Translation with Large Language Models

Longyue Wang; Chenyang Lyu; Tianbo Ji; Zhirui Zhang; Dian Yu; Shuming Shi; Zhaopeng Tu

Source-linked AI summary

Document-Level Machine Translation with Large Language Models

Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, Zhaopeng Tu

arXiv:2304.02210v2Apr 5, 2023cs.CLcs.AI

TL;DR

Document-level MT requires modelling discourse phenomena that sentence-level translation can miss. This paper evaluates LLMs through prompt studies, model comparisons, and discourse-knowledge probing across benchmarks, finding improved discourse-aware performance with GPT-4 while identifying ranking difficulties and evaluation limitations.

Problem
Existing MT work has largely focused on sentence-level translation, which can produce outputs lacking coherence and context, leaving document-level discourse modelling to be examined.
Method
The paper evaluates GPT-3.5 and GPT-4 on document-level MT through context-aware prompt experiments, comparisons with commercial and advanced MT systems, and contrastive probing with explanations.
Results
GPT-4 shows significantly improved discourse-aware performance and stronger probing ability, but struggles to discern subtle distinctions for ranking.
Takeaways & Limitations
LLMs show potential as a new paradigm for document-level translation, and the released benchmark, outputs, and annotations support future research.
Takeaways & Limitations
The conclusions come from a limited set of datasets, and the human-evaluation criteria still have room for improvement.

Abstract

from arXiv · show

Large language models (LLMs) such as ChatGPT can produce coherent, cohesive, relevant, and fluent answers for various natural language processing (NLP) tasks. Taking document-level machine translation (MT) as a testbed, this paper provides an in-depth evaluation of LLMs' ability on discourse modeling. The study focuses on three aspects: 1) Effects of Context-Aware Prompts, where we investigate the impact of different prompts on document-level translation quality and discourse phenomena; 2) Comparison of Translation Models, where we compare the translation performance of ChatGPT with commercial MT systems and advanced document-level MT methods; 3) Analysis of Discourse Modelling Abilities, where we further probe discourse knowledge encoded in LLMs and shed light on impacts of training techniques on discourse modeling. By evaluating on a number of benchmarks, we surprisingly find that LLMs have demonstrated superior performance and show potential to become a new paradigm for document-level translation: 1) leveraging their powerful long-text modeling capabilities, GPT-3.5 and GPT-4 outperform commercial MT systems in terms of human evaluation; 2) GPT-4 demonstrates a stronger ability for probing linguistic knowledge than GPT-3.5. This work highlights the challenges and opportunities of LLMs for MT, which we hope can inspire the future design and evaluation of LLMs.We release our data and annotations at https://github.com/longyuewangdcu/Document-MT-LLM.

1 Introduction

The paper examines whether LLMs can model document-level discourse in machine translation, where sentence-level systems may produce translations lacking coherence and context. It evaluates context-aware prompting, model comparisons, and discourse-knowledge probing across diverse benchmarks.

Sentence-level MT can produce translations that lack coherence and context, motivating document-level translation.
The study investigates document-level translation through context-aware prompts, advanced-model comparisons, and discourse-modelling analysis.The analysis also considers how code pretraining, supervised fine-tuning, and reinforcement learning from human feedback relate to discourse modelling, while noting confounding variables.
Experiments cover Chinese⇒English, English⇒German, and English⇒Russian across seven domains using automatic metrics and human evaluation.The domains are news, social, fiction, Q&A, TED, Europarl, and subtitle.
The empirical study reports superior LLM capabilities over advanced MT systems and methods on document-level translation, indicating potential for a new paradigm.
The paper introduces a probing benchmark for document-level translation quality and discourse knowledge, and releases instruction-based data, system outputs, and human annotations.

2 Experimental Step

The experiments evaluate document-level translation across diverse datasets, languages, and domains using automatic, discourse-specific, and human-centered measures. The study also addresses data contamination and defines targeted measures for terminology consistency and zero-pronoun translation.

Datasets and benchmarks: The benchmark suite covers three language pairs and seven domains, combining recent and established document-level translation datasets.The experiments include Chinese⇒English, English⇒German, and English⇒Russian across news, social, fiction, Q&A, TED, Europarl, and subtitle domains.
Datasets and benchmarks: Average document length serves as a proxy for discourse-modeling complexity, making mZPRT Fiction and IWSLT TED especially challenging.Longer documents require modeling more cohesive devices and discourse structure accurately.
Data contamination: The evaluation uses recent, public, and diverse test sets to reduce contamination risks, but publicly available benchmarks may still have entered LLM training data.The paper recommends consistently incorporating newer datasets when evaluating LLMs.
Evaluation methods: Evaluation combines sacreBLEU, TER, COMET, document-level BLEU, discourse-specific metrics, and human ratings.Document-level sacreBLEU matches n-grams across whole documents, while sentence-level outputs are split and aligned for evaluation.
Evaluation methods: CTT measures repeated terminology consistency, while AZPT measures the accuracy of translating zero pronouns that require discourse understanding.CTT rewards consistent translations of repeated terms within a document; AZPT uses a binary scorer for zero-pronoun translations.
Evaluation methods: The prompt ablation compares BLEU, d-BLEU, AZPT, and CTT against a sentence-level InstructGPT baseline without contextual chat history.The prompts and baseline are specified for Chinese⇒English datasets.

3 Effects of Context-Aware Prompts

The study tests whether context-aware prompting can activate ChatGPT’s document-level translation ability and improve discourse modeling. It compares three prompt designs, finding broadly stable performance but an advantage for prompts that maximize continuous document context.

3.1 Motivation: ChatGPT’s long-text handling and contextual recall motivate prompts designed to trigger document-level translation.The approach targets the limitations of sentence-by-sentence translation for coherence and context.
3.1 Motivation: Prompt engineering is treated as necessary for robust instruction interpretation and long-term dependency modeling.The study reports that its prompts are neutral and representative for use by other researchers.
3.1 Motivation: Table 4 compares commercial MT systems and LLM applications using automatic scores plus 0∼5 human ratings for general quality and discourse awareness.The human criteria include general quality and discourse-aware quality dimensions.
3.2 Comparison of Different Prompts: The study obtains candidate prompts from ChatGPT and refines them into three document-level designs processed within one chat box.Each prompt is supplied through conversational turns while the document is processed in order.
3.2 Comparison of Different Prompts: P1 translates sentences separately within one chat box, whereas P2 and P3 translate multiple continuous sentences in each conversational turn.P2 and P3 differ in whether sentential boundary tags are retained.
3.2 Comparison of Different Prompts: The prompt comparison evaluates Chinese⇒English translation on WMT2022 News and mZPRT Fiction using multiple automatic metrics.These test sets provide the experimental basis for comparing the three candidate prompts.
3.2 Comparison of Different Prompts: P3 achieves the best scores on most automatic metrics except COMET and performs best on terminology consistency and zero-pronoun accuracy.Overall differences among prompts are minor, but P3 is strongest on the reported discourse phenomena.
3.2 Comparison of Different Prompts: Longer contextual input can improve translation quality and discourse awareness, while ChatGPT may translate across sentential boundaries rather than preserving strict sentence alignment.The paper characterizes this boundary-crossing behavior as resembling human document translation.

4 Comparison of Translation Models

The paper compares ChatGPT with commercial MT systems and document-level NMT methods using automatic and human evaluations across document-level translation benchmarks. ChatGPT often performs strongly in human evaluation and against prior document-level methods, but results vary by domain and may be affected by training-data bias.

ChatGPT vs. Commercial Systems: The comparison uses automatic d-BLEU and human evaluations, including general and discourse-aware quality, against commercial translation products and document-level NMT methods.The evaluated systems include Google Translate, DeepL Translate, Tencent TranSmart, and five advanced document-level NMT models.
ChatGPT vs. Commercial Systems: Commercial MT systems generally outperform LLM-based systems on d-BLEU, except in the Q&A domain involving informal spoken language.The d-BLEU comparison reflects reference n-gram similarity rather than the full set of qualities considered by human evaluators.
ChatGPT vs. Commercial Systems: GPT-4 and GPT-3.5 perform significantly better than MT systems in human evaluation, despite commercial systems generally leading on d-BLEU.The paper suggests that human evaluation captures coherence, fluency, naturalness, and discourse-level information that may not correlate with d-BLEU.
ChatGPT vs. Document NMT Methods: ChatGPT outperforms MR-Doc2Doc⋆ on TED (EN-DE) and News (EN-DE), matches it on TED (ZH-EN), and performs poorly on Europarl (EN-DE).MR-Doc2Doc with extra model pre-training is the strongest previous model in the comparison, while ChatGPT is even worse than Sent2Sent on Europarl (EN-DE).
ChatGPT vs. Document NMT Methods: ChatGPT’s unstable outputs sometimes contain omissions and obvious copying behaviors, and its Europarl weakness may reflect domain distribution bias in training data.The paper also notes that commonly used benchmarks may have entered newer models’ training data, complicating evaluation.

5 Analysis of Large Language Models

The analysis probes discourse knowledge in LLMs through contrastive prediction, explanation quality, and training-technique comparisons. GPT-4 generally improves over GPT-3.5, while supervised fine-tuning, code pretraining, and PPO are associated with stronger translation and discourse performance.

Probing Discourse Knowledge: The study probes discourse knowledge through contrastive prediction and explanation, while examining how training techniques affect discourse modeling.The contrastive evaluation covers deixis, lexical consistency, and two types of ellipsis in English⇒Russian translation.
Prediction: GPT-3.5 performs worse than DocRepair across all tested discourse phenomena, whereas GPT-4 substantially improves but still trails DocRepair on several phenomena.GPT-4 remains behind DocRepair in deixis, lexical consistency, and ellipsis (inflection).
Explanation: Explanation accuracy often fails to reflect prediction accuracy, but GPT-4 explains discourse knowledge better than GPT-3.5 and also predicts more accurately.Human evaluation uses 100 examples per contrastive test set and calculates the Phi coefficient between prediction and explanation.
Training Techniques: High-quality supervised fine-tuning reaches 14.1 d-BLEU, while code pretraining adds 2 d-BLEU points and improves discourse probing by 4/1.5.These comparisons are reported for Chinese⇒English Fiction translation and English⇒Russian discourse probing.
Training Techniques: Adding PPO outperforms the other training combinations on translation quality, discourse awareness, and discourse knowledge probing.The comparison is between CodexGPT +FeedME-2 +PPO and the other reported strategies.

6 Conclusion and Future Work

The paper evaluates LLMs for document-level translation through discourse-aware prompts, model comparisons, and discourse-modeling analyses. GPT-4 shows improved discourse-aware performance and promise as a document-level translation paradigm, but struggles with subtle ranking distinctions.

The evaluation covers discourse-aware prompts, comparisons with advanced translation models, and analyses of discourse-modeling abilities.
GPT-4’s significantly improved discourse-aware performance makes it a promising paradigm for document-level translation.
Despite strong generative performance, GPT-4 struggles to discern subtle distinctions for ranking.

Limitations

The paper’s conclusions are constrained by limited datasets and imperfect human-evaluation criteria. The authors also report reproducibility checks across model evaluations, with stable overarching conclusions despite minor numerical variation.

Experiments on a limited set of datasets may reduce the accuracy or applicability of conclusions across contexts.The authors note possible bias, overlooked phenomena, and incomplete coverage despite using varied datasets.
Performance figures varied slightly between March and August 2023 evaluations, but the overarching conclusions remained consistent.
Human-evaluation criteria need refinement because score gaps are uneven in the Discourse Awareness category.The paper reports only a slight difference between Scores 5 and 4 but a larger gap between Scores 3 and 2.

Ethical Considerations

The ethical procedures address annotator selection, conflicts of interest, compensation, consent, and institutional review. The study used a professional language-service company and obtained Tencent IRB approval.

A professional language-service company with translation and linguistics expertise was selected after trial annotations were evaluated for accuracy and consistency.
Annotators were screened for model familiarity and affiliations, with conflicts of interest excluded and annotations routinely cross-checked.
Annotators received an hourly wage of $37.4, aligned closely with the cited compensation benchmarks.
Annotators received informed consent procedures, and the study underwent Tencent Institutional Review Board approval.The cited approval was IRB Protocol Number IRB-2023-00067, dated 01/12/2023.

A.1 Significance Testing

The study uses non-parametric and unequal-variance significance tests to assess translation results and probing-task differences. GPT-3.5 and GPT-4 significantly outperform other systems in the reported translation comparison, while GPT-4 differs significantly from GPT-3.5 on probing tasks.

p < 0.05 indicates that GPT-3.5 and GPT-4 significantly improve translation quality compared with other systems.The comparison uses a non-parametric one-tailed Wilcoxon signed-rank test.
The overall translation significance test combines all datasets, with consistent results across domains.
p < 0.001 indicates a significant difference between GPT-3.5 and GPT-4 in the probing tasks.The analysis uses Welch’s t-test with unequal variances and a two-tailed p-value.

A.2 Human Evaluation Guidelines

Human evaluation rates document-level translations on overall quality, grammaticality, word choice, terminology, context, tone, fluency, and clarity. The study used two professional evaluators, while the compared language models represent progressively different supervised fine-tuning, human-feedback, and dialogue-training approaches.

Evaluation criteria: Human evaluation scores range from 0 to 5, measuring overall translation quality and discourse consistency.A score of 5 represents excellent quality with accurate word choice, consistent key terms, context, and tone; a score of 0 represents poor quality with substantial mistranslation or omission.
Human evaluators: Two professional evaluators were recruited through Tencent’s designated supplier for the study.Their task-related backgrounds are detailed in Table 11.
Compared models: The evaluated model set spans GPT-3, supervised fine-tuned InstructGPT variants, RLHF-trained InstructGPT, and dialogue-oriented ChatGPT.The variants include human demonstrations, model-generated examples rated by humans, and reward modeling from human comparisons.