PaperlayerView exact revision on arXiv

Source-linked AI summary

Leveraging GPT-4 for Automatic Translation Post-Editing

Vikas Raunak, Amr Sharaf, Yiren Wang, Hany Hassan Awadallah, Arul Menezes

arXiv:2305.14878v2cs.CLcs.AI

TL;DR

NMT can produce unreliable outputs even in higher-resource language pairs, motivating post-editing for critical use. This paper formalizes direct GPT-4 post-editing and reports more trustworthy edits and improved WMT-22 results across four language pairs.

  • Problem

    NMT can produce catastrophic outputs in rare cases despite high average performance, making post-editing important for use in critical settings.

    Source: §1 · p. 1

  • Method

    The paper formalizes direct GPT-4 post-editing, in which the model proposes edits to a translation and generates a revised output without prior quality estimation or error detection.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    GPT-4 post-editing improved upon WMT-22 Best translation performance on four language pairs and produced more trustworthy edits than a previous-generation LLM.

    Source: §7 · p. 8

  • Takeaways & Limitations

    The authors characterize GPT-4 post-editing as promising for improving NMT translations on the evaluated WMT-22 language pairs.

    Source: §7 · p. 8

    Because GPT-4 is accessible only as a black box, the authors cannot rule out data contamination on the WMT-22 test sets despite their curation postdating its training cutoff.

    Source: §8 · p. 9

Abstract

from arXiv · show

While Neural Machine Translation (NMT) represents the leading approach to Machine Translation (MT), the outputs of NMT models still require translation post-editing to rectify errors and enhance quality under critical settings. In this work, we formalize the task of direct translation post-editing with Large Language Models (LLMs) and explore the use of GPT-4 to automatically post-edit NMT outputs across several language pairs. Our results demonstrate that GPT-4 is adept at translation post-editing, producing meaningful and trustworthy edits to translations that help improve its general quality as well as remove different classes of major errors in translations. In particular, human evaluations on assessing edit trustworthiness show that GPT-4 exhibits a large improvement over the prior state-of-the-art LLM. Notably, we improve upon state-of-the-art performance on WMT-22 English-Chinese, English-German, Chinese-English and German-English language pairs using GPT-4 based post-editing, as evaluated by state-of-the-art MT quality metrics. However, we also show that GPT-4 could produce hallucinated edits, thereby urging caution in its use as an expert translation post-editor.

1 Introduction

The paper addresses NMT reliability problems and asks whether LLMs can directly post-edit translations. It reports that GPT-4 post-editing improves WMT-22 Best performance across four language pairs.

  • Web-mined NMT can produce catastrophic outputs in rare cases despite strong average performance, and detecting such behavior remains challenging.Source: §1 · p. 1
  • Post-editing remains important for NMT use in critical translation and localization settings, and LLMs could help detect and fix translation errors.Source: §1 · p. 1
  • GPT-4 post-editing improved upon WMT-22 Best translation performance on English-Chinese, English-German, Chinese-English, and German-English, as measured by state-of-the-art MT quality metrics.Source: §1 · pp. 1–2
  • The paper formalizes direct LLM post-editing and reports that GPT-4 produces meaningful, human-judgment-aligned edits with high trustworthiness.Source: §1 · p. 1

2 The Translation Post-Editing Task

The paper formalizes direct translation post-editing as proposing edits to an existing translation and generating a revised output. It frames questions about how outputs relate to the input, quality gains, error-span edits, and the fidelity of proposed edits.

  • Given a source and translation, the task proposes improvements as edits and generates a revised translation.Source: §2 · p. 2
  • With CoT, edits are verbalized before the final translation; without CoT, the model generates the improved translation without that intermediate reasoning chain.Source: §2 · p. 2
  • The paper asks whether post-edited translations revise the supplied translation or resemble the LLM's zero-shot translation, and how CoT affects that relationship.Source: §2 · p. 2
  • It asks whether post-editing improves general translation quality and whether CoT contributes to those improvements.Source: §2 · pp. 2–3
  • It examines whether LLMs modify human-annotated error spans, with frequent changes alongside quality gains treated as evidence of closer alignment with human error judgments.Source: §2 · p. 3
  • It asks whether proposed CoT edits appear in the final output, since their realization is relevant to edit fidelity and trustworthiness.Source: §2 · p. 3

3 Experimental Settings

Experiments cover WMT-22 general MT data and MQM-annotated WMT-20/21 outputs, using GPT-4 and gpt-3.5-turbo across several post-editing settings. The paper evaluates quality with COMET models and translation similarity with TER.

  • Experiments use the full WMT-22 test datasets and WMT-20/21 samples annotated with major MQM errors.Source: §3 · p. 3
  • The study compares GPT-4 and gpt-3.5-turbo using CoT, no-CoT, and Structured-CoT settings; Microsoft Translator supplies initial WMT-22 translations.Source: §3 · p. 3
  • General quality is measured with reference-free and reference-based COMET models, while translation similarity is measured with TER.Source: §3 · p. 4

4 Results and Measurements

The results examine whether LLM post-editing changes initial translations, improves translation quality and removes annotated errors, and whether proposed edits are realized. GPT-4 improves quality and edit trustworthiness, but CoT can constrain output to a lower-quality initial translation and proposed edits may be only partly realized.

4.1 Nature of Post-Edited Translations

  • With CoT, post-edited translations were closer to initial translations than zero-shot outputs; without CoT, outputs were closer to zero-shot translations, and CoT could lower quality when the initial translation was weaker.The closeness pattern is reported for WMT-22 En-Zh and Zh-En and across other metrics, language pairs, and gpt-3.5-turbo.Source: Table 3 · p. 4, Table 4 · p. 4, §4.1 · p. 4

4.2 General Quality Improvements

  • GPT-4 direct post-editing of MS-Translator consistently surpassed WMT-22-Best, while both CoT and direct post-editing improved high-quality MS-Translator translations.The reported general-quality comparisons cover WMT-22 De-En, Zh-En, En-De, and En-Zh.Source: §4.2 · p. 4, Table 5 · p. 5, Table 6 · p. 5

4.3 Edits On Human Annotated Error Spans

  • GPT-4 modified more than half of major MQM error spans and improved COMET-KIWI quality across 14 WMT-22 system outputs in three language pairs.The reported error-span rates exceed 60% for En-De, 70% for Zh-En, and 50% on average for En-Ru.Source: Table 9 · p. 6, Table 10 · p. 6, Table 11 · p. 6

4.4 Trustworthiness of the Proposed Edits

  • Human evaluation found higher edit-realization rates for GPT-4 than gpt-3.5-turbo, though proposed edits were sometimes unrealized, especially for Zh-En and De-En.The evaluation labels whether all, some, or none of the proposed edits appear in the final translation.Source: Figure 1 · p. 6, §4.4 · pp. 6–7

5 Further Discussion

GPT-4 post-editing improves translation quality across language pairs, with larger gains for translations into English. The discussion also weighs CoT’s trade-off and the trustworthiness of proposed edits, while leaving joint MQM assessment and post-editing for future evaluation.

Post-Editing Across Language Pairs

  • GPT-4 post-editing yields consistent quality gains across language pairs, with larger gains for translations into English.These results are reported for CoT post-editing with MS-Translator as the initial translation.Source: §5 · p. 7

Utility of the Chain-of-Thought

  • CoT edit proposals constrain outputs toward initial translations but can reduce final translation quality; the proposals may still aid trustworthiness.The paper questions whether CoT’s variable computation is necessary for post-editing.Source: §5 · pp. 7–8
  • MQM-category structure did not improve final quality but did not harm it; whether GPT-4 post-editing can be combined with MQM-based quality assessment remains unevaluated.The joint evaluation is identified as future work.Source: §5 · p. 8

Trustworthiness of the Proposed Edits

  • GPT-4’s proposed edits are realized in final translations more often than gpt-3.5-turbo’s, marking a substantial trustworthiness improvement on the two evaluated language pairs.The paper describes this as an emergent ability under a common definition of emergent abilities.Source: §5 · p. 8

6 Related Work

Prior work explored neural APE architectures, training strategies, and context integration, while LLM translation research addressed a different task. This paper positions GPT-4 direct post-editing as a new generative setting.

Automatic Post-Editing of Translations

  • Earlier neural automatic post-editing research explored model architectures, learning strategies, training-data choices, and source-context integration.Reviews also covered traditional and neural-based approaches to APE.Source: §6 · p. 8

Leveraging LLMs for Post-Editing

  • Vidal et al. (2022) studied GPT-3 post-editing with glossaries, whereas this work presents GPT-4 direct post-editing in a purely generative setting.The paper describes its work as the first investigation of GPT-4 for automatic translation post-editing.Source: §6 · p. 8
  • Although prior work explored LLMs for translation, this paper distinguishes direct automatic post-editing as a different task.The paper formalizes direct post-editing without a prior quality-estimation or error-detection step.Source: §6 · p. 8

7 Summary and Conclusions

The paper formalizes direct generative post-editing and finds that GPT-4 makes meaningful, trustworthy edits that improve translation quality. GPT-4 post-editing improves on WMT-22 Best performance across four language pairs.

  • The authors formalized direct automatic post-editing with LLMs and proposed research questions and measurements for evaluating its utility.Source: §7 · p. 8
  • Zero-shot chain-of-thought was critical for constraining post-edited translations to remain close to the initial translation.Source: §7 · p. 8
  • GPT-4 produced meaningful, human-judgment-aligned edits that improved general translation quality and were more trustworthy than edits from a previous-generation LLM.Source: §7 · p. 8
  • GPT-4 post-editing improved on WMT-22 Best translation performance for English-Chinese, English-German, Chinese-English, and German-English.Source: §7 · p. 8

8 Limitations

The authors note that their GPT-4 experiments rely on API-level, black-box access, so possible data contamination cannot be ruled out, including on WMT-22 test sets.

  • Because GPT-4 was available only through a black-box API, the authors cannot rule out data contamination, including on WMT-22 test sets.Source: §8 · p. 9
  • The experiments used WMT-22 test sets and system outputs curated before GPT-4's training-data cutoff, but black-box access leaves contamination uncertain.Source: §8 · p. 9

C Prompts Used for Post-Editing

The appendix describes the prompt baseline without chain-of-thought: it uses Step 2 of the user prompt and the same system prompt as the chain-of-thought baseline.

D Edit Hallucinations by GPT-4

The appendix presents examples of unrealized GPT-4 edits and cases where annotators judged that only some proposed edits appeared in the final translation. It also includes evaluation prompts, error-labeling guidance, and tables on edit efficacy and translation quality.

Related papers

Loading 2305.14878v2…