PaperlayerView exact revision on arXiv

Source-linked AI summary

How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation

Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, Hany Hassan Awadalla

arXiv:2302.09210v1cs.CL

TL;DR

GPT machine translation had been insufficiently evaluated relative to established systems. The paper therefore assesses three GPT models across 18 translation directions and multiple translation settings, finding competitive strengths alongside language- and direction-specific limitations, while identifying complementary uses with NMT.

  • Problem

    GPT models’ machine translation performance remained under-investigated relative to commercial and state-of-the-art research systems.

  • Method

    The paper evaluates three GPT models across 18 language pairs and translation settings, using automatic, human, qualitative, and adapted document-level evaluations.

  • Results

    GPT translations are more fluent and paraphrastic than the investigated NMT system for X-E directions while remaining faithful to the source, whereas E-X results can reflect less adequate translations.

  • Takeaways & Limitations

    GPT’s translation characteristics suggest applications in figurative-text translation, noisy input domains, and hybrid systems combining GPT with NMT.

  • Takeaways & Limitations

    Conclusions should not be generalized to other languages without further evaluation, especially where test sets or baselines are weak.

Abstract

from arXiv · show

Generative Pre-trained Transformer (GPT) models have shown remarkable capabilities for natural language generation, but their performance for machine translation has not been thoroughly investigated. In this paper, we present a comprehensive evaluation of GPT models for machine translation, covering various aspects such as quality of different GPT models in comparison with state-of-the-art research and commercial systems, effect of prompting strategies, robustness towards domain shifts and document-level translation. We experiment with eighteen different translation directions involving high and low resource languages, as well as non English-centric translations, and evaluate the performance of three GPT models: ChatGPT, GPT3.5 (text-davinci-003), and text-davinci-002. Our results show that GPT models achieve very competitive translation quality for high resource languages, while having limited capabilities for low resource languages. We also show that hybrid approaches, which combine GPT models with other translation systems, can further enhance the translation quality. We perform comprehensive analysis and human evaluation to further understand the characteristics of GPT translations. We hope that our paper provides valuable insights for researchers and practitioners in the field and helps to better understand the potential and limitations of GPT models for translation.

1 Introduction

This paper addresses the under-investigated performance of GPT models in machine translation through a systematic evaluation of their capabilities, prompting, robustness, and integration with NMT systems. It examines multiple languages, models, and translation settings while combining automatic, human, and qualitative analyses.

  • Contribution: The paper aims to clarify GPT models’ strengths and limitations for machine translation and provide insights for researchers and practitioners.Its motivation is the emergence of large language models with translation capabilities and the possibilities they create for more effective translation systems.
  • Research gap and scope: The study systematically assesses GPT machine translation against commercial and state-of-the-art research systems, focusing on performance, prompts, document-level translation, domain robustness, and NMT integration.The evaluation targets the research gap identified for GPT translation performance relative to established systems.
  • Evaluation coverage: Experiments cover 18 language pairs spanning high- and low-resource languages and English-centric and non-English-centric directions.The paper compares text-davinci-002, GPT3.5 (text-davinci-003), and ChatGPT, which differ significantly in translation capabilities.
  • Experimental questions: The experiments investigate prompt selection, zero-shot translation, resource-level differences, document-level translation, domain shift, and hybrid GPT–NMT translation.These experiments are organized across sections covering prompting, translation directions, robustness, and hybrid systems.
  • Analysis and broader capabilities: Human evaluation and detailed analyses examine GPT translation quality, differences from NMT, language-modeling artifacts, language-direction characteristics, and parallel-data bias.The paper also explores multilingual capabilities beyond translation and cross-lingual natural language tasks.

2 Experimental Setup

The experimental setup evaluates three GPT models across diverse translation directions using recent public test sets and comparisons with leading research and commercial systems. It combines neural automatic metrics, human evaluation, translation analysis, and an adapted document-level metric, while acknowledging limitations in current evaluation methods.

  • Datasets: The evaluation covers 18 translation directions across high- and low-resource languages, including English-centric and non-English-centric directions.Languages include European, Asian, Cyrillic, and African language groups.
  • Datasets: Recent WMT22 test sets are used for most languages, while Icelandic and Hausa use WMT21 test sets to reduce training-data overlap and avoid unnatural translationese sources.The datasets and analyses are publicly available to support reproducibility and data sharing.
  • Comparison systems: GPT outputs are compared with WMT-Best systems, which combine top-ranked commercial and research systems, and with Microsoft Translator.WMT-Best outputs come from WMT evaluation campaigns, while Microsoft Translator is accessed through Azure’s public API.
  • GPT systems: The study evaluates text-davinci-002, text-davinci-003, and ChatGPT, accessed through Microsoft Azure OpenAI Service APIs.The models include an InstructGPT system, its improved version, and a conversationally optimized model.
  • Evaluation methods: Sentence-level evaluation uses COMET-22, while human evaluation uses source-based contrastive DA+SQM annotations from professional annotators.The setup also includes analysis of translation characteristics.
  • Document-level evaluation: Doc-COMETkiwi adapts COMET metrics by scoring overlapping document segments to address non-one-to-one sentence mappings and incorporate contextual evaluation.The authors do not claim this is an optimal document-level metric and argue that more robust metrics remain necessary.
  • Limitations: Current metrics may inadequately measure GPT translation quality, motivating metrics that account for the models’ distinctive characteristics.The limitation concerns the adequacy of existing evaluation measures rather than the translation systems alone.

3 Experiments

Across its experiments, GPT performs competitively for high-resource and some direct language pairs, but shows weaker low-resource performance and metric-dependent weaknesses. Document context, domain, prompting, and hybrid routing substantially affect results.

  • 3.2 Zero-Shot Translation Capabilities of GPT Models: text-davinci-003 outperforms the other GPT models across evaluated languages, with zero-shot performance comparable to the best DE-EN system and better than the best ZH-EN system.
  • 3.3 GPT Performance on High-resource Languages: GPT surpasses WMT-Best and MS-Translator for DE-EN, JA-EN, and ZH-EN, while few-shot prompting generally adds little over zero-shot translation into English.
  • 3.4 GPT Performance on Low-resource and non English-centric Languages: Single high-quality shots produce consistent gains from English and significantly improve direct DE-FR and FR-DE translation, but low-resource Hausa and Icelandic remain below WMT-Best.
  • 3.5 Document-Level MT with GPT: Increasing document context improves all metrics, with larger gains for BLEU and ChrF, while performance surpasses MS-Translator and approaches WMT-Best as the window grows.
  • 3.6 Robustness Toward Domain Shift: Domain results are mixed: GPT performs strongly on conversational, news, e-commerce, and social directions, but lexical metrics often remain lower despite high COMET scores.
  • 3.7 Hybrid GPT and NMT Translation: Hybrid Max-Routing achieves the highest COMET-22 scores across 12 language pairs, while GPT can outperform MS-Translator on selected directions and serve as a fallback.

4 Human Evaluation and Analysis

The authors compare GPT translations with WMT-Best and Microsoft Translator using human evaluation and COMETKiwi, finding strong performance across many high-resource directions. Analysis shows GPT handles noisy and colloquial inputs well but can produce unnatural translations for complex expressions.

  • Human evaluation: GPT outperforms WMT-Best systems on CS-EN, ZH-EN, EN-ZH, and DE-FR, while achieving comparable results on most high-resource languages.The human evaluation results are consistent with COMETKiwi scores.
  • Human evaluation: Human evaluation and COMETKiwi results show that GPT win rates are higher for highly performing language pairs.The relationship is reflected in both item-level human judgments and COMETKiwi scores.
  • Translation analysis: GPT performs better and more robustly than WMT on erroneous, short, or colloquial source sentences, including misspelled or incomplete inputs.GPT generally preserves semantic information, whereas WMT-Best sometimes adds or omits content.
  • Translation analysis: GPT tends to produce unnatural translations for sentences containing uncommon or complex expressions.

5 GPT Translation Characteristics

The analysis characterizes GPT translations through fluency, alignment, reordering, and language-modeling artifacts, contrasting English-involving and non-English-centric directions. GPT is often more fluent and paraphrastic for X-E translation, but E-X quality and artifacts vary by target language and may reduce faithfulness.

  • 5 GPT Translation Characteristics: GPT and NMT differ through two proposed biases: Parallel Data Bias and Language Modeling Bias.The former concerns training without noisy parallel supervision, while the latter concerns fluency-oriented generation and task control.
  • Translation directions: For non-English-centric De-Fr and Fr-De directions, measurements of fluency, alignment, and non-monotonicity provide no conclusive evidence of greater GPT paraphrasticity.
  • Parallel Data Bias Artifacts: GPT often gains more over MS Translator on high-perplexity inputs, including several script-divergent and low-resource language pairs.The pattern is reported for English-Chinese, English-Japanese, English-Russian, and low-resource settings.
  • 5.7 Summary: GPT X-E translations are more fluent and paraphrastic than MS Translator while remaining generally faithful to the source.They show lower perplexity, more unaligned source words, and longer-range reorderings without prominent unsupported factual insertions.
  • 5.7 Summary: GPT E-X translations show greater source-word unalignment and non-monotonicity, alongside slightly higher target-word unalignment, making reduced adequacy plausible.The effects of language-modeling bias cannot be generalized across E-X language pairs because the target language is critical.
  • Areas of improvement: GPT inserts unsupported end-of-sentence punctuation more frequently than NMT across X-E, E-X, and X-Y directions.The authors identify this as an artifact of language-modeling bias toward well-formed sentences.
  • Areas of improvement: GPT’s greater fluency and paraphrasticity may be unsuitable where literal, highly faithful translation is required.The authors identify noisy or ill-formed inputs and figurative text as potential application areas instead.

6 Multilingual Capabilities beyond Translation

The paper tests whether translation performance predicts multilingual reasoning ability using the MGSM benchmark across ten non-English languages. Results show strong model and prompting differences, with persistent limitations for low-resource languages and only moderate reasoning performance in some languages with strong translation results.

  • Benchmark: MGSM evaluates multilingual grade-school arithmetic reasoning across ten languages besides English.The languages include Bengali, Chinese, French, German, Japanese, Russian, Spanish, Swahili, Telugu, and Thai.
  • Native-CoT: text-davinci-003 surpasses text-davinci-002 for every language, with especially high performance in English, German, French, and Spanish.Russian, Japanese, and Chinese score lower than the Latin-script languages, while low-resource languages remain limited.
  • Translate-EN: Translate-EN improves performance for Russian, Japanese, Chinese, and low-resource languages, but its gains are not uniform and it deteriorates results for Latin languages.
  • Translate-EN+: Translate-EN+ significantly improves results in some languages, including French, Spanish, and Russian, and matches Translate-EN in others.This setup keeps the prompt template in English while translating the remaining content.
  • Translation versus reasoning: Despite strong translation performance for Russian, Japanese, and Chinese, text-davinci-003 achieves only moderate MGSM performance in those languages.The authors hypothesize that reasoning benefits from programming-language training and that multilingual data proportions may explain the gap.

7 Conclusions and Future Directions

The study evaluates GPT translation across languages, domains, prompting settings, and model combinations, finding strong high-resource performance but persistent challenges for underrepresented languages and evaluation.

  • Conclusions and Future Directions: The evaluation covers 18 language pairs across four domains and includes human, neural, and conventional automatic metrics.The study also analyzes multilingual reasoning and GPT translation outputs relative to state-of-the-art NMT systems.
  • Conclusions and Future Directions: GPT systems produce fluent, competitive zero-shot translations, especially for high-resource languages, and few-shot in-context examples further improve quality.
  • Conclusions and Future Directions: Future directions include improving underrepresented-language translation, exploiting in-context learning, making model fusion more efficient, and developing better contextual metrics.The paper identifies computational expense and the limitations of lexical matching metrics as motivations for these directions.
  • Conclusions and Future Directions: GPT models excel on well-represented languages but struggle with less-resourced languages, revealing uneven support across languages and tasks.The authors connect broader multilingual capability to the need for more data across languages and identify scalability, diversity, and fairness challenges.

Limitations

The study’s conclusions are bounded by its evaluated languages, test sets, baselines, and the limited ability of automatic metrics to capture GPT translation quality.

  • Limitations: The evaluation covers 18 translation directions, so its conclusions should not be generalized to other languages without further evaluation.The authors also caution against conclusions based on low-quality test sets or weak baselines, especially for low-resource languages.
  • Limitations: BLEU and chrF can give misleading signals for GPT outputs, while document-level metrics may miss context-based translation effects.The authors therefore combine metrics with comprehensive analysis and human evaluation.

Ethics Statement

The authors report language-specific performance disparities, potential stereotypes and misinformation, and hallucinations in GPT translations.

  • Ethics Statement: GPT models perform better for some languages than others, which may create unfair and unequal outcomes for users paying the same cost.
  • Ethics Statement: The models may amplify training-data stereotypes, including misgendering a female name such as “Julia” in translation.
  • Ethics Statement: False insertions and hallucinations were observed, especially in E-X translation directions.

A Prompt Templates

The appendix provides prompt templates for sentence-level, zero-shot, and document translation, specifying source-target instructions and document-line handling.

  • Prompt Templates: The sentence-level template follows the OpenAI Playground’s default translation instruction and format.
  • Prompt Templates: The zero-shot ChatGPT template instructs translation from a named source language to a named target language.
  • Prompt Templates: The document template includes a document or context window and instructs the model to translate each line into the target language.

B Few-shot Example Selection Data Pool

The few-shot example selection pool is documented for each translation direction, distinguishing original WMT training data from cleaned high-quality data.

  • The data pool is reported separately for each translation direction.
  • The Raw column gives the size of the original WMT training dataset.
  • The Cleaned column gives the size of the high-quality data retained after cleaning.

C Human Evaluation details

Human evaluation and COMETkiwi results are presented in detail for WMT Best Systems and GPT.

  • The table reports human evaluation results for WMT Best Systems and GPT.
  • The table reports COMETkiwi results for WMT Best Systems and GPT.
  • The results are provided in detailed form for both system groups.

C.1 Human Evaluation Analysis

The qualitative analysis identifies both strengths and weaknesses in GPT’s English–Japanese translations, including robustness to imperfect inputs but unnatural or hallucinated outputs.

  • Positive characteristics: GPT handles misspellings and unclosed quotes while preserving semantic information in translations.
  • Positive characteristics: GPT gives reasonable translations for partial or incomplete colloquial sentences, unlike WMT-Best outputs with extra or missing content.
  • Negative characteristics: GPT tends to produce unnatural translations for sentences containing unusual or complex expressions.
  • Negative characteristics: GPT can hallucinate words, including “udon” when the source does not mention it and a non-existing Japanese rendering of “PM.”
  • Analysis scope: Table 14 presents English-to/from-Japanese examples illustrating positive and negative characteristics identified by human item scores.
Loading 2302.09210v1…