PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Making the Most of ChatGPT for Machine Translation

Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, Dacheng Tao

arXiv:2303.13780v4cs.CL

TL;DR

The paper addresses whether simple settings and prompts can better elicit ChatGPT’s machine translation ability, especially for difficult language pairs and domains. It evaluates temperature, task and domain information, and in-context learning, finding generally better results with lower temperatures and targeted prompts, while chain-of-thought prompting degrades translation. Hallucinations remain a concern for non-English-centric translation, and the prompt designs are acknowledged as limited.

  • Problem

    Prior studies found ChatGPT competitive for high-resource translation but limited for low-resource and distant languages while using simple prompts and basic settings.

  • Method

    The paper evaluates temperature, task information, domain information, and advanced in-context learning, proposing Task-Specific Prompts and Domain-Specific Prompts.

  • Results

    Lower temperatures generally improve performance; task and correct domain information improve translation, few-shot in-context learning helps, and chain-of-thought prompting degrades it through word-by-word translation.

  • Takeaways & Limitations

    Prompting and decoding choices can materially affect ChatGPT’s machine translation performance, particularly for complex tasks and domains.

  • Takeaways & Limitations

    The study uses simple prompts that may not fully elicit ChatGPT’s capabilities and does not fully study few-shot scenarios, especially chain-of-thought effects.

Abstract

from arXiv · show

ChatGPT shows remarkable capabilities for machine translation (MT). Several prior studies have shown that it achieves comparable results to commercial systems for high-resource languages, but lags behind in complex tasks, e.g., low-resource and distant-language-pairs translation. However, they usually adopt simple prompts which can not fully elicit the capability of ChatGPT. In this paper, we aim to further mine ChatGPT's translation ability by revisiting several aspects: temperature, task information, and domain information, and correspondingly propose an optimal temperature setting and two (simple but effective) prompts: Task-Specific Prompts (TSP) and Domain-Specific Prompts (DSP). We show that: 1) The performance of ChatGPT depends largely on temperature, and a lower temperature usually can achieve better performance; 2) Emphasizing the task information can further improve ChatGPT's performance, particularly in complex MT tasks; 3) Introducing domain information can elicit ChatGPT's generalization ability and improve its performance in the specific domain; 4) ChatGPT tends to generate hallucinations for non-English-centric MT tasks, which can be partially addressed by our proposed prompts but still need to be highlighted for the MT/NLP community. We also explore the effects of advanced in-context learning strategies and find a (negative but interesting) observation: the powerful chain-of-thought prompt leads to word-by-word translation behavior, thus bringing significant translation degradation.

1 Introduction

The paper investigates how temperature, task information, and domain information affect ChatGPT’s machine translation, proposing prompts intended to elicit stronger translation ability. Experiments also examine hallucinations and advanced in-context learning, including a degradation associated with chain-of-thought prompting.

  • Prior studies found ChatGPT competitive with commercial systems for high-resource languages but limited on low-resource and distant-language translation.
  • The paper proposes an optimal temperature setting, Task-Specific Prompts (TSP), and Domain-Specific Prompts (DSP) to improve ChatGPT’s translation.
  • Lower temperatures generally improve ChatGPT’s translation performance, especially for difficult languages.
  • Emphasizing task information improves performance, particularly on complex machine translation tasks.
  • Correct domain information improves performance, whereas incorrect domain information causes significant degradation.
  • ChatGPT may hallucinate on non-English-centric translation tasks, while few-shot in-context learning improves performance and chain-of-thought prompting induces word-by-word translation with degradation.

2 Evaluation Setting

The evaluation compares ChatGPT with Google Translator across multilingual and cross-domain translation test sets. COMET-20 is the primary metric, with BLEU and ChrF also reported.

  • Models: The study mainly compares ChatGPT with Google Translator, using gpt-3.5-turbo-0301 as the default ChatGPT model.
  • Data: Flores-200 provides 1,012 sentences translated into 204 languages for multilingual translation and in-context learning evaluation.
  • Data: Cross-domain translation is evaluated using WMT19 Biomedical, WMT19 News Translation, and WMT22 E-Commerce test sets.
  • Metrics: COMET-20 is the primary reference-based metric, while BLEU and ChrF are additionally reported for completeness.

3 Zero-Shot Translation

The study examines temperature, task-specific prompting, and domain-specific prompting as ways to improve ChatGPT’s zero-shot machine translation, including difficult and non-English-centric language pairs. Lower temperature and targeted prompts generally improve performance, but hallucinations remain a concern for non-English-centric translation.

  • 3.1 The Effect of Temperature: ChatGPT’s performance declines as temperature increases, with the strongest degradation for difficult language pairs such as English⇒Chinese.When temperature changes from 0 to 1, Chinese translation loses 4.3 COMET points and 3.7 BLEU points; the experiments therefore use T = 0.
  • 3.2 The Effect of Task Information: Task-Specific Prompts (TSP) prepend a machine-translation instruction to emphasize the translation task and bridge ChatGPT’s task gap.The prompt begins with “You are a machine translation system.”
  • 3.2.1 English-Centric Language Pairs: TSP achieves comparable COMET results to Google Translator and exceeds it for English⇒Romanian, scoring 92.9 versus 91.6.TSP improves vanilla ChatGPT particularly for low-resource or distant languages, including +0.8 COMET for English⇒Chinese and +0.5 for English⇒Romanian.
  • 3.2.2 Non-English-Centric Language Pairs: For non-English-centric pairs, ChatGPT generates hallucinated translation-related text, while lower temperature and TSP reduce—but do not eliminate—these hallucinations.The post-editing approach only roughly removes recurring irrelevant patterns, and TSP yields only slight performance improvements for Chinese⇔Romanian.
  • 3.3 The Effect of Domain Information: Domain-Specific Prompts (DSP) provide the correct domain of the sentence to facilitate ChatGPT’s cross-domain generalization, with incorrect domain information used as a control.The prompt identifies the domain through a template such as “You are a machine translation system that translates sentences in the [DOM] domain.”
  • 3.3 The Effect of Domain Information: DSP consistently improves ChatGPT’s COMET scores and surpasses Google Translator on two datasets, although its BLEU improvements are inconsistent and remain substantially behind Google Translator.The reported wins occur on WMT19 Biomedical Chinese⇒English and WMT19 News English⇒Chinese.

4 Few-shot Machine Translation

The paper finds that few-shot in-context learning improves ChatGPT’s machine translation, especially with more and better-selected demonstrations, while conventional CoT prompting substantially degrades translation quality.

  • 4.1 Few-Shot In-Context Learning: The few-shot experiments evaluate English⇒Chinese, English⇒Romanian, and English⇒German translation on Flores-200 using random and TopK demonstrations.
  • 4.1 Few-Shot In-Context Learning: Few-shot in-context learning consistently improves BLEU and COMET over zero-shot translation, with further gains as the number of shots increases.
  • 4.1 Few-Shot In-Context Learning: TopK demonstration selection further improves performance and surpasses Google Translator for English⇒Romanian and English⇒Chinese.The reported scores are 94.0 versus 91.6 for English⇒Romanian and 68.8 versus 68.5 for English⇒Chinese.
  • 4.2 Chain-of-Thought: Zero-shot CoT causes a significant COMET decline, including an 8.8-point drop for English⇒Chinese, while 1-shot CoT remains below zero-shot prompting.
  • 4.2 Chain-of-Thought: The authors attribute CoT’s translation degradation to word-by-word translation behavior and propose exploring MT-inspired CoT variants in future work.

5 Related Work

The related work situates ChatGPT within large language model research and prior machine translation evaluations, while noting that its translation capabilities remain incompletely explored.

  • Large Language Models: Large language models are trained on massive text collections and commonly categorized as encoder-only, decoder-only, or encoder-decoder architectures.
  • Large Language Models: Decoder-only language models have developed notable zero-shot and few-shot abilities as model scale and training have increased.
  • ChatGPT for Machine Translation: Earlier studies found ChatGPT competitive with commercial systems for high-resource European languages but significantly weaker on low-resource and distant-language translation.
  • ChatGPT for Machine Translation: Subsequent studies extended ChatGPT translation research with additional linguistic information and CoT-like frameworks for human-like translation generation.

6 Conclusion

The paper mines ChatGPT’s translation ability through temperature, task, and domain information, then evaluates few-shot learning and CoT prompting. It reports gains from lower temperature and few-shot learning, but degradation from conventional CoT.

  • 6 Conclusion: The paper proposes an optimal temperature setting, Task-Specific Prompts, and Domain-Specific Prompts to improve ChatGPT’s machine translation.
  • 6 Conclusion: Across language pairs and domains, the experiments support the effectiveness of the proposed prompts and show that lower temperature usually improves performance.
  • 6 Conclusion: Few-shot in-context learning consistently improves performance, whereas conventional CoT degrades it through word-by-word translation behavior.
  • 6 Conclusion: Future work includes EBMT-inspired prompt design, statistical-MT-inspired CoT, improved demonstrations, human-like CoT, and additional MT settings such as document translation.

Limitations

The authors identify limitations in prompt design and in the evaluation of few-shot scenarios, particularly Chain-of-Thought for machine translation.

  • Limitations: The proposed prompts are simple rather than carefully designed, so they may not fully elicit ChatGPT’s capabilities.
  • Limitations: Few-shot scenarios remain incompletely studied, especially the effect of Chain-of-Thought prompting on machine translation.

Ethics Statement

The paper states that it follows the EMNLP Ethics Policy and uses publicly available models and datasets for machine translation research without identified ethical issues.

  • The authors state that they strictly adhere to the EMNLP Ethics Policy.
  • The study explores ChatGPT's translation ability using open-sourced machine translation datasets.
  • The compared models and evaluation datasets are publicly available and widely adopted by researchers.
  • The authors believe the research does not pose ethical issues.
Loading 2303.13780v4…