Source-linked AI summary
Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine
Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, Shuming Shi, Zhaopeng Tu
TL;DR
The paper addresses limited evidence about ChatGPT's machine-translation ability relative to commercial systems. It evaluates prompting, multilingual translation, and robustness, then examines pivot prompting and GPT-4; performance is competitive for high-resource European languages and reaches commercial-system levels for distant languages with GPT-4.
Problem
The report investigates how ChatGPT performs for machine translation and how it compares with commercial translation products across languages and settings.
Method
The study evaluates ChatGPT using advised translation prompts, Flores-101 language-pair tests, three robustness sets, pivot prompting, and GPT-4 re-evaluation.
Results
ChatGPT is competitive on high-resource European languages, weaker on low-resource or distant languages and some robustness sets, while GPT-4 reaches commercial-system performance even for distant languages.
Takeaways & Limitations
With GPT-4 as the engine, the report concludes that ChatGPT has become a good translator.
Takeaways & Limitations
The evaluation randomly samples 50 items per test set, and repeated queries can produce varying results; repeated trials and averaged results would be more reliable.
Abstract
from arXiv · showhide
This report provides a preliminary evaluation of ChatGPT for machine translation, including translation prompt, multilingual translation, and translation robustness. We adopt the prompts advised by ChatGPT to trigger its translation ability and find that the candidate prompts generally work well with minor performance differences. By evaluating on a number of benchmark test sets, we find that ChatGPT performs competitively with commercial translation products (e.g., Google Translate) on high-resource European languages but lags behind significantly on low-resource or distant languages. As for the translation robustness, ChatGPT does not perform as well as the commercial systems on biomedical abstracts or Reddit comments but exhibits good results on spoken language. Further, we explore an interesting strategy named $\mathbf{pivot~prompting}$ for distant languages, which asks ChatGPT to translate the source sentence into a high-resource pivot language before into the target language, improving the translation performance noticeably. With the launch of the GPT-4 engine, the translation performance of ChatGPT is significantly boosted, becoming comparable to commercial translation products, even for distant languages. Human analysis on Google Translate and ChatGPT suggests that ChatGPT with GPT-3.5 tends to generate more hallucinations and mis-translation errors while that with GPT-4 makes the least errors. In other words, ChatGPT has already become a good translator. Please refer to our Github project for more details: https://github.com/wxjiao/Is-ChatGPT-A-Good-Translator
1 Introduction
This preliminary report evaluates ChatGPT for machine translation across prompting, language pairs, and robustness settings, finding strong performance on high-resource European languages and a substantial GPT-4 improvement for distant languages.
- Evaluation scope: ChatGPT is evaluated for translation prompting, multilingual language-pair performance, and robustness across domain-specific or noisy text.The study investigates how prompt style, resource level, language family, and input domain relate to translation quality.
- Translation prompting: Candidate prompts generally work well with only minor performance differences, so the study adopts the best-performing prompt for subsequent experiments.The prompts were obtained by asking ChatGPT itself for translation advice.
- Multilingual translation: ChatGPT performs competitively with commercial systems on high-resource European languages but lags significantly on low-resource or distant languages.This pattern is reported from evaluations on four selected languages using Flores-101 test sets.
- Translation robustness: ChatGPT is weaker than commercial systems on biomedical abstracts and Reddit comments but performs well on spoken language.These findings come from three translation-robustness test sets.
- Improving ChatGPT: Pivot prompting translates through a high-resource pivot language, while GPT-4 significantly boosts performance to commercial-system levels even for distant languages.Human analysis also reports more hallucinations and mis-translation errors with GPT-3.5, whereas GPT-4 makes the fewest errors.
2 ChatGPT for Machine Translation
This preliminary study evaluates ChatGPT for machine translation through prompt design, multilingual translation, and robustness testing. ChatGPT is competitive on high-resource European languages and spoken language but weaker on low-resource, distant, biomedical, and noisy data.
- Evaluation Setting: The study compares ChatGPT with Google Translate, DeepL Translate, and Tencent TranSmart using multilingual and robustness test sets.The evaluation uses Flores-101, WMT19 Biomedical Translation, and WMT20 Robustness data.
- Evaluation Setting: Because early ChatGPT access could not process large batches efficiently, the study randomly samples 50 sentences from each test set.This constraint made obtaining translation results time-consuming.
- Translation Prompts: Three ChatGPT-generated translation prompts generally work well, but TP3 performs best across BLEU, ChrF++, and TER on Chinese-to-English translation.ChatGPT still trails the commercial baselines by at least 5.0 BLEU points on this task.
- Multilingual Translation: ChatGPT performs competitively with commercial systems for German-English translation but lags significantly for Romanian-English translation.Its BLEU score is 46.4% lower than Google Translate for English-to-Romanian and 10.3% lower for Romanian-to-English.
- Multilingual Translation: Performance gaps grow for language pairs from different families, especially when translation is both low-resource and cross-family.The report attributes this pattern to weaker knowledge transfer and competition for model capacity across translation and other NLP tasks.
- Translation Robustness: ChatGPT underperforms Google Translate and DeepL Translate on biomedical abstracts and Reddit comments but significantly outperforms them on crowdsourced speech data.The results suggest stronger handling of natural spoken language than of domain-specific or noisy text.
3 Improving ChatGPT for MT
The report explores pivot prompting and GPT-4 engine improvements to address ChatGPT’s weaker machine translation on low-resource or distant languages. Pivot prompting noticeably improves distant-language translation, while GPT-4 brings performance to the level of top commercial systems.
- 3.1 Pivot Prompting: Pivot prompting translates the source into a high-resource pivot language before producing the target translation.The adjusted prompt asks for pivot-language and target-language translations sequentially.
- 3.1 Pivot Prompting: Pivot prompting is motivated by transferring knowledge from a high-resource pivot language to low-resource target languages.The approach is more convenient in ChatGPT because it identifies the pivot and target languages within one prompt.
- 3.1 Pivot Prompting: Nearly 3.9 and 6.6 BLEU points improve German⇒Chinese and Romanian⇒Chinese translations, respectively, with pivot prompting.The method also noticeably fixes under-translation, though generating longer sentences may slow inference.
- 3.2 GPT-4 as the Engine: GPT-4 significantly boosts performance on all four evaluated translation directions, reaching the level of top commercial translation systems.These results use zero-shot settings; demonstrations could further improve performance.
4 Analysis
The analysis examines ChatGPT’s translation behavior through automatic word- and sentence-level measures, length and frequency buckets, and human error assessments. GPT-4 especially improves low-frequency word translation and produces fewer annotated errors than ChatGPT and Google Translate.
- 4.1 Automatic Analysis: ChatGPT performs worst on low-frequency words, while GPT-4 mainly addresses this shortcoming.The analysis uses F-measure across target-word frequency categories.
- 4.1 Automatic Analysis: ChatGPT performs worst on short sentences in the sentence-length analysis.Target sentences are divided into length categories with an average length of 23.2 tokens.
- 4.2 Human Evaluation: Human annotators classify under-translation, over-translation, and mis-translation, then rank outputs without system-identifying information.The ranking allows tied judgments when outputs are difficult to distinguish.
- 4.2 Human Evaluation: ChatGPT produces more over-translation and mis-translation errors than Google Translate but slightly fewer under-translation errors.The authors associate this pattern with greater likelihood of hallucinations.
- 4.3 Case Study: GPT-4 makes the fewest errors across all three error classes and receives the best human rankings.Examples include more context-appropriate terminology translations than Google Translate and ChatGPT.
5 Conclusion
This preliminary study finds that ChatGPT matches commercial systems on high-resource European languages but lags on low-resource or distant languages. Pivot prompting improves distant-language translation, and GPT-4 makes performance comparable to commercial products even for distant languages.
- ChatGPT competes with commercial translation products on high-resource European languages but lags significantly on low-resource or distant languages.It performs well on spoken language but worse than commercial systems on biomedical abstracts and Reddit comments.
- Pivot prompting noticeably improves translation performance for distant languages.
Limitations
The study is preliminary and constrained by limited evaluation coverage and randomness across repeated ChatGPT queries. It evaluates multilingual and robustness abilities but leaves other translation abilities for future work.
- Only 50 randomly selected samples from each test set are evaluated because of ChatGPT’s response delay.The authors describe this coverage as insufficiently comprehensive.
- Repeated queries can produce different results, so more reliable evaluation would repeat translations and report averages.
- The report focuses on multilingual translation and translation robustness, leaving constrained and document-level translation unevaluated.