PaperlayerView exact revision on arXiv

Source-linked AI summary

Is Translation All You Need? A Study on Solving Multilingual Tasks with Large Language Models

Chaoqun Liu, Wenxuan Zhang, Yiran Zhao, Anh Tuan Luu, Lidong Bing

arXiv:2403.10258v3cs.CL

TL;DR

Multilingual LLM evaluation has been dominated by English-centric models and NLP tasks, leaving translation effectiveness across real-world and culture-related queries insufficiently examined. This study compares prompting strategies across tasks, languages, and both English-centric and non-English-centric LLMs, finding that translation often helps English-centric models on NLP tasks but native-language prompting can work better for culture-related tasks.

  • Problem

    Prior evidence on translation for multilingual LLMs is concentrated on specific NLP tasks and English-centric models, with limited evaluation of real user queries and non-English-centric LLMs.

    Source: §1 · pp. 1–2

  • Method

    The study compares translation-based and native prompting strategies across multilingual NLP tasks, real user queries, and multiple English-centric and non-English-centric LLMs.

    Source: §1 · p. 2, §2.1.2 · p. 3, §3.3 · p. 6

  • Results

    Translation can achieve the best results in multiple NLP categories, but its effect on real user queries varies by language and LLM, with native prompts often better for certain culture-related settings.

    Source: §1 · p. 2, §3.3 · p. 7

  • Takeaways & Limitations

    Multilingual evaluation should cover broader tasks and model families because translation is not optimal in every scenario and native prompts can preserve cultural and linguistic nuances.

    Source: §1 · p. 2, §5 · p. 9

    The culture-related evaluation uses two Chinese-optimized non-English-centric LLMs and the multiple-choice M3Exam dataset, which may limit applicability.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) have demonstrated multilingual capabilities, yet they are mostly English-centric due to the imbalanced training corpora. While prior works have leveraged this bias to enhance multilingual performance through translation, they have been largely limited to natural language processing (NLP) tasks. In this work, we extend the evaluation to real-world user queries and non-English-centric LLMs, offering a broader examination of multilingual performance. Our key contribution lies in demonstrating that while translation into English can boost the performance of English-centric LLMs on NLP tasks, it is not universally optimal. For culture-related tasks that need deep language understanding, prompting in the native language proves more effective as it better captures the nuances of culture and language. Our experiments expose varied behaviors across LLMs and tasks in the multilingual context, underscoring the need for a more comprehensive approach to multilingual evaluation. Therefore, we call for greater efforts in developing and evaluating LLMs that go beyond English-centric paradigms.

1 Introduction

LLMs are multilingual but often English-centric, motivating broader evaluation of translation-based prompting across NLP tasks, real user queries, and non-English-centric models. The study finds that translation helps in many NLP settings, whereas native-language prompts can better capture cultural and linguistic nuances.

Research gap

  • Existing studies largely focus on specific NLP tasks and English-centric LLMs, leaving real-world multilingual user queries underexplored.The paper frames this as a need for more nuanced evaluation across multilingual contexts.Source: §1 · pp. 1–2

Approach

  • The study compares translate-test prompting with baselines on multilingual NLP tasks, then evaluates real user queries and non-English-centric LLMs.Translate-test translates tasks into English and may optionally translate responses back into the original languages.Source: §1 · p. 2

Findings

  • For real user queries, translation effects depend on the language and LLM; native prompts can be more effective for advanced models and certain languages.The results associate native prompting with better preservation of culture- and language-related nuances.Source: §1 · p. 2

Contributions

  • Translation into English remains a strong baseline for multilingual NLP tasks, and the study identifies factors affecting performance.The work compares multilingual prompting strategies rather than assuming one universally best approach.Source: §1 · p. 2
  • The evaluation expands beyond prior NLP-focused studies to include actual user queries and non-English-centric LLMs.These additions address limitations identified in earlier multilingual evaluations.Source: §1 · p. 2

Implications

  • The findings expose gaps in current multilingual evaluations and motivate broader benchmarks covering more tasks and LLMs.The paper emphasizes that evaluation should not remain limited to English-centric paradigms.Source: §1 · p. 2

2 Translation for NLP Tasks

This section evaluates native-language, English-instruction, chain-of-thought, and translation-based prompting across six multilingual benchmarks, languages, and LLMs. Translation generally performs strongly for NLP tasks, while its benefits depend on translation quality, language characteristics, and model capability.

2.1.1 Tasks

  • The evaluation covers six benchmarks spanning reasoning, understanding, and generation across 24 non-English languages, using accuracy, token-overlap F1, and ROUGE-1 as appropriate.The benchmarks are MGSM, XCOPA, XNLI, PAWS-X, MKQA, and XL-Sum.Source: §2.1.1 · pp. 2–3

2.1.2 Models

  • The study compares ChatGPT and Llama-2-70B-Chat as primary models, with additional experiments on Mistral-7B-Instruct, Llama-2-13B-chat, and bloomz-7b1.The primary evaluation therefore includes both closed-source and open-source LLMs, while appendix results extend the comparison to smaller models.Source: §2.1.2 · p. 3

2.1.3 Prompting Strategies

  • Prompting strategies range from native or English instructions and chain-of-thought to XLT and direct translation with Google Translate or NLLB.Translation-based methods translate questions into English before solving, while native and mixed-language strategies retain the original query.Source: §2.1.3 · p. 3

2.2 Main Results

  • TRANS-GOOGLE achieves the highest overall performance across models and tasks, with consistently strong results for both high- and low-resource languages.It does not always achieve the top score, but remains a strong baseline across the evaluation.Source: §2.2 · p. 4, Table 1 · p. 4
  • English instructions generally improve performance, chain-of-thought helps strong models and reasoning tasks, and basic prompts can better suit weaker models or directly answerable tasks.EN-COT underperforms TRANS-GOOGLE on average, while its relative performance differs between high- and low-resource languages.Source: §2.2 · p. 4

2.3 Analysis and Discussions

  • Higher BLEU translation quality generally corresponds to better MGSM performance, while MGSM accuracy significantly correlates with syntactic distance from English but not the other measured distances.Google Translate has the highest translation quality except for Japanese; languages with greater syntactic distance tend to perform worse.Source: §2.3 · pp. 4–5, Table 2 · p. 5

3 Translation for Real User Queries

The study evaluates translation-based prompting for multilingual NLP tasks, real user queries, and culture-related tasks across English-centric and non-English-centric LLMs. Translation often helps English-centric models, but native-language prompting can better preserve cultural and linguistic nuance.

3.1 Experiment Setup

  • The evaluation compares original queries with Google-translated queries, translating outputs back and scoring responses from 1 to 10 with GPT-4o5.It samples 10 languages, generally using 100 requests per language, with fewer samples for Romanian, Ukrainian, and Norwegian because of source-data limits.Source: §3.1 · pp. 5–6

3.2 Main Results

  • Translation helps Llama-2-70B-Chat across all languages, whereas ChatGPT favors original queries for several high-resource languages and varies across low-resource languages.For Llama-2-70B-Chat, improved understanding after translation outweighs potential information loss; ChatGPT’s behavior differs by language.Source: §3.2 · p. 6
  • Between 30% and 74% of queries per language require cultural knowledge, showing that the real-user data contains substantial culture-related content.This motivates evaluating translation beyond standard NLP benchmarks, which may omit culture-specific elements.Source: §3 · p. 5, §3.2 · p. 6

3.3 Analysis and Discussions

  • The study uses M3Exam language and social-science questions, whose language coverage differs because not every subject is available in every language.The selected data are real-world questions rather than translations from English and can require local cultural knowledge.Source: §3.3 · pp. 6–7, Figure 6 · p. 7
  • On M3Exam, translation does not always improve ChatGPT, helps Llama-2-70B-Chat in every language, and hurts Chinese performance for Qwen1.5-72B-Chat and Yi-34B-Chat.The results attribute differences to translation errors, language understanding, and translationese; translation can still help the non-English-centric models on some other languages.Source: §3.3 · pp. 6–7, Figure 6 · p. 7
  • For Chinese NLP tasks, TRANS-GOOGLE achieves Yi-34B-Chat’s best average score, while Qwen1.5-72B-Chat performs best with EN-COT and both models favor NATIVE-BASIC on XL-Sum.Translation benefits are smaller than for ChatGPT and Llama-2-70B-Chat; on MGSM Chinese, the TRANS-GOOGLE–NATIVE-BASIC gaps are 2.8% and 8% for the two models, versus 37.2% and 16% for ChatGPT and Llama-2-70B-Chat.Source: §3.3 · p. 7, Table 3 · p. 8
  • Qwen1.5-7B-Chat processes Chinese hidden representations mainly in Chinese, while Llama-2-7B-Chat represents them in multiple other languages.The authors hypothesize that native-language processing avoids information loss, making it more suitable for multilingual tasks.Source: §3.3 · pp. 7–8, Figure 7 · p. 8

4 Related Work

Prior multilingual evaluations and prompting methods largely target standard NLP benchmarks and English-centric models. This study extends that line of work to real-world queries, cultural knowledge, and non-English-centric LLMs.

Multilingual Evaluation

  • Earlier multilingual evaluations mainly cover standard NLP tasks, overlooking real-world scenarios and cultural knowledge relevant to practical applicability.The cited evaluations span many datasets and languages but remain centered on benchmark-style tasks.Source: §4 · p. 8

Multilingual Prompting Strategies

  • Translate-test and related prompting methods often improve multilingual NLP performance, but their effectiveness in real-world applications remains uncertain.Their success is tied to the English-centric nature of the evaluated LLMs, motivating broader testing across tasks and model types.Source: §4 · p. 8

5 Conclusion

The study evaluates translation across traditional NLP benchmarks, real user queries, and culture-related tasks. Its results show that translation is useful but not universally optimal, with native-language prompting often preferable for non-English-centric and culture-sensitive settings.

5 Conclusion

  • Translation-based methods are simple and effective for overcoming limitations of English-centric LLMs, but they are not optimal in every multilingual scenario.The conclusion emphasizes comprehensive evaluation across tasks and models rather than relying on translation alone.Source: §5 · p. 8
  • Native-language prompting performs better for non-English-centric LLMs and culture-related tasks because it captures language-specific subtleties and cultural intricacies.The conclusion identifies multilingual proficiency as a central challenge for this setting.Source: §5 · p. 9
  • The findings motivate continued development and evaluation of strong multilingual LLMs beyond English-centric paradigms.This recommendation follows the study’s broader comparison across task types and model families.Source: §5 · pp. 8–9

Limitations

The evaluation is constrained by incomplete strategy coverage, Chinese-optimized models for culture-related tasks, multiple-choice M3Exam data, and samples of at most 500 items per language.

Limitations

  • The study could not evaluate all prompting strategies, selected two LLMs optimized for Chinese on culture-related tasks, and used only multiple-choice M3Exam questions.These choices may limit how broadly the findings apply across prompting methods, languages, models, and benchmark formats.Source: Limitations · p. 9
  • Sampling was capped at 500 items per language, so results may not be directly comparable with studies evaluating entire benchmarks.The cap was imposed to manage computational constraints while retaining a broad evaluation scope.Source: Limitations · p. 9

A.1 Translation for NLP Tasks

This section reports the experimental setups and results for multilingual NLP tasks.

  • The experiments examine multilingual NLP tasks using varied prompting and translation strategies.Source: §A.1 · p. 12
  • The section focuses on how translation affects performance across multilingual NLP evaluations.Source: §A.1 · p. 12
  • The reported analyses provide setup and result details for the NLP-task experiments.Source: §A.1 · p. 12

A.1.1 Details about NLP Benchmarks

The evaluation spans six multilingual benchmarks covering reasoning, inference, paraphrase identification, question answering, and summarization, with task-specific language coverage and metrics.

  • Six benchmarks evaluate arithmetic, commonsense, inference, paraphrase, question answering, and summarization across multiple languages.Source: §A.1.1 · p. 12

Arithmetic Reasoning

  • MGSM evaluates grade-school mathematical problem solving across 10 languages using accuracy.Source: §A.1.1 · p. 12

Commonsense Reasoning

  • XCOPA tests identifying a premise’s cause or effect across 11 languages using accuracy.Source: §A.1.1 · p. 12

Natural Language Inference

  • XNLI evaluates whether hypotheses are entailed, contradicted, or neutral across 15 languages using accuracy.Source: §A.1.1 · p. 12

Paraphrase Identification

  • PAWS-X asks whether sentence pairs are paraphrases across seven languages using accuracy.Source: §A.1.1 · p. 12

Question Answering

  • MKQA evaluates short answers to open-domain questions in nine selected languages using token-overlap F1.Source: §A.1.1 · p. 12

Summarization

  • XL-Sum evaluates concise summaries of news articles in five selected languages using ROUGE-1.Source: §A.1.1 · p. 12

A.1.2 More LLMs for Experiment

Beyond ChatGPT and Llama-2-70B-Chat, the experiments evaluate three additional language models with different multilingual and instruction-tuning backgrounds.

  • The study adds Mistral-7B-Instruct, Llama-2-13B-chat, and bloomz-7b1 to the NLP-task evaluation.Source: §A.1.2 · p. 12
  • Mistral-7B-Instruct (v0.2) is the instructed version of Mistral-7B.Source: §A.1.2 · p. 12
  • Llama-2-13B-chat is a chat model in the Llama-2 family, while bloomz-7b1 was fine-tuned on multiple tasks including multilingual tasks.Source: §A.1.2 · p. 12

A.1.3 More Details about Prompt Strategies

The appendix details prompting templates, expanded NLP results, cultural-query evaluation, and layerwise analyses of larger models.

A.1.3 More Details about Prompt Strategies

A.1.4 Additional Results

  • XLT is competitive on reasoning tasks but less effective for generation, with ChatGPT refusing 26.4% of XL-Sum questions.Source: §A.1.4 · p. 13
  • For open-source models, instruction-following problems also degrade XLT performance, although refusal behavior was not observed.Source: §A.1.4 · p. 13

A.2 Translation for Real User Queries

  • Cultural-query evaluation uses GPT-4o to score responses and identify whether requests require local cultural knowledge.Source: §A.2 · p. 13, Figure 8 · p. 13, Figure 9 · p. 17
  • For culturally knowledgeable queries, ChatGPT favors native prompts in high-resource languages but translation often helps low-resource languages, whereas Llama-2-70B-Chat favors translation across languages.Source: §A.2 · p. 13, Figure 10 · p. 19

A.2.1 Additional Results

  • Additional judging with GPT-4o, Claude-3.5-Sonnet, and Gemini-Pro-1.5 confirms stronger direct prompting for ChatGPT in Japanese and Chinese, while Llama-2-70B-Chat favors translated prompts.Source: §A.2.1 · p. 13, Figure 11 · p. 21

A.3 Layerwise Language Distribution in Larger Model

  • Larger Llama-2-70B-Chat models retain diverse hidden-state languages, while Qwen1.5-72B-Chat retains Chinese and English until its final layers and is more Chinese-oriented.Source: §A.3 · pp. 13–14, Figure 12 · p. 21

A.1.4 Additional Results

Loading 2403.10258v3…