Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting

• LLMs have imbalanced capabilities across languages, and non-English performance is especially challenging in low-resource settings among the world’s over 7,000 languages.The vast majority of languages are low-resource or extremely low-resource.Source: §1 · pp. 1–2
• Data augmentation and fine-tuning (Chen et al., 2021, 2022) are computationally expensive, while in-context prompting is described as limited to monolingual tasks (Sanh et al., 2022).Source: §1 · p. 2
• XLT uses a generic, language-independent prompt without parameter updates, guiding models through problem understanding, cross-lingual thinking, task analysis, execution, and output formatting.It can also use an LLM-generated few-shot demonstration.Source: §1 · p. 2
• Over 10 points of average gain on MGSM and MKQA accompany improvements across seven benchmarks in zero-shot and few-shot settings and reduced gaps between average and best language performance.Source: §1 · pp. 2–3

2.1 Construction of XLT

• XLT is an English-written template that elicits multilingual capability through cross-lingual thought, using six sequential logical instructions and seven task-specific placeholders.Its design aims to emulate how humans handle multilingual tasks.Source: §2.1 · p. 3
• The opening instructions assign a task-and-language role, then present the request as a structured task input, such as premise and hypothesis in natural language inference.Source: §2.1 · p. 3
• XLT prompts cross-lingual thinking by rephrasing the request in English, with “retell” or “repeat” suggested as useful keywords in practice.The rephrasing is intended to help the model understand the request in its native language and knowledge.Source: §2.1 · p. 3
• After rephrasing, the prompt specifies the task goal and asks the model to solve it step by step, using intermediate outputs to aid reasoning.Source: §2.1 · p. 4
• The final instruction regularizes answers by specifying an output type and, optionally, constraints such as length or language.Source: §2.1 · p. 4

2.2 XLT for Few-shot Learning

• For few-shot learning, the method samples development examples, obtains zero-shot XLT responses, and uses only response-aligned requests and model responses as demonstrations.The demonstrations can also be corrected or designed to better align with the instruction logic.Source: §2.2 · p. 4

3 Experiments

• XLT is evaluated on multiple LLMs and NLP tasks to test its language-independent generality.The benchmarks cover reasoning, understanding, and generation across high- and low-resource languages.Source: §3 · p. 4

3.1.1 Tasks and Benchmarks

• The seven benchmarks cover 27 languages ordered by frequency, including eight whose frequency is below 0.1%.The underrepresented languages listed are Bengali, Tamil, Galician, Urdu, Telugu, Javanese, Haitian Creole, and Southern Quechua.Source: §3.1.1 · p. 4
• MGSM (Shi et al., 2023) tests grade-school arithmetic, while XCOPA (Ponti et al., 2020) tests commonsense cause-and-effect choices; both use accuracy.Each benchmark covers 11 languages; XCOPA draws them from 11 language families.Source: §3.1.1 · pp. 4–5
• XNLI (Conneau et al., 2018) evaluates entailment, contradiction, or neutrality across 15 languages, and PAWS-X (Yang et al., 2019) evaluates paraphrase identification across seven; both use accuracy.Source: §3.1.1 · p. 5
• MKQA (Longpre et al., 2021) uses token-overlap F1 on a 10-language subset, after unanswerable or imprecise questions are removed.The benchmark covers 25 languages overall; the selected subset comprises German, English, Spanish, French, Japanese, Russian, Thai, Turkish, Vietnamese, and Chinese.Source: §3.1.1 · p. 5
• XL-Sum* uses 250 sampled test examples per language and ROUGE-1, while FLORES* uses 200 examples per language and SacreBLEU across 12 translation directions.XL-Sum* selects six languages; FLORES* includes high-to-high, high-to-low, and low-to-low resource directions.Source: §3.1.1 · p. 5

3.1.2 Baselines

• The baselines are English basic prompting, Chain-of-Thought prompting, and Translate-English; XLT applies its proposed multi-instruction template.Translate-English is not used for generation tasks, which require output in the target language.Source: §3.1.2 · p. 5

3.1.3 LLMs

• Evaluations use GPT-3.5 models and LLaMA-2-Chat, with comparable existing results from other LLMs; inference uses greedy search and extracts labels after “Answer format:”.Few-shot evaluations use five examples, except XL-Sum*, which uses three due to input-length constraints; the five parallel benchmarks use semantically equivalent instances across languages.Source: §3.1.1 · p. 5, §3.1.3 · p. 6, §3.1.2 · p. 6

Multilingual Capability

• XLT benefits LLaMA-2-Chat on MGSM, though its gains are smaller than those for GPT-based models, primarily because of weaker multi-step instruction-following.The compatibility results are reported in Table 7.Source: §3.2 · p. 6

Language Democratization

• The democratization score measures each language’s performance as a percentage of the best-performing language on that task, averaged across languages.The paper averages scores across settings and models for its Figure 1(b) summary.Source: §3.2 · p. 6

Multilingual Capability

• Nearly 20 points on average in MGSM and around 10 points on average in MKQA are XLT’s reported gains; it significantly improves over three baselines across two LLMs and all tasks and settings, except a slight zero-shot drop on PAWS-X.Table 1 summarizes average scores across seven benchmarks and both settings.Source: §3.2 · p. 6, Table 1 · p. 7

Language Democratization

• Higher democratization scores are observed in general, particularly for XCOPA and MKQA; XLT also improves performance across multiple languages and reduces gaps on MGSM, XNLI, and PAWS-X.Table 2 presents task democratization by language for zero-shot and few-shot settings.Source: §3.2 · p. 6, Table 2 · p. 7

3.3.1 Ablation of XLT

• XLT surpasses almost all tested variants, supporting the effectiveness and reasonableness of the proposed method.The variants were evaluated with gpt-3.5-turbo under the zero-shot setting.Source: §3.3.1 · p. 7, Table 3 · p. 8
• Cross-lingual Thinking contributes more than the other ablated instructions, while Role Assigning and CoT Task Solving also support XLT performance.The analysis attributes these effects to using English knowledge for multilingual tasks, recognizing diverse task goals, and solving complex instructions step by step.Source: §3.3.1 · pp. 7–8, Table 3 · p. 8
• Switching the designed instruction order causes a performance drop; placing Role Assigning later may confuse the model, and Cross-lingual Thinking before Task Analyzing is crucial.The proposed ordering reflects the authors’ considered process for solving multilingual problems using LLMs’ English task-solving abilities.Source: §3.3.1 · p. 8
• Rephrasing keywords affect XLT performance, with “repeat” better for summarization and translation and “retell” more suitable for the other five tasks.The authors favor a unified template while allowing keyword fine-tuning for specific tasks.Source: §3.3.1 · p. 8

3.3.2 Effectiveness of XLT Few-shot Learning

• XLT few-shot learning outperforms all tested demonstration variants, while normal demonstrations reduce performance and XLT inputs alone do not improve results.Demonstration input-output formats should align with the zero-shot format; consistency is crucial.Source: §3.3.2 · pp. 8–9, Table 4 · p. 8

4.1 LLM Capability Understanding

• LLM evaluations find stronger performance in high-resource than low-resource languages, motivating this paper’s assessment of reasoning, understanding, and generation across languages.Prior studies report limitations for low-resource and non-Latin-script languages, while this paper’s evaluations likewise indicate high-/low-resource capability differences.Source: §4.1 · p. 9

4.2 Multilingual Task Processing

• Cross-lingual knowledge can transfer between languages, but multilingual generation is especially challenging for low-resource and non-English target languages.The related work cites Devlin et al. (2019), Conneau et al. (2020), Raffel et al. (2020), Ouyang et al. (2021), and Chi et al. (2021) on transferable multilingual knowledge, and Ma et al. (2021) and Liu et al. (2020) on generation challenges.Source: §4.2 · p. 9
• Multilingual task processing has used supervised multilingual training and fine-tuning for cross-language transfer, while LLMs enable direct multilingual in-context learning.The cited approaches include Costa-jussà et al. (2022) for supervised multilingual processing, Chen et al. (2021, 2022) for fine-tuning, and Brown et al. (2020) and Ahuja et al. (2023) for in-context learning.Source: §4.2 · p. 9
• Because LLM training requires substantial computation, researchers explore parameter-free prompting methods, and this work proposes a generic, language-independent prompt to improve multilingual capability.Examples include few-shot prompting (Vilar et al., 2022), automatic prompt learning (Shin et al., 2020), task-instruction prompting (Ye et al., 2023), and chain-of-thought prompting (Wei et al., 2022c).Source: §4.2 · p. 9

• XLT is proposed to enhance multilingual capability and reduce performance gaps across languages in understanding, reasoning, and generation tasks, including non-English and low-resource languages.Source: §5 · p. 9
• Although XLT is generally applicable across tasks and languages, instruction logic and word choice affect its effectiveness, and cross-language thinking is particularly effective.Source: §5 · p. 9
• The authors hope the work inspires research on generic prompting so LLMs can encompass a wider range of modalities and languages.Source: §5 · p. 9

• 27 languages were tested, which the authors say only scratch the surface of the world’s languages, partly due to benchmark and OpenAI API cost limitations.Source: Limitations · p. 10
• Because XLT is based on English, the authors leave open whether task-language templates could improve performance and how instructions should be constructed for each language.Source: Limitations · p. 10
• The method was verified only on text-davinci-003, gpt-3.5-turbo, and LLaMA-2-Chat, leaving its generality on models such as BLOOM and PaLM untested.Source: Limitations · p. 10

A.1 Results on Reasoning Tasks

• XLT significantly improves both models’ MGSM arithmetic reasoning in zero-shot and few-shot settings while reducing the performance gap across languages.For some high-resource languages, performance is comparable to English.Source: §A.1 · p. 13
• The authors hypothesize that prior supervised fine-tuning with arithmetic chain-of-thought samples may help XLT activate gpt-3.5-turbo’s arithmetic reasoning ability directly.Source: §A.1 · p. 13
• On XCOPA, XLT improves both models over basic prompting in both settings, with few-shot gains spanning high- and low-resource languages.In zero-shot, it improves several lower-resource languages but underperforms the baseline for zh and it.Source: §A.1 · p. 13
• Over 10 accuracy points of improvement are reported for gpt-3.5-turbo on XCOPA in sw, th, ta, and ht, especially supporting XLT’s effectiveness for lower-resource languages.Source: §A.1 · p. 13

A.2 Results on Understanding Tasks

• 9.4 accuracy scores for text-davinci-003 and 5.3 for gpt-3.5-turbo are the average XNLI improvements XLT achieves on low-resource languages th, bg, hi, and ur.In zero-shot XNLI, XLT significantly outperforms the basic prompt in all languages; in few-shot settings, both models show significant improvements over the basic prompt on high- and low-resource languages.Source: §A.2 · p. 13
• XLT outperforms the basic prompt on PAWS-X in all languages, particularly for low-resource languages in few-shot settings.For some high-resource languages (en, de, and fr), zero-shot performance drops slightly on average compared with gpt-3.5-turbo; the authors infer from intermediate outputs that cross-lingual thinking may alter sentence meaning and hinder judgment.Source: §A.2 · p. 13

A.3 Results on Generation Tasks

• Over 10 points of improvement for under-resourced languages on MKQA accompanies XLT’s gains over the basic prompt across languages in zero- and few-shot settings.The authors suggest cross-lingual thinking shared with English helps leverage LLM ability on open-domain QA.Source: §A.3 · p. 13, Table 11 · p. 16
• On XL-Sum*, XLT outperforms the basic prompt across languages in zero- and few-shot settings, and few-shot improves summary generation while reversing the basic-prompt model ranking.The authors suggest that providing examples can guide multilingual summarization.Source: §A.3 · pp. 13–14, Table 12 · p. 16
• Nearly 4.0, 2.8, and 3.3 BLEU points of improvement are reported for gpt-3.5-turbo on th→gl, jv→zh, and zh→th, respectively.XLT improves zero-shot results across the 12 non-English FLORES* directions, with more notable benefits for high-to-low, low-to-high, and low-to-low translations; few-shot evaluation was left for future work.Source: §A.3 · p. 14, Table 13 · p. 17
• The Spanish example includes an instruction to repeat the entire text in English, as well as source text about China’s Solar City.The excerpt says the city remains the only Solar City in China and the world, and notes that Huang dislikes being called “El rey sol.”Source: §A.3 · p. 29
• Figure 23 presents a Spanish XL-Sum* example with XLT under few-shot prompting, alongside the instruction to produce a one-sentence Spanish summary.The prompt also asks for step-by-step summarization in no more than two sentences.Source: Figure 23 · p. 29
• Figures 24 and 25 show a jv→zh FLORES* example under zero-shot prompting, contrasting the basic prompt with XLT.The supplied captions identify the prompting conditions but do not report comparative output quality.Source: Figure 24 · p. 30, Figure 25 · p. 30