Source-linked AI summary
Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting
Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, Furu Wei
TL;DR
LLMs perform unevenly across languages, especially for non-English and low-resource languages. The paper introduces cross-lingual-thought prompting (XLT), a generic language-independent template that stimulates cross-lingual and logical reasoning. Across seven multilingual benchmarks, XLT improves task performance and reduces performance gaps between languages, with over 10-point average gains on arithmetic reasoning and open-domain question answering.
Problem
LLMs do not have equal capability across languages and struggle particularly with non-English, low-resource, and extremely low-resource languages.
Method
XLT is a generic, language-independent prompt that guides models through cross-lingual thinking and logical task-solving steps without updating model parameters.
Results
Across seven multilingual benchmarks, XLT significantly improves performance across languages and settings while reducing the gap between average and best performance across languages.
Takeaways & Limitations
XLT improves multilingual reasoning, understanding, and generation performance and supports greater language democratization, including in non-English and low-resource languages.
Takeaways & Limitations
The evaluation covers only 27 languages and tests XLT on two GPT-based models and LLaMA-2-Chat; the template is written in English.
Abstract
from arXiv · showhide
Large language models (LLMs) demonstrate impressive multilingual capability, but their performance varies substantially across different languages. In this work, we introduce a simple yet effective method, called cross-lingual-thought prompting (XLT), to systematically improve the multilingual capability of LLMs. Specifically, XLT is a generic template prompt that stimulates cross-lingual and logical reasoning skills to enhance task performance across languages. We conduct comprehensive evaluations on 7 typical benchmarks related to reasoning, understanding, and generation tasks, covering both high-resource and low-resource languages. Experimental results show that XLT not only remarkably enhances the performance of various multilingual tasks but also significantly reduces the gap between the average performance and the best performance of each task in different languages. Notably, XLT brings over 10 points of average improvement in arithmetic reasoning and open-domain question-answering tasks.
1 Introduction
LLMs have uneven multilingual performance, especially for non-English and low-resource languages. XLT addresses this gap with a language-independent prompting method that improves multilingual task performance and narrows cross-language disparities.
- LLMs struggle with understanding and generating non-English languages, particularly in low-resource and extremely low-resource languages.
- Existing approaches based on data augmentation or fine-tuning are computationally expensive, while prompting-based in-context learning has been limited to monolingual tasks.
- XLT is a universal in-context learning method that uses a generic prompt to enhance multilingual capability across diverse tasks.
- XLT uses language-independent instructions that guide problem understanding, cross-lingual thinking, task analysis, execution, and output formatting.
- XLT was evaluated on seven multilingual benchmarks spanning reasoning, understanding, and generation across high-resource and low-resource languages.
- Over 10 points of average improvement was observed on both MGSM and MKQA, while XLT also reduced the gap between average and best performance across languages.
2 Cross-Lingual-Thought Prompting
XLT constructs a language-independent prompt from task and request metadata, then guides the model through cross-lingual reasoning, task solving, and output formatting. Few-shot XLT extends this process by constructing demonstrations with step-by-step outputs.
- 2.1 Construction of XLT: XLT uses an English template with placeholders filled from task and request information to elicit multilingual capability through cross-lingual thoughts.
- 2.1 Construction of XLT: Role assigning specifies the task category and task language to establish the model’s intended behavior.
- 2.1 Construction of XLT: Task inputting structures the request according to its task type so the model can comprehend the input.
- 2.1 Construction of XLT: Cross-lingual thinking rephrases the requested content in English as a pivot to support understanding of the original request.
- 2.1 Construction of XLT: Task analyzing converts the rephrased input into a task goal, while CoT task solving asks the model to work step by step.
- 2.1 Construction of XLT: Output formatting specifies the desired output type and optional constraints to obtain the exact answer format.
- 2.2 XLT for Few-shot Learning: For few-shot learning, XLT samples development examples and constructs demonstrations using formatted step-by-step outputs.
3 Experiments
The experiments evaluate XLT across seven multilingual benchmarks, multiple task types, languages, models, and prompting baselines. Results generally favor XLT in zero-shot and few-shot settings, with especially large gains on arithmetic reasoning and question answering.
- Tasks and Benchmarks: Seven benchmarks cover arithmetic reasoning, commonsense reasoning, natural language inference, paraphrase identification, question answering, summarization, and machine translation.
- Tasks and Benchmarks: The evaluation spans 27 languages, including both high-resource and low-resource languages.
- Tasks and Benchmarks: MGSM evaluates arithmetic reasoning with accuracy, while XCOPA evaluates cause-or-result commonsense reasoning across 11 languages with accuracy.
- Tasks and Benchmarks: MKQA uses token-overlap F1 across 10 selected languages, XL-Sum uses ROUGE-1 on six languages, and FLORES uses SacreBLEU across 12 translation directions.
- Baselines and Setup: XLT is compared with basic prompting, Chain-of-Thought prompting, and Translate-English prompting across zero-shot and few-shot settings.
3.2 Experimental Results
XLT improves multilingual task performance across languages and learning settings, while reducing performance disparities between languages. Its largest reported gains occur on arithmetic reasoning and open-domain question answering.
- XLT significantly improves performance across seven multilingual benchmarks in both zero-shot and few-shot settings.The evaluation covers reasoning, understanding, and generation tasks across languages.
- Nearly 20 points on average are gained on MGSM, while around 10 points on average are gained on MKQA.These correspond to arithmetic reasoning and open-domain question answering, respectively.
- XLT achieves significant improvements over three baseline methods across tasks and languages, except for a slight zero-shot drop on PAWS-X.The comparison includes two LLMs and both zero-shot and few-shot settings.
- XLT increases democratization scores in general, particularly for XCOPA and MKQA.The democratization score measures average performance relative to the best-performing language.
- XLT reduces performance gaps between languages, including on MGSM, XNLI, and PAWS-X.For these tasks, the baseline performance is consistently lower overall while the language gap is smaller.
3.3 Further Analysis
Further analyses show that XLT depends on the interaction of cross-lingual thinking, role assignment, chain-of-thought solving, instruction order, and wording. Few-shot demonstrations work best when their input-output format consistently matches XLT.
- Variant comparison: XLT surpasses almost all tested variants, supporting the effectiveness and reasonableness of the proposed method.The comparison is reported for the zero-shot gpt-3.5-turbo experiments in Table 3.
- Instruction effectiveness: Cross-lingual thinking yields more significant gains than the other ablated instructions.The analysis attributes this to activating cross-lingual thinking and allowing the model to use English knowledge for task solving.
- Instruction effectiveness: Removing role assignment impedes understanding of the ultimate goal for diverse multilingual tasks.This finding highlights task transferability within XLT.
- Instruction effectiveness: CoT task solving contributes to XLT performance by requiring step-by-step responses to complex instructions.
- Instruction order: Switching the designed instruction order causes an evident performance drop.The reported analysis emphasizes placing role assignment early and cross-lingual thinking before task analysis.
- Word choice: “Retell” performs better for five tasks, whereas “repeat” performs better for text summarization and machine translation.The authors find wording affects performance but report lower sensitivity to other prompt variants.
- Few-shot demonstrations: XLT few-shot learning outperforms all other demonstration variants, while XLT input without XLT output produces no improvement.The experiments use five demonstrations on MGSM with gpt-3.5-turbo.
- Few-shot demonstrations: Consistency between few-shot demonstration formats and zero-shot input-output formats is crucial.Using normal demonstrations decreases performance, and adding XLT only to demonstration inputs does not improve results.
4 Related Work
Prior work evaluates LLM capabilities and develops multilingual models through supervised training, pretraining, fine-tuning, and in-context learning. These approaches address multilingual processing, while multilingual generation remains more challenging.
- LLM evaluations examine accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency across natural language processing settings.
- ChatGPT performs well in high-resource languages but has limitations in other languages.
- Multilingual knowledge can be exploited and transferred between languages to improve model performance.
- Multilingual capability can be developed through supervised models covering multiple languages or pretrained models fine-tuned for cross-language transfer.
- LLMs enable direct multilingual task processing through in-context learning, but their scale requires substantial computational resources.
5 Conclusion
The paper proposes XLT as a generic prompt for multilingual understanding, reasoning, and generation. It reports reduced language performance gaps, while showing that instruction logic and word choice affect effectiveness.
- XLT is a generic prompt intended to enhance multilingual capability and reduce performance gaps across non-English and low-resource languages.
- XLT is generally applicable across tasks and languages, but instruction logic and word choice explicitly affect its effectiveness.
- Cross-language thinking is particularly effective within XLT.
- The authors hope generic prompting will motivate further development across a wider range of modalities and languages.
Limitations
The evaluation covered 27 languages, leaving much of the world’s linguistic diversity untested. XLT is English-based and was verified on only three GPT-based models.
- Evaluation was limited to 27 languages, which covers only a small fraction of the world’s languages.The authors attribute this constraint to benchmark limitations and OpenAI API cost.
- The XLT template is based on English, leaving task-language templates and improved language-specific instructions for future study.
- XLT’s generality was verified only on text-davinci-003, gpt-3.5-turbo, and LLaMA-2-Chat.The authors suggest evaluating it on additional models such as BLOOM and PaLM.
A.1 Results on Reasoning Tasks
Across arithmetic and commonsense reasoning benchmarks, XLT improves performance across resource levels and learning settings, with especially large gains for several low-resource languages.
- MGSM: XLT improves MGSM arithmetic reasoning for both models across low-resource and high-resource languages in zero-shot and few-shot settings.In few-shot evaluation, it also reduces the performance gap across languages.
- MGSM: For some high-resource MGSM languages, including German, Russian, French, and Spanish, XLT performance is comparable to English.
- XCOPA: XLT enhances XCOPA performance in both models and settings compared with basic prompting.Zero-shot gains are strongest for several relatively low-resource languages, while few-shot gains span both resource groups.
- XCOPA: Over 10 accuracy points are gained for gpt-3.5-turbo on XCOPA Swahili, Thai, Tamil, and Haitian Creole in zero- or few-shot comparisons.
A.2 Results on Understanding Tasks
On natural-language understanding benchmarks, XLT generally outperforms basic prompting, with particularly clear improvements for low-resource languages and some setting-dependent trade-offs.
- XNLI: In zero-shot XNLI, XLT significantly outperforms the basic prompt in all languages.Few-shot evaluation also shows significant improvements for high- and low-resource languages with both models.
- XNLI: For low-resource XNLI languages, XLT averages 9.4 additional accuracy scores with text-davinci-003 and 5.3 with gpt-3.5-turbo.
- PAWS-X: XLT outperforms basic prompting on PAWS-X across languages, particularly for low-resource languages under few-shot evaluation.
- PAWS-X: PAWS-X shows a slight average zero-shot drop for gpt-3.5-turbo on some high-resource languages, including English, German, and French.The authors infer that cross-lingual thinking may alter sentence meaning and hinder judgment.
A.3 Results on Generation Tasks
Across multilingual generation tasks, XLT improves open-domain question answering, summarization, and translation, with especially notable gains for under-resourced languages and cross-resource translation directions.
- MKQA: XLT significantly improves MKQA performance over basic prompting across all evaluated languages in both zero-shot and few-shot settings.It brings over 10 points of improvement for under-resourced languages such as Chinese, Japanese, Vietnamese, and Turkish.
- MKQA: XLT’s cross-lingual reasoning leverages English as a pivot for solving English open-domain questions across languages.
- XL-Sum*: XLT outperforms basic prompting on XL-Sum* across all languages in both zero-shot and few-shot settings.Few-shot examples further improve multilingual summary generation compared with zero-shot evaluation.
- FLORES*: XLT improves gpt-3.5-turbo translation by nearly 4.0, 2.8, and 3.3 BLEU points for th→gl, jv→zh, and zh→th, respectively.The reported gains span high-resource-to-low-resource, low-resource-to-high-resource, and low-resource-to-low-resource directions.
- Evaluation setup: The generation evaluation uses token-overlap F1 for MKQA, ROUGE-1 for XL-Sum*, and BLEU for FLORES*.The supplied table captions identify these metric assignments.