Source-linked AI summary
Language Models are Multilingual Chain-of-Thought Reasoners
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, Jason Wei
TL;DR
The paper asks how well language models perform complex reasoning across languages, where existing multilingual benchmarks provide limited evidence. It introduces and evaluates MGSM with multilingual prompting strategies, finding strong reasoning across languages and extension to commonsense and semantic-judgment tasks. The authors also report important scope boundaries around model selection and whether XL-WiC benefits from genuine reasoning.
Problem
Existing multilingual benchmarks usually test simple reasoning, leaving complex multilingual reasoning abilities unclear.
Method
The paper introduces MGSM from 250 manually translated GSM8K problems in ten languages and evaluates GPT-3 and PaLM with multilingual chain-of-thought prompting.
Results
PaLM-540B solves more than 40% of MGSM problems in every investigated language, averages 55% in its best setting, and reaches 89.9% on XCOPA.
Takeaways & Limitations
Multilingual reasoning abilities extend across arithmetic, commonsense reasoning, and word-in-context semantic judgment, including underrepresented languages such as Bengali and Swahili.
Takeaways & Limitations
The experiments focus on GPT-3 and PaLM because of their notable few-shot performance, while it remains unclear whether XL-WiC requires reasoning that benefits from intermediate steps.
Abstract
from arXiv · showhide
We evaluate the reasoning abilities of large language models in multilingual settings. We introduce the Multilingual Grade School Math (MGSM) benchmark, by manually translating 250 grade-school math problems from the GSM8K dataset (Cobbe et al., 2021) into ten typologically diverse languages. We find that the ability to solve MGSM problems via chain-of-thought prompting emerges with increasing model scale, and that models have strikingly strong multilingual reasoning abilities, even in underrepresented languages such as Bengali and Swahili. Finally, we show that the multilingual reasoning abilities of language models extend to other tasks such as commonsense reasoning and word-in-context semantic judgment. The MGSM benchmark is publicly available at https://github.com/google-research/url-nlp.
1 INTRODUCTION
Existing multilingual benchmarks largely test simple reasoning, leaving complex multilingual reasoning unclear. This paper introduces MGSM and shows that large language models can reason across languages, with chain-of-thought prompting extending beyond arithmetic.
- Existing multilingual benchmarks usually require only simple reasoning steps, leaving complex multilingual reasoning abilities unclear.
- MGSM bridges English chain-of-thought reasoning and multilingual NLP by manually translating GSM8K problems into ten typologically diverse languages.
- More than 40% of MGSM problems are solved in every investigated language by PaLM-540B with intermediate reasoning steps, including Bengali and Swahili.
- 55% average solve rate across languages is achieved by PaLM in the best setting.
- 89.9% on XCOPA establishes a new state-of-the-art result using few-shot multilingual chain-of-thought examples.
2 THE MGSM BENCHMARK
MGSM extends GSM8K into a multilingual arithmetic-reasoning benchmark built from 250 manually translated problems. Its design spans ten typologically diverse languages and preserves consistent Arabic-numeral answers.
- MGSM is described as the first multilingual benchmark for evaluating language-model arithmetic reasoning.
- The benchmark uses the first 250 GSM8K test examples, each requiring two to eight reasoning steps according to its official solution.
- Arabic numerals are retained across languages to facilitate cross-lingual prediction.
- The ten target languages span eight language families and different representation levels in standard pretraining datasets.
- Professional native-speaking translators manually translated the selected examples, with additional quality verification and checks against machine-translation overlap.
3 MULTILINGUAL CHAIN-OF-THOUGHT PROMPTING
The paper compares direct answers with native-, English-, and translation-based chain-of-thought prompting, alongside several few-shot exemplar strategies. These settings isolate how reasoning language, translation, and exemplar language affect multilingual prompting.
- Experimental design: The experimental combinations pair exemplar selection with solution strategies, as illustrated through prompt formats and example outputs.
- Solution strategies: DIRECT prompting asks models to predict answers without intermediate reasoning steps.
- Solution strategies: NATIVE-COT predicts reasoning steps in the problem’s original language, measuring language-specific understanding and solving.
- Solution strategies: EN-COT predicts the chain of thought in English regardless of the problem language.
- Solution strategies: TRANSLATE-EN translates problems into English before solving them with English chain of thought.
- Exemplar selection: Few-shot prompts can use native, English, or generic multilingual question-and-solution exemplars.
4 EXPERIMENTS ON MGSM
On MGSM, explicit intermediate reasoning substantially improves multilingual performance, especially for PaLM-540B, whose abilities strengthen with scale and transfer across languages and prompt settings.
- Main results: Intermediate reasoning steps substantially outperform direct answer prediction across languages for both GPT-3 and PaLM.
- Main results: English chain-of-thought consistently matches or exceeds native-language chain-of-thought, making it a useful multilingual reasoning baseline.
- Further analysis: Underrepresented languages average 44.9% accuracy versus 47.9% for high-resource languages, a 3% gap in PaLM results.
- Further analysis: Multilingual reasoning generally improves with model scale, emerging substantially at text-davinci-001 for GPT-3 and PaLM-62B for PaLM.
- Further analysis: PaLM-540B generally benefits from more few-shot exemplars, although accuracy does not increase strictly monotonically for every language.
- Further analysis: With English chain-of-thought, multilingual exemplars outperform English-only exemplars across all non-English languages, including languages absent from the examples.
5 EXTENSION TO OTHER MULTILINGUAL REASONING BENCHMARKS
The authors test whether multilingual reasoning extends beyond MGSM to causal commonsense and word-in-context judgments. PaLM-540B reaches a new XCOPA high while chain-of-thought does not improve XL-WiC over direct prediction.
- Setup: PaLM-540B and Codex are evaluated on XCOPA and XL-WiC using direct prediction and English chain-of-thought prompting.
- XCOPA: On XCOPA, PaLM-540B performs noticeably better than other models on underrepresented languages including Estonian, Haitian Creole, and Swahili.
- XL-WiC: With four examples, PaLM-540B matches or exceeds XLM-R Large on 6 of 12 investigated XL-WiC languages.
- XL-WiC: The authors note that prompt formats may be suboptimal and that XL-WiC rationales are straightforward and example-specific.
6 RELATED WORK
Prior multilingual research established broad task performance and some reasoning evaluation, but existing benchmarks generally required relatively simple reasoning. This work positions MGSM as a benchmark for multilingual multi-step arithmetic reasoning.
- Multilingual pretrained models have performed strongly across languages on information extraction, commonsense, temporal, and other NLP tasks.
- Earlier multilingual prompting studies found that English prompts with non-English examples could produce strong few-shot performance.
- Existing multilingual benchmarks covered tasks including question answering, natural language inference, and bilingual lexicon induction.
- These benchmarks generally required relatively simple reasoning processes rather than multilingual multi-step reasoning.
- The paper introduces MGSM to evaluate multilingual arithmetic reasoning and reports strong cross-lingual generalization of PaLM.
7 CONCLUSION
The paper introduces MGSM, analyzes multilingual reasoning in large language models, and reports complex reasoning across languages, including underrepresented ones. It also finds multilingual chain-of-thought prompting effective for commonsense reasoning on XCOPA.
- MGSM is the first multilingual benchmark designed to evaluate language-model arithmetic reasoning abilities.
- MGSM extends GSM8K with 250 examples in ten typologically diverse languages.
- The analysis covers GPT-3 and PaLM across MGSM and multiple multilingual benchmarks.
- Large-scale models appear to perform complex multi-step reasoning across multiple languages, including languages represented in less than 0.01% of training corpora.
- Multilingual chain-of-thought prompting outperforms the previous best XCOPA model by 13% on average.
A DETAILS OF MGSM EXPERIMENTS
The MGSM experiments document the exemplars and prompts used for GPT-3 and summarize the prompting settings in Table 2.
- The experiments include details about the number of GPT-3 exemplars used.
- The prompting settings are summarized in Table 2, with detailed prompts described in an accompanying appendix section.
A.1 NUMBER OF EXEMPLARS FOR EACH LANGUAGE
GPT-3’s 2048-token input limit constrains the number of multilingual exemplars in several languages because BPE produces more tokens for underrepresented languages. The appendix presents the prompts used in these experiments.
- BPE tokenizes underrepresented languages, especially those using a different alphabet from English, into more tokens.
- GPT-3’s API limit of 2048 input tokens prevents 6-shot prompting in Russian, Chinese, Japanese, Thai, Telugu, and Bengali.
- For affected languages, the experiments use the maximum possible number of exemplars under the token limit.
- The appendix provides prompts for the MGSM experiments, including TRANSLATE-EN with EN-COT and ENGLISH-EXEMPLARS.
B DETAILED MGSM PERFORMANCE
This section identifies the prompt materials and supporting table used to report detailed MGSM, XCOPA, and XL-WiC analyses.
- The EN-COT template uses English chain-of-thought exemplars while solving a problem written in German.The dotted lines distinguish few-shot exemplars from the target question and expected answer, but are excluded from experiments.
- The NATIVE-COT template uses chain-of-thought in the question language while solving a German problem.Its dotted lines similarly separate exemplars from the target question and expected answer and are not included experimentally.
- Table 8 reports the detailed performances corresponding to Figures 4 and 5.
- The paper presents chain-of-thought prompt details for XCOPA and XL-WiC tasks.
- The XCOPA prompt contains four randomly selected multilingual validation examples with author-written rationales.Under direct prompting, answers are provided directly and rationales are omitted.
XL-WiC
The XL-WiC materials present paired sentences and ask whether a marked word is used in the same sense in both.
- The example contrasts “Approach a task” with “To approach the city.”
- XL-WiC asks whether the marked word “approach” is used in the same sense across two sentences.
- The experiments include separate multilingual and English-language chain-of-thought prompts.