PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts

• PolyMath spans four difficulty levels, with 125 problems per language at each level, and includes examples across its difficulty structure.The levels are defined by Thought Depth and Knowledge Breadth; Figure 2 illustrates the structure and examples.Source: §1 · p. 3, Figure 2 · p. 5
• Each PolyMath problem has 18 parallel language versions, covering over 75% of the world’s native speakers and major language families.The benchmark includes both high-resource and low-resource languages.Source: §1 · p. 4
• Language experts calibrate each translation rather than relying directly on LLM-generated outputs, aiming for precise terminology and logical clarity.Source: §1 · p. 4
• Qwen-3-235B-A22B-Thinking scored 54.6 and Gemini-2.5-pro 52.2, with accuracy about 40% at the highest level and language differences up to 10 points.The reported language differences occur even at low accuracy settings.Source: §1 · p. 4
• Reasoning models show lower input-output language consistency than non-reasoning models, especially in their thinking, and decreased consistency can accompany improved reasoning performance.Non-reasoning models typically adhere strictly to the input language.Source: §1 · p. 4
• Forcing reasoning LLMs to think and answer in English improves reasoning performance, while forcing them to follow the input language typically worsens it.Source: §1 · p. 4
• Reasoning LLMs show slow-thinking behavior across languages, whereas non-reasoning models have a significantly larger cross-language gap in reasoning length.Source: §1 · p. 4

2.1 Difficulty Level Partition

• Difficulty levels combine Thought Depth and Knowledge Breadth, addressing inconsistent and overly fine-grained partitions based only on learning stages or problem sources.Learning-stage and problem-source categories remain useful for organizing problems within each level.Source: §2.1 · p. 6

2.2 Data Collection

• PolyMath contains 500 problems—125 per level—sourced from public benchmarks, official repositories, Olympiads, and frontier mathematics, with levels assigned by experts and confirmed by two others.Low-level problems come from MGSM (Shi et al., 2023); the top level includes 100 Olympiad problems and 25 from HLE (Phan et al., 2025).Source: §2.2 · pp. 6–7

2.4 Benchmark Statistics

• The 18 languages cover several major language families and 75% of the world population.Problem length generally rises across levels with minimal language differences; answer forms diversify, while Natural Language Coverage rises but remains around 50%.Source: §2.3 · p. 7, §2.4 · p. 8

2.3 Multilingual Translation Annotation

• GPT-4o pre-translations are calibrated by experts because mathematical translation requires specialized knowledge, and annotators find content or fluency disagreements in almost all languages.Experts extract key terms, then language experts check term precision and formula transfer; any content disagreement marks an unusable sample.Source: §2.3 · pp. 7–8, Table 3 · p. 8

2.4 Benchmark Statistics

• Problem domains remain diverse at every level, while embeddings separate low-level problems from the others but show overlap among the remaining three levels.The authors attribute this overlap partly to embeddings insufficiently capturing differences in thought depth, motivating human involvement in collection and level assessment.Source: §2.4 · pp. 8–9, Figure 3 · p. 9

2.5 Benchmark Score: Difficulty-Weighted Accuracy

• Difficulty-Weighted Accuracy doubles problem weights at each level, from 1 for low-level to 8 for top-level, giving greater contribution to harder correct answers.Under the default weights, eight low-level correct answers contribute as much as one top-level correct answer.Source: §2.5 · p. 9

3.1 Setup

• The evaluation covers eight non-reasoning and eight reasoning LLMs, measuring pass@1 ACC by level and language and calculating an overall DW-ACC score.Models are tested 16 times under fixed hyperparameters, with average@16 reported.Source: §3.1 · pp. 9, 11

3.2 Main Results

• Qwen-3-235B-A22B-Thinking leads across levels, while Gemini-2.5-pro and Qwen-3-235B-A22B-Thinking achieve nearly 40% ACC at the top level.Some non-reasoning models, including Llama-3.3-70B-Instruct, fail almost completely at that level; reasoning models may underperform non-reasoning models at the low level.Source: Table 5 · p. 10, §3.2 · p. 11
• All models lose accuracy as difficulty rises, but reasoning models generally decline more gradually than non-reasoning models.For example, Gemini-2.0-flash-thinking drops from 87.3 to 22.3 ACC across levels, while ChatGPT-4o-latest drops from 91.6 to 13.7.Source: Table 5 · p. 10, §3.2 · p. 11
• Language gaps remain substantial as difficulty increases, with ranges typically around 10% at the three higher levels and nearly 20% for Qwen-QwQ-32B.These differences exceed the reported 0.5–1.5% ACC run-to-run fluctuations.Source: Table 5 · p. 10, §3.2 · p. 11
• Stronger reasoning models often show larger language gaps, but Qwen-3-235B-A22B-Thinking combines strong performance with a relatively small gap.The paper attributes this exception largely to multilingual slow-thinking alignment during post-training (Yang et al., 2025).Source: Table 5 · p. 10, §3.2 · p. 11

4.1 Input-Output Language Consistency

• Non-reasoning LLMs had near-perfect language consistency, while reasoning LLMs often showed around 40% consistency in thinking and answers, with substantial model- and language-level variation.Claude-3.7-sonnet-thinking had around 40% thinking consistency but around 90% answer consistency; Qwen-QwQ-32B was near-perfect in English, Chinese, Japanese, Korean, and Russian but close to zero in many other languages.Source: §4.1 · p. 12, Figure 4 · p. 13
• For Qwen-QwQ-32B and Deepseek-R1-671B, lower language consistency strongly correlated with better reasoning accuracy, whereas non-reasoning models showed no significant consistency-accuracy correlation.The authors suggest that inconsistent responses, often in English or Chinese, may better evoke these models’ slow-thinking abilities; they also note that language used for thinking and answering can influence performance.Source: §4.1 · pp. 12–13, Table 6 · p. 13

4.2 Language Control

• Forcing English gave the best overall performance and minimized language disparities, while matching the query language yielded the poorest performance and often amplified cross-lingual variance.For Qwen-QwQ-32B, English control improved Bangali (bn) from 36.8 to 39.0, Portuguese (pt) from 39.4 to 47.5, and Russian (ru) from 40.1 to 47.7; query-language control reduced Arabic (ar) from 40.0 to 25.9 and Bangali (bn) from 36.8 to 32.8.Source: §4.2 · p. 14, Table 7 · p. 14
• English instructions were generally followed in over 90% of thinking outputs and nearly 100% of answers, but query-language instructions were often not followed and could reduce accuracy.Even when query-language control improved Qwen-QwQ-32B thinking consistency for German (de) from 0.6 to 80.2, Swahili (sw) from 1.8 to 55.2, and Italian (it) from 0.8 to 97.6, accuracy fell for each language.Source: §4.2 · p. 14, Table 7 · p. 14, Table 9 · p. 15

4.3 Thinking Length Across Languages

• As difficulty increased, non-reasoning models’ thinking lengths tended to stabilize, while reasoning models’ lengths increased and varied substantially across models.At the highest level, OpenAI-o1-mini and o3-mini typically generated 6k–8k tokens, Qwen-QwQ-32B and Deepseek-R1-671B generated 10k–20k, and Claude-3.7-sonnet-thinking reached 20k–30k.Source: §4.3 · p. 16, Figure 5 · p. 16, Figure 6 · p. 17
• At the highest level, reasoning models’ maximum-to-minimum thinking-length ratios were 1.21, 1.45, 1.37, 1.25, and 1.21, while non-reasoning models reached 6.30, 7.92, and 2.32.Reasoning models did not consistently have their longest thinking in any one language; non-reasoning models’ extreme lengths were usually concentrated in one or two languages. The authors speculate that English slow-thinking capability or low-resource-language data scarcity may contribute to these patterns.Source: §4.3 · pp. 16–17
• At low difficulty, longer thinking tended to accompany degraded performance, but the relationship became less evident as difficulty rose and was inconsistent across languages.Only Deepseek-R1-671B and Qwen-QwQ-32B among reasoning models showed relatively strong positive correlations; among non-reasoning models, ChatGPT-4o-latest showed a clear positive trend, while Llama-3.3-70B-Instruct and Qwen-2.5-Max tended negative.Source: §4.3 · p. 18

Multilingual Mathematical Reasoning Benchmarks

• PolyMath matches advanced LLMs’ reasoning level while offering high data quality and scale, unlike easier MGSM and MSVAMP or fully LLM-translated, small-sample MT-AIME.The comparison covers multiple benchmark dimensions; the passage describes MGSM as a translated version of GSM8K and says MT-AIME’s limited per-language samples introduce high randomness.Source: §5 · p. 18, Table 10 · p. 18

Multilingual Research Challenges in Current LLMs

• Language mixing, language alignment, and low-resource language support remain multilingual LLM challenges, which PolyMath’s findings and analyses also support.The Deepseek-R1 technical report notes unresolved language mixing; recent surveys identify alignment and low-resource support as future directions.Source: §5 · p. 18

6 Conclusion

• PolyMath introduces a multilingual mathematical reasoning benchmark with multiple intelligence levels.The authors present it as a resource to advance multilingual reasoning LLMs.Source: §6 · p. 18
• The experiments identify language-related issues in reasoning LLMs: performance varies across languages, input-output language inconsistencies occur, and thinking lengths differ.The thinking-length differences are observed across languages among various LLMs.Source: §6 · pp. 18–19
• Explicit language control may improve reasoning performance.The conclusion describes these performance benefits as potential.Source: §6 · p. 19

A.1 Annotator Background

• The team prioritized second-language experts with linguistics degrees and specialized translation experience, recruiting native speakers when suitable experts were difficult to find.Source: §A.1 · p. 22
• The first author, a Chinese computer-science Ph.D. candidate with strong mathematical and competition experience, handled mathematical-term extraction and Chinese calibration.Source: §A.1 · p. 22
• Calibration used experienced language experts for Arabic, French, Italian, Japanese, Korean, Spanish, Thai, and Vietnamese; graduate language specialists for German, Indonesian, Portuguese, and Russian; and math-background native speakers for Bengali, Malay, Swahili, and Telugu.Source: §A.1 · p. 22

A.2 Annotation Guidance

• Translated text had to preserve all English information, remove redundant additions, and allow direct transliteration of names without requiring specific phrasing.Source: §A.2 · p. 23
• Optional submission fields recorded major term errors, fluency problems, and other remarks, with brief reasons when relevant.Source: §A.2 · p. 23
• The platform showed each English source and terms needing attention; some China-based annotators also received calibrated Chinese translations, but English remained the gold standard.Source: §A.2 · p. 23
• Annotators were instructed to translate mathematical terms accurately, make problem logic smooth and concise, and migrate formula blocks without alteration.They also flagged term and fluency issues for later statistical analysis.Source: §A.2 · p. 23

B.1 Specific Data Source

• Table 11 documents the data sources used for each PolyMath difficulty level.Source: Table 11 · p. 24

B.2 Detailed Metadata of All Languages

• Table 12 presents PolyMath metadata across all languages.Source: Table 12 · p. 24

B.3 Data Domain

• Low-level questions are all K–12 mathematics and need no classification; middle-level questions are divided into specific knowledge-point subdomains, while high- and top-level questions are not further subdivided.The detailed domain statistics are shown in Table 13 for the medium level and Table 14 for the high and top levels.Source: §B.3 · pp. 24–25, Table 13 · p. 25, Table 14 · p. 25

C.1 Model Citation and Source

• Table 15 lists paper citations and URL source links for all models used in the paper.Source: Table 15 · p. 26
• The dagger symbol marks closed-source models.Source: Table 15 · p. 26

C.2 Main Prompts

• Figure 8 presents instruction prompts appended after the input query in the main experiments.Source: Figure 8 · p. 26

C.3 Language Control Prompts

• Figure 9 presents the language-control prompts, distinct from Figure 8’s main-experiment instructions.Source: Figure 8 · p. 26, Figure 9 · p. 27

C.4 Sampling Details

• To reduce instability and repetition associated with greedy decoding, the authors use sampling-based decoding instead.The passage attributes the concern about greedy decoding to Guo et al. (2025).Source: §C.4 · p. 27
• For each model, language, and difficulty level, the authors run 16 trials with identical hyperparameters on 125 samples per trial and report average accuracy.The reported measure is called average16.Source: §C.4 · p. 27
• To preserve accuracy granularity in 0.8-point increments, the authors round the mean number of correct answers across trials to the nearest integer, then divide by the total number.Each trial contains 125 samples, so each problem corresponds to 0.8% accuracy.Source: §C.4 · p. 27
• Across 16 trials, most accuracy standard deviations are 0.5–1.5, equivalent to under two questions’ variation per test.The paper considers this fluctuation reasonable given reasoning models’ sensitivity to sampling and hyperparameters; it reports standard deviations by language and difficulty level in Tables 16–19.Source: §C.4 · p. 27, Table 16 · p. 27, Table 17 · p. 27

D.1 Consistent Input-Output Language

• The examples show problems posed in Swahili, Chinese, Korean, and French, and in each case the displayed answer uses the same language as its problem.Source: §D.1 · pp. 28, 29, 31, 32
• Qwen-QwQ-32B’s Japanese example keeps the problem, thinking, and answer all in Japanese.Source: §D.1 · p. 33
• 2023 attempts are stated as the worst-case minimum for finding the monster-free column and reaching the final row.The worst case is 2022 failed attempts followed by success on the 2023rd attempt.Source: §D.1 · p. 36
• Trying a monster-filled column fails, while choosing the unoccupied column reaches the final row and ends the game.Trying distinct columns in turn finds the unoccupied column by the 2023rd attempt in the worst case.Source: §D.1 · pp. 36–37
• The board has 2024 rows and 2023 columns, with 2022 monsters, leaving exactly one column without a monster.There is one monster in each of the 2022 interior rows and at most one per column.Source: §D.1 · p. 37

D.2 Inconsistent Input-Output Language

• Qwen-QwQ-32B (output length 2996) received a Japanese problem, reasoned in Chinese, and answered in Japanese.Source: §D.2 · p. 37
• Across these examples, the thinking language differs from the input language; the answer matches the input only in the Japanese case.Source: §D.2 · pp. 37, 39
• Deepseek-R1-671B received a Spanish problem, but both its thinking and answer were in English.Source: §D.2 · p. 39
• The example pairs a Korean problem and Korean answer with English reasoning.It illustrates an input–reasoning language mismatch while the answer language matches the problem language.Source: §D.2 · p. 45

• 10 is the reported value of a + b + c, with a = 5, b = −8, and c = 13.The derivation gives s_{k+1} = 5s_k − 8s_{k−1} + 13s_{k−2} for k ≥ 2.Source: §D.2 · p. 49
• The Korean-language continuation states the same recurrence and coefficients, then verifies k = 2 gives s3 = 44 and a + b + c = 10.Source: §D.2 · p. 50