Source-linked AI summary
More Capable, Less Faithful: A Multilingual Analysis of Mathematical (Un)Solvability Detection in LLMs
Maria-Eleni Zoumpoulidi, Nikolaos Xiros, Georgios Paraskevopoulos
TL;DR
Multilingual solvability detection is poorly understood because prior analyses largely focused on English and did not separate internal Solvability Belief from its verbalization. The paper extends ReliableMath to French and Greek, probes internal representations, and compares them with textual judgments, finding broadly universal belief representations but lower faithfulness in higher-resource languages such as English.
Problem
Prior solvability-detection analyses were limited largely to English, leaving multilingual internal belief and verbalization insufficiently studied.
Method
The paper extends ReliableMath to French and Greek and combines multilingual prompting, hidden-state probing, and textual-verdict annotation.
Results
Models generally perform best in English but multilingual models are more faithful to internal Solvability Belief in underrepresented languages than English.
Takeaways & Limitations
Dominant-language strength yields stronger raw capability but not more trustworthy self-reports.
Takeaways & Limitations
The analysis has limited language coverage and should be extended to additional high-resource and typologically diverse lower-resource languages.
Abstract
from arXiv · showhide
Solvability detection is one of the most challenging aspects of mathematical reasoning for Large Language Models (LLMs). While prior work has studied this capability extensively, these analyses have been limited to English. Consequently, it remains unclear whether multilingual failures arise from differences in internal Solvability Belief or from language-dependent failures to express it. To address this gap, we introduce the first multilingual benchmark of paired solvable and unsolvable mathematical problems, extending ReliableMath to French and Greek. Using this, we train multilingual probes predicting Solvability Belief and analyze the solvability detection capabilities of state-of-the-art LLMs behaviorally, representationally, and in terms of faithfulness. We find that Solvability Belief is encoded as a largely universal, language-agnostic feature, and that higher-resource languages such as English, despite achieving stronger mathematical reasoning performance, exhibit lower solvability-detection faithfulness.
1 Introduction
Multilingual solvability detection remains underexplored despite extensive English-focused research and growing multilingual mathematical-reasoning work. The paper addresses whether cross-language failures reflect internal Solvability Belief differences or failures to verbalize that belief.
- Prior multilingual mathematical-reasoning research has emphasized accuracy on solvable problems, while multilingual solvability detection remains underexplored.
- English studies distinguish internal Solvability Belief from verbalization and show that models may recognize unsolvable problems without abstaining.
- The paper investigates whether multilingual failures arise from different internal Solvability Beliefs or language-dependent failures to express them.
- The authors extend ReliableMath into the first multilingual benchmark of paired solvable and unsolvable mathematical problems in French and Greek.
- The study analyzes multilingual mathematical reasoning and unsolvability detection behaviorally, representationally, and through faithfulness to textual outputs.
2 Methodology
The methodology combines multilingual benchmark translation, prompted reasoning and hidden-state extraction, probing of internal solvability representations, and annotation of textual solvability verdicts.
- The pipeline has four stages: benchmark translation, CoT generation and hidden-state extraction, solvability probing, and textual-verdict annotation.
- ReliableMath solvable problems and unsolvable counterparts were translated into French and Greek, then human-validated for preserved mathematical content and solvability type.
- Standard prompts request step-by-step solutions, whereas aware prompts additionally permit explicit statements that problems are unsolvable.
- Hidden states are represented by 20 uniformly sampled token vectors from the layer with the highest probing performance.
- An L1-regularized logistic-regression probe predicts Solvability Belief from pooled hidden states using ground-truth solvability labels as a proxy.
- An LLM judge assigns the dominant solvability verdict in reasoning traces, with validation against human annotations on 100 samples.
3 Experiments
Experiments evaluate multiple state-of-the-art and language-specialized LLMs on a translated paired solvability benchmark spanning English, French, and Greek.
- The experiments query Qwen3, Llama-3.1, and Gemma4 models, plus Greek- and French-specialized Llama variants.
- The translated ReliableMath dataset contains 313 solvable problems and 1,102 unsolvable variants from school, college, and olympiad-level sources.
4 Results
Models generally solve problems and encode Solvability Belief best in English, while a universal probe captures shared structure and faithfulness reverses this capability pattern. Prompting and language adaptation raise abstention but can also increase false refusals.
- 4.1 Models Achieve Strongest Mathematical Reasoning On Their Dominant Language: Gemma-4-31B-it and both Qwen3 variants outperform Llama-based models across English, French, and Greek.
- 4.1 Models Achieve Strongest Mathematical Reasoning On Their Dominant Language: Accuracy is generally best in English and declines through French to Greek across models.
- 4.2 Internal Solvability Belief Is Strongest In The Dominant Language, But Highly Universal: Solvability Belief probe AUC is highest in English and declines monotonically through French to Greek for every model and probe type.
- 4.2 Internal Solvability Belief Is Strongest In The Dominant Language, But Highly Universal: Pooling languages into a universal probe matches or exceeds per-language probe performance for every model, indicating a largely shared Solvability Belief subspace.
- 4.3 Language Adaptation Raises Abstention: Under standard prompting, only the Qwen3 variants abstain substantially, at approximately 25%, while they perform better on unsolvable detection in Greek than English.
- 4.3 Language Adaptation Raises Abstention: Aware prompting sharply raises abstention for every model, but language adaptation increases both true abstentions and false refusals in the target language.
- 4.4 Models Are Least Faithful In Their Dominant Language: Multilingual Gemma and Qwen models are more faithful in underrepresented languages than English, reversing the capability and internal-belief pattern.
- 4.4 Models Are Least Faithful In Their Dominant Language: Aware prompting raises faithfulness for every model, with larger gains in French or Greek than English on unsolvable problems.
5 Conclusions
The study extends ReliableMath to French and Greek to examine solvability detection behaviorally and through internal representations. Models perform best in English and worst in Greek, yet are more faithful to internal Solvability Belief in underrepresented languages.
- The authors introduce the first multilingual ReliableMath benchmark, extended to French and Greek, for paired solvable and unsolvable problems.
- Models show a consistent capability hierarchy, with performance highest in English and lowest in Greek.
- Faithfulness follows the opposite pattern: multilingual models more faithfully express internal Solvability Belief in underrepresented languages than in English.
- The findings suggest that a model’s dominant language provides stronger raw capability but not more trustworthy self-reports.
Limitations
The analysis provides informative insights but is limited by its language coverage. Future work will extend the study to additional high-resource and typologically diverse lower-resource languages.
- The analysis would benefit from broader language coverage.
- Future work will examine additional high-resource languages such as Chinese and typologically diverse lower-resource languages.
- The authors also plan methods to better align textual judgments with internal Solvability Beliefs.
Ethical Considerations
The work is analytical and uses translated mathematical benchmark items without releasing a model or introducing deployment-oriented methods. Supplementary prompts and selected probing layers document the evaluation setup.
- The study evaluates and probes existing publicly available models rather than releasing a new model or deployment method.
- The benchmark directly translates ReliableMath into French and Greek while preserving the original items without adding, removing, or reweighting them.
- The dataset contains mathematical problems without personal, sensitive, or offensive content.
- Translation, chain-of-thought, and LLM-judge prompts are documented in Tables 7, 9, 10, and 11.
- Table 4 reports the layer selected for each model based on highest probing performance.
E.1 Significance of Solvability Belief Results
Significance analyses compare language effects and probe types using paired bootstrap confidence intervals and two-proportion z-tests. English often has stronger solvability-related capability, while non-English languages frequently show higher faithfulness in aware-prompt unsolvable judgments.
- 1000-resample, 95%-level paired bootstrap intervals quantify AUC differences across parallel English, French, and Greek problems.
- English significantly outperforms Greek for Qwen3-4B (+3.8, CI [1.3, 6.5]), Qwen3-30B (+2.8, CI [0.5, 5.2]), Gemma-4-31B (+3.8, CI [1.8, 6.0]), and Llama-Krikri-8B (+7.9, CI [4.8, 10.7]).
- The universal probe is never significantly worse than the per-language probe across the 16 tested model-language combinations.
- Two-proportion z-tests find significant pooled faithfulness differences for Qwen3-4B (p < 0.001) and Llama-3.1-8B (p < 0.01), with opposite English-direction effects.
- For aware-prompt unsolvable-only judgments, non-English advantages are highly significant for Krikri-8B in Greek (p < 10^-10), Alpaca-8B in French (p < 10^-7), and Qwen3-4B in French (p < 0.05) and Greek (p < 0.001).
F Human Validation of the LLM Judge for Solvability Verdict Annotation
The LLM judge’s solvability-verdict annotations were checked against human annotations on a balanced subset of 100 reasoning traces.
- 100 reasoning traces were manually annotated to validate the LLM judge’s verbal solvability verdicts.The subset was balanced across English, French, and Greek; models; and solvable versus unsolvable problem types.
- The validation used the same criterion and prompt for human annotations and the LLM judge.Labels reflected whether the model’s final verbal behavior treated each problem as solvable or unsolvable.
- Table 5 reports agreement between the human annotations and the LLM judge on the 100-trace validation subset.
G Human Validation of the Translation
The French and Greek ReliableMath translations were manually compared with their English sources to assess preservation of mathematical content and naturalness.
- 200 translated problems were manually reviewed: 100 French and 100 Greek.Two authors reviewed the translations, with each author annotating 100 problems for one target language.
- The review assessed whether mathematical content was faithfully preserved and whether translations were fluent and natural.Each translation was compared against its original English source.
- Table 6 reports human validation agreement for French and Greek translations on 100 problems per language.
- The annotation interface displayed each original English problem beside its translation for meaning and mathematical-content validation.Annotators marked whether the translation preserved the source’s meaning and mathematical content.
- Table 8 provides an example of a contradictory unsolvable problem alongside its Greek and French translations.The contradiction combines 1000 residents with “each of the 900 residents,” making the problem underdetermined.