PaperlayerView exact revision on arXiv

Source-linked AI summary

Choosing the Right Language Mode at Inference Time for Multilingual Reliability

Ekata Mitra, Ameeta Agrawal

arXiv:2609.04653v1cs.CL

TL;DR

Multilingual models can reason less reliably in low- and mid-resource languages, motivating a controlled study of how translation scope and language mode affect accuracy and calibration. The paper evaluates target-only, English-only, and bilingual prompting with LLaMA and Qwen, then introduces RAAI for selective language assistance and adaptive reasoning. RAAI improves multilingual reasoning while reducing calibration error, especially in the lowest-resource settings.

  • Problem

    The paper asks how much translation multilingual models need for reliable reasoning and when additional bilingual context causes interference or overconfidence.

  • Method

    The study varies text scope and language mode in multilingual MCQA and introduces training-free RAAI with ECE-aware routing and risk-gated sequential reasoning.

  • Results

    RAAI improves multilingual reasoning accuracy and reliability across two model families, with the strongest benefits in low-resource settings.

  • Takeaways & Limitations

    Reliable multilingual inference benefits from selectively allocating English assistance and additional reasoning rather than using a fixed language mode.

  • Takeaways & Limitations

    The evaluation is limited to closed-set multilingual MCQA, two model families, nine languages, and a development-derived risk profile for SEQGATE.

Abstract

from arXiv · show

Multilingual large language models often struggle to reason in low- to mid-resource languages. Prior work has shown that translation can improve multilingual reasoning by helping models access stronger English-centric representations. This raises a central question: How much translation is needed for multilingual large language models to reason reliably, and when does more translation instead trigger interference and overconfidence? Using LLaMA and Qwen models, we run extensive experiments varying text scope and language mode (target-only, English-only, bilingual) to evaluate both accuracy and reliability. Our results reveal a clear trade-off: English context often improve understanding and recover errors caused by non-English comprehension, yet adding redundant bilingual context intensifies interference. We address this trade-off with Reliability-Aware Adaptive Inference (RAAI), a training-free test-time framework that (i) performs Expected Calibration Error (ECE)-aware routing and prompt fusion, and (ii) uses a mid-layer Risk Index (RI) to gate sequential reasoning, allocating compute only when it is likely to help and suppressing harmful bilingual redundancy. Across two model families, RAAI enhances accuracy by 25-37.7% on low-resource languages and lowers calibration error by approximately 3-6%, with the most pronounced benefits in the lowest-resource language tiers.

1 Introduction

The paper examines why multilingual reasoning lags in non-English languages and introduces a controlled framework for studying how text scope and language mode affect accuracy and reliability. It proposes RAAI, a training-free test-time method for selective English assistance and adaptive reasoning.

  • Multilingual reasoning gaps can cause misinterpretation, failed multistep reasoning, and unreliable confidence estimates in non-English settings.
  • English-centric internal representations may introduce semantic-grounding errors, representation drift, and language-specific output inconsistencies.
  • The study controls text scope and language mode in parallel multilingual MCQA to isolate their effects on multilingual failures and reliability.
  • The diagnostic framework varies five Belebele text scopes and three language modes, while evaluating accuracy, ECE, confidence, and entropy.
  • RAAI combines ECE-aware bilingual fallback with risk-gated sequential reasoning to allocate language assistance and computation selectively.

2 Related Work

Prior work links multilingual reasoning gaps to English-centric representations, selective translation, and calibration problems. This paper positions its contribution as a controlled comparison of matched language modes rather than a training-based system.

  • Multilingual models often use English-centric internal structures and language-specific activations, which can contribute to inconsistent non-English outputs.
  • Bilingual prompting and selective translation report reasoning gains, especially when English support is applied selectively rather than uniformly.
  • Intermediate layers can be better calibrated than final layers, motivating the use of layer-wise multilingual reliability signals.
  • Existing uncertainty-aware multilingual methods use confidence or related-language feedback, whereas this work triggers selective translation and fusion for high-risk inputs.
  • The experiments compare matched static T, EN, and TEN baselines while holding the model, examples, answer space, and scoring procedure fixed.

3 Experimental Setup

The study evaluates two model families on parallel multilingual MCQA datasets spanning nine languages and three resource tiers. It standardizes option scoring and reports accuracy alongside calibration and uncertainty measures.

  • Experiments use LLaMA 3.1-8B Instruct and Qwen 3-4B to test whether cross-lingual reasoning patterns generalize across model families.
  • Evaluation covers Belebele and MMLU-ProX(-Lite), with languages grouped into High, Mid, and Low resource tiers.
  • Belebele supports controlled translation and resource-tier comparisons, whereas MMLU-ProX(-Lite) stresses knowledge-intensive reasoning with ten answer options.
  • The decoding procedure forces digit-based option outputs and computes logit-lens probabilities for directly comparable predictions.
  • Accuracy measures selected-option correctness, while ECE measures confidence alignment with empirical accuracy; confidence and entropy quantify prediction certainty and uncertainty.

4 Analysis: Text scope×Language mode

Translation helps multilingual understanding unevenly across resource tiers and text scopes, but broader bilingual context can introduce interference and calibration costs. Layer-wise results further show that sharper later predictions are not necessarily more reliable.

  • Correctness analysis: Translation fixes more understanding errors in lower-resource languages, but reasoning failures persist across resource tiers.
  • Text scope: Broader translation reduces evidence-grounding failures while increasing bilingual interference, especially in low-resource languages.
  • Resource-tier effects: There is no universally best language mode: LLaMA gains little from English on Belebele, while Qwen benefits more in mid- and low-resource tiers.
  • Resource-tier effects: TEN often provides the best MMLU-ProX-Lite accuracy but can worsen ECE, revealing a trade-off between correctness and reliability.
  • Layer- and language-level patterns: Accuracy generally improves in later layers, whereas calibration is often best earlier, with confidence eventually rising faster than reliability.
  • Layer- and language-level patterns: Entropy continues declining into mid-to-upper layers, but sharper predictions are not necessarily better calibrated, particularly for lower-resource languages.

5 Reliability-Aware Adaptive Inference (RAAI)

RAAI adaptively combines reliability-aware bilingual fallback with risk-gated sequential reasoning. It uses calibrated layer signals and resource-tier risk to apply extra translation or compute selectively, avoiding unnecessary bilingual processing and instability.

  • Framework: RAAI is a training-free framework with two independently deployable components: selective bilingual fallback and risk-gated sequential reasoning.ROUTEGATE routes uncertain or disagreeing target-only predictions to bilingual inference, while SEQGATE escalates only high-risk groups.
  • RAAI-ROUTEGATE: ROUTEGATE first evaluates target-only predictions using calibrated layer signals, escalating to bilingual prompts when reliability is poor or predictions disagree.The fallback aims to lower ECE while avoiding bilingual prompting costs and possible interference on low-risk cases.
  • RAAI-ROUTEGATE: The layer ensemble selects the K layers with lowest ECE and provides an alternative prediction for comparison with the final layer.Layer-wise ECE is computed on a small stratified development set, using temperature-scaled probabilities and calibration bins.
  • RAAI-SEQGATE: SEQGATE escalates difficult cases to a bilingual sequential prompt only when resource-tier risk is high.The sequential prompt performs target-language reasoning, English reasoning over translated content, and reconciliation before producing the final answer.
  • RAAI-SEQGATE: The Risk Index identifies groups that are error-prone, poorly calibrated, and confidently wrong using reasoning-failure rate, worst layer-wise ECE, and predictive entropy.SEQ is applied only to groups exceeding a fixed risk threshold; its risk is computed offline from target-only inference without adding test-time cost.

6 Results and Discussion

RAAI improves multilingual reasoning through two complementary interventions: ROUTEGATE primarily reduces calibration error, while SEQGATE primarily improves accuracy in mid- and low-resource settings. Their selective, risk-based use is especially valuable where static language modes and baseline reasoning remain unreliable.

  • Main results: ROUTEGATE and SEQGATE outperform the best static baseline in several settings, with the largest gains in mid- and low-resource tiers.The comparison reports both accuracy and ECE across resource tiers, models, and benchmarks.
  • Low-resource tiers: +25.1 points for LLaMA and +37.7 points for Qwen are the Belebele-Low accuracy gains from SEQGATE.On MMLU-ProX-Lite-Low, the corresponding gains over T are +10.6 and +22.8, respectively.
  • Calibration: ROUTEGATE consistently reduces ECE relative to the target-only baseline across both models and datasets.On Belebele, ECE falls in every tier, including by 7.5 points in the mid-resource tier; on MMLU, the mid-resource decrease reaches 10 points.
  • Accuracy: SEQGATE delivers the largest accuracy gains in mid- and low-resource tiers by applying sequential bilingual reasoning only to Risk-Index-identified high-risk inputs.Across datasets, it improves low-resource scores by 10% for LLaMA and 22% for Qwen over T.
  • Accuracy-calibration trade-off: ROUTEGATE usually achieves the lowest ECE, whereas SEQGATE usually achieves the highest accuracy but often with higher ECE.The results therefore expose a direct accuracy-calibration trade-off between the two interventions.
  • Risk-based gating: Risk-based escalation is supported because RI attains AUROC=1.00 in the hardest low-resource slices where SEQGATE produces its largest improvements.SEQGATE is not uniformly advantageous and can degrade calibration relative to ROUTEGATE.
  • Qualitative analysis: Qualitative examples show that target-only can succeed where English-only fails, while bilingual prompting can rescue cases where both single-language modes fail.These examples support selective rather than uniform English intervention.

7 Conclusion

The paper argues that robust multilingual reasoning requires adaptive intervention rather than a single fixed prompting strategy. RAAI allocates language assistance and computation according to risk, using selective bilingual fallback and sequential reasoning for harder cases.

  • 7 Conclusion: RAAI adaptively allocates language assistance and computation based on risk instead of applying one fixed prompting strategy everywhere.ROUTEGATE selectively uses bilingual fallback, while SEQGATE reserves sequential bilingual reasoning for harder slices.

Limitations

The evaluation excludes translation-generation costs and is limited in task, model, language, and deployment scope. Translation quality, open-ended generation, broader language coverage, scaling, and new-domain profiling remain practical boundaries.

  • Translation costs: The cost analysis covers LLM inference but excludes the latency and expense of producing translations.Deployed EN, TEN, or SEQ systems would require cached translations or an external translation service.
  • Translation quality: Translation quality could affect both predictions and routing decisions.Comparisons among human translations, machine translations, and controlled translation perturbations remain an important extension.
  • Task scope: The evaluation is restricted to closed-set multilingual MCQA, so findings may not transfer directly to open-ended generation.Open-ended generation introduces uncertainty around semantic equivalence, output language, fluency, and sequence-level confidence.
  • Model and language scope: The study evaluates one model size from each of two families and nine languages, limiting scaling and language-level coverage.The Low tier contains only Urdu and Yoruba.
  • Deployment scope: SEQGATE uses a development-derived language–scope risk profile, so new languages or domains require a small profiling set.

Ethics Statement

The paper frames multilingual reasoning as an equity and responsible-use concern, particularly for low-resource and historically underserved communities. It reports calibration and risk measures alongside accuracy and states that no new user data or sensitive-content training was used.

  • Ethics Statement: The work targets equity across languages and dialects, particularly for low-resource and historically underserved communities.
  • Ethics Statement: The study is observational and inference-time, without collecting new user data or training models on sensitive content.
  • Ethics Statement: Because multilingual improvements may affect high-stakes domains, the paper reports calibration and risk measures in addition to accuracy.

B.1 Experimental Details

Experiments use LLaMA 3.1-8B Instruct and Qwen3-4B across nine languages from Belebele and corresponding MMLU-ProX-Lite variants. Results are reported across static prompt controls, translation settings, correctness-taxonomy breakdowns, and resource tiers.

  • Experimental setup: Experiments use LLaMA 3.1-8B Instruct and Qwen3-4B across nine languages selected from Belebele’s 126-language collection.MMLU uses the STEM subset matching the selected Belebele language variants.
  • Translation analyses: Full-resolution plots show per-language accuracy, confidence, ECE, and entropy trajectories for representative question- and choice-translation settings.
  • Correctness taxonomy: Figures 15–17 provide remaining correctness-taxonomy breakdowns for LLaMA and Qwen across MMLU-ProX-Lite and Belebele.
  • Statistical stability: Statistical intervals characterize stability over evaluated examples in the Low tier, which contains Urdu and Yoruba.

D.2 Risk Index Ablation and Routing-Budget Sensitivity

The Risk Index combines failure prevalence, calibration, and confident prediction, while routing-budget experiments show that broader intervention can improve accuracy but is dataset-dependent. The evaluation also separates offline calibration choices from online routing decisions and excludes translation-system costs from reported inference costs.

  • Risk Index construction: The Risk Index uses a normalized low-entropy certainty term rather than raw entropy.This qualification specifies how the entropy factor is constructed.
  • Risk Index ablation: The full Risk Index product conservatively combines failure prevalence, miscalibration, and confident prediction.No single ablated variant dominates every metric.
  • Routing-budget sensitivity: Broader SEQGATE routing often improves accuracy, but its relationship with performance is dataset-dependent and not monotonic in every setting.Routing-budget sensitivity reports accuracy and ECE changes relative to target-only inference.
  • Routing protocol: ECE selects low-ECE layers offline, while online routing uses per-example confidence and final-versus-ensemble disagreement.The answer-scoring rule remains unchanged.
  • Efficiency and cost: Reported inference cost includes processing longer bilingual or sequential prompts but excludes external machine-translation cost.Deployment with EN, TEN, or SEQ would require cached English fields or a runtime translation system.
Loading 2609.04653v1…