PaperlayerView exact revision on arXiv

Source-linked AI summary

Disentangling Question, Response, and Verifier Languages in Multilingual P(True) Self-Evaluation

Jewon Yeom, Jaewon Sok, Seonghyeon Park, Jeongjae Park, Hwiyeong Lee, Taesup Kim

arXiv:2610.32144v1cs.CL

TL;DR

Multilingual confidence evaluations often conflate question, requested-response, and verifier languages, limiting comparisons across settings. We vary these languages independently across models and benchmarks, finding benchmark-dependent accuracy effects, calibration shifts, and consequential scorer differences.

  • Problem

    Multilingual confidence evaluations seldom separate question, requested-response, and verifier languages, although each may affect accuracy, discrimination, or calibration.

    Source: §1 · p. 1

  • Method

    A six-language factorial design varies question, requested-response, and verifier languages across ten open-weight models and three benchmarks, holding each generated response fixed across verifier languages and measuring first-token P(True).

    Source: §1 · pp. 1–2

  • Results

    Accuracy effects depend on benchmark; matching response language improves discrimination but not calibration, verifier language shifts ECE by up to 0.04 without changing accuracy, and judge labels shrink the MMMLU English–Hindi gap from 0.198 to 0.038.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    Comparisons require reporting all three languages, realized response language, and per-benchmark scorer validation.

    Source: §1 · p. 2

    The results concern one first-token P(True) estimator, ten open-weight models from 3B to 14B parameters, and one prompt template per benchmark; other estimators, larger or closed models, and prompt formats may behave differently.

    Source: Limitations · p. 9

Abstract

from arXiv · show

When a language model is asked whether its own answer is correct, three languages are involved: that of the question, that requested for the answer, and that of the verifier instruction asking for the judgment. Multilingual confidence evaluations seldom separate the three. We vary them independently in a six-language factorial design with ten open-weight models (3B-14B) on LSQA, MMMLU and TriviaQA, holding each answer fixed across verifier languages and measuring confidence with a first-token P(True) probe. The three languages move different parts of the evaluation: which of them governs accuracy depends on the benchmark, matching the response language to the question or verifier language sharpens discrimination but not calibration, and the verifier language alone shifts calibration error by up to 0.04 without changing accuracy. The scorer matters as well: an unvalidated judge would have shrunk the MMMLU English-Hindi accuracy gap from 0.198 to 0.038. Multilingual confidence results, model rankings included, are therefore comparable only when all three languages are reported and the scorer is validated.

1 Introduction

Multilingual confidence evaluations often conflate question, requested-response, and verifier languages, although each may affect different evaluation metrics. This study varies all three independently and finds benchmark- and metric-dependent effects, with scorer choice also changing conclusions.

1 Introduction

  • Because evaluations seldom report question, response, and verifier languages separately, results from different language settings may not be comparable.Applications can mix them, such as using an English verifier to audit answers to Hindi questions.Source: §1 · p. 1
  • Six model pairs reverse accuracy order between the all-English and fully localized settings, and language effects differ by benchmark and metric.Figure 1 averages over five target languages and three benchmarks; the benchmark comparison does not identify a causal effect of task regime or response format.Source: Figure 1 · p. 1, §1 · pp. 1–2
  • A six-language factorial design varies question, requested-response, and verifier languages across ten open-weight models and three benchmarks, holding each response fixed across verifier languages.Confidence is measured with a first-token P(True) probe that reads the probability assigned to “True” about the model’s own answer.Source: §1 · pp. 1–2
  • The benchmark determines which language governs accuracy: question language dominates MMMLU, content subset and response language dominate LSQA, and both languages matter on TriviaQA.Matching response language to question or verifier language improves discrimination on LSQA and TriviaQA but not calibration; verifier language shifts ECE without changing accuracy.Source: §1 · p. 2
  • The automated judge would shrink MMMLU’s English–Hindi question-language accuracy gap from 0.198 to 0.038, so scorer validation can change the apparent result.The study recommends reporting all three languages, realized response language, and per-benchmark scorer validation.Source: §1 · p. 2

2 Related Work

Prior work studies confidence, multilingual performance, and automated judging, but commonly changes prompt language as a whole rather than separating its components.

2 Related Work

3 Experimental Design

The experimental design crosses six question and response languages with verifier languages across three parallel benchmarks and ten open-weight models, using a first-token confidence probe.

3.1 Benchmarks and Languages

  • Six languages cover 960-item parallel benchmarks: LSQA separates content-language subsets, MMMLU uses shared-option questions, and TriviaQA has English-only answer aliases.LSQA uses concise free-text responses; MMMLU requests an option letter and brief explanation, while TriviaQA requests a short answer.Source: §3.1 · pp. 2–3

3.2 Factorial Units and Counts

  • Each item and model yields 36 question-by-response combinations, each evaluated under six verifier languages; the full panel contains 1,036,800 responses and 6,220,800 verifier evaluations.The LSQA content-language match refers to the item’s subset; MMMLU subject groups are not native languages.Source: §3.2 · p. 3

3.3 Model Panel

  • The panel contains ten instruction-tuned open-weight models from five developers, spanning Qwen, Llama, Gemma, Mistral, and Aya families.The named models range from 3B to 14B parameters.Source: §3.3 · p. 3

3.4 Generation and Runtime

  • Generation used greedy decoding with up to 256 new tokens; prompts place localized response-language directives differently for MMMLU versus LSQA and TriviaQA.Qwen3 disabled thinking mode after two-thirds of default-mode responses ended in unfinished reasoning blocks; right truncation affected 0.02%–0.03% of prompts and excluding affected responses changed no MMMLU result.Source: §3.4 · pp. 3–4

3.5 Confidence Extraction

  • Verifier instructions were localized while the question and generated response stayed fixed; English True/False token variants supplied the max-logit P(True) score.A separate LSQA run evaluated the same Qwen3 responses’ verifier logits with thinking enabled as a template-mode ablation.Source: §3.5 · p. 4

4 Correctness Labels and Validation

The section combines exact-option matching for MMMLU with an automated multilingual judge for free-text answers, then checks those labels against human ratings. Validation finds different error patterns by benchmark: the judge is conservative on LSQA and TriviaQA but over-credits MMMLU.

4.1 Correctness Sources

  • MMMLU correctness uses exact matching between the extracted option and the gold option, while free-text LSQA and TriviaQA answers use the automated judge.Responses without an extractable final option are scored incorrect; 1.5% of 345,600 MMMLU responses had no extractable option.Source: §4.1 · p. 4

4.2 Automated Judge

  • The multilingual judge scores responses from 1–5, with parse-valid scores 4–5 mapped to correct and scores 1–3 to incorrect.Malformed outputs are also counted as incorrect, a conservative fallback policy rather than verified incorrectness.Source: §4.2 · p. 4
  • The judge evaluates central factual correctness without penalizing code-switching or a response language that differs from the request; language compliance is labeled separately.Its inputs include the question and choices, unchanged response, response-language reference, English cross-check, and English aliases.Source: §4.2 · p. 4

4.3 Human Validation

  • Human validation used blinded ratings of a frozen, stratified sample, with Qwen3-8B’s superseded responses excluded from the reported ratings.Of 972 remaining responses, 37 were marked unjudgeable; Table 1 reports agreement with 95% bootstrap intervals.Source: §4.3 · p. 5, Table 1 · p. 5
  • The judge agrees with human labels on 80.3% of responses overall, but is conservative on LSQA and TriviaQA and over-credits MMMLU.On MMMLU, it labels 66.0% correct against reviewers’ 37.9%; exact option match agrees with reviewers’ option-level judgments on 98.1%.Source: §4.3 · p. 5, Table 1 · p. 5
  • The judge’s errors differ across benchmarks: it under-credits LSQA and TriviaQA, while MMMLU’s remaining gap mainly reflects correct options with incorrect justifications.On LSQA, the judge’s correct rate is 5.7 points below the human rate; on TriviaQA, it is 7.7 points below.Source: §4.3 · p. 5

5 Analysis

The analysis separates accuracy from confidence discrimination and calibration, then examines language matching and decomposes variation across language factors. It uses configuration-level summaries and models, while treating the resulting shares as descriptive rather than identified effects.

Metrics

  • Accuracy, Brier score, and 15-bin ECE measure correctness, confidence error, and calibration, respectively.Correctness is shared across the six verifier-language copies of each response, so those copies add no accuracy observations.Source: §5 · p. 5

Aggregation

  • For LSQA, the 36 question/response conditions comprise one native-language match, five question-only matches, five response-only matches, and 25 matching neither.A configuration cell pools 960 item rows, and summary tables weight listed cells or units equally across models and verifier languages.Source: §5 · pp. 5–6

Accuracy decomposition

  • The analysis decomposes response correctness by question language, response language, stratum, and their two-way interactions, averaging explained-sum shares across models.The stratum is the LSQA content subset or MMMLU subject group; TriviaQA has no comparable stratum.Source: §5 · p. 6

Configuration-level decomposition

  • Configuration-level decompositions include question, response, and verifier languages and their two-way interactions, with shares treated as provisional descriptions rather than identified effects.LSQA models also include the content subset at the subset-cell level.Source: §5 · p. 6

Matching effects

  • Matching effects on AUROC and ECE are estimated per dataset with mixed models over 2,160 configuration cells and a random intercept for each model.The analysis also resamples ten models and fits per-model regressions, but does not account for shared base items, has limited model-level generalization, and reports 24 coefficients without adjustment.Source: §5 · p. 6

6 Results

Across three benchmarks, which language governs accuracy depends on the benchmark, while response-language matching and verifier language affect confidence measures differently. Scoring and translation choices also qualify the MMMLU comparison.

6.1 Overview

  • Across datasets, TriviaQA has lowest accuracy but strongest discrimination, while exact-match MMMLU has weakest discrimination and worst calibration; ECE spans 0.27–0.37.Table 2 reports means over configuration cells; Figure 2 shows how far each language axis moves each metric.Source: §6.1 · p. 6, Table 2 · p. 6, Figure 2 · p. 6

6.2 MMMLU Accuracy under Exact Match

  • Under exact match, MMMLU English questions exceed Hindi by 0.198 (95% interval [0.182, 0.213]), versus a 0.026 response-language span; judge scoring shrinks the gap to 0.038.The English–Hindi difference is best read as an upper bound on the question-language effect because audited translation defects were more frequent in Chinese and Hindi items.Source: §6.2 · pp. 6–7, Figure 2 · p. 6

6.3 Question and Response Language

  • Accuracy drivers vary by benchmark: MMMLU is question-language-dominated (56.6%), LSQA by content subset and response language (39.1%, 30.4%), while TriviaQA shares question/response contributions (51.0%, 37.4%).For TriviaQA, question–response interaction accounts for 11.6% of explained variation. These are provisional configuration-level decomposition shares, and LSQA response-language shares vary from 9.8% to 55.5% across models.Source: §6.3 · p. 7, Figure 3 · p. 8
  • On TriviaQA, English questions and responses maximize accuracy and AUROC, but non-English questions with English responses have worse calibration; on LSQA, content-language alignment maximizes accuracy.On LSQA, matching only the response or only the question recovers part of the accuracy advantage about equally; AUROC is highest when question and response share a non-content language. Content-language alignment adds 0.17 and 0.19 accuracy on Korean and Hindi subsets, respectively.Source: §6.3 · p. 7, Table 3 · p. 7, Figure 3 · p. 8

6.4 Language Matching and Confidence

  • Matching response language to question or verifier raises AUROC by 0.02–0.03 on LSQA and TriviaQA, but MMMLU effects are four to six times smaller and calibration effects are inconsistent.No response-language match lowers ECE on LSQA or TriviaQA; the positive ECE coefficients appear in only six to eight models, and model-level tests do not distinguish them from zero.Source: §6.4 · p. 7, Table 4 · p. 8

6.5 Verifier Language

  • Verifier language cannot alter accuracy, but ECE varies by up to 0.04 across verifier languages, with French instructions yielding the lowest ECE on all three benchmarks.Verifier-language main effects account for 6%–18% of explained AUROC and ECE variation, and interactions involving verifier language add 6%–13%.Source: Figure 2 · p. 6, §6.5 · pp. 7–8

6.6 Robustness

  • Robustness checks preserve LSQA and TriviaQA AUROC matching results and the MMMLU question-language gap, but restricting requested-language responses removes small MMMLU AUROC effects and shifts TriviaQA’s English advantage.The restricted analysis shifts TriviaQA’s English advantage from the question to the response.Source: §6.6 · p. 8

7 Conclusion

The study separates question, requested-response, and verifier languages in multilingual confidence evaluation. Their effects vary by benchmark and metric, while scorer choice and the scope of the probe qualify interpretation.

7 Conclusion

  • Accuracy drivers differ across benchmarks: question language dominates MMMLU, content subset and response language dominate LSQA, and both languages matter on TriviaQA.Source: §7 · p. 8
  • Response-language matching sharpens discrimination on LSQA and TriviaQA without improving calibration, while verifier language shifts ECE by up to 0.04 without changing accuracy.Source: §7 · p. 8
  • The findings concern one first-token P(True) probe and ten 3B–14B open-weight models, not languages in general.Source: §7 · p. 8

Limitations

The findings are bounded by the estimator, model set, prompt setup, and translated benchmark items used in the study.

Limitations

  • The evaluation covers one first-token P(True) estimator and ten 3B–14B open-weight models; other estimators and larger or closed models may behave differently.Source: Limitations · pp. 8–9
  • One prompt template per benchmark limits the setup, and localized verifier instructions retain the English labels True and False, which may contribute to verifier-language effects.Source: Limitations · p. 9
  • Non-English MMMLU and TriviaQA items are translated from English, which can favor English questions.Source: Limitations · p. 9

Ethics and Reproducibility

The authors document their human-review process, research-use constraints, computational costs, and reproducibility artifacts. The prompt appendix specifies the multilingual evaluation and judging setup, including a tokenization limitation for translated verifier labels.

Ethics and Reproducibility

  • Five authors conducted human review across the requested response languages, without LLM assistance or external participants; no institutional ethics review was sought.One author covered two languages, and all reviewers were native or fluent speakers of the languages reviewed.Source: Ethics and Reproducibility · p. 9
  • The results cover one confidence probe, three benchmarks, and ten 3B–14B models; the authors caution against treating them as language properties or speaker reliability guarantees.Released model outputs contain factual errors, including about public figures, and are for research use only.Source: Ethics and Reproducibility · p. 9
  • The final runs used about 1,717 GPUhours, while failed, cancelled, superseded, and auxiliary runs used another 304 GPUhours.The final-run total includes 1,248 GPUhours for the correctness judge.Source: Ethics and Reproducibility · p. 9
  • The archives include code, prompts, 1,036,800 responses, labels, P(True) values, ratings, and recorded hashes that support reproducing reported values.They also record selected items, model revisions, and runtime versions.Source: Ethics and Reproducibility · p. 9

A Prompts and Data Provenance

  • The verifier prompt asks for a True-or-False judgment of a proposed answer, while the judge prompt scores factual correctness against a rubric and reference answer.The parser accepts nonempty feedback with an optional prefix and terminal [RESULT] plus one integer from 1–5; other outputs are malformed.Source: A Prompts and Data Provenance · p. 13
  • Translated verifier labels can span multiple tokens, so a first-token probability may score only a word prefix rather than the full translated label.This occurs for at least one word of the French, Japanese, Korean, and Hindi True/False pairs in every tokenizer without a leading space.Source: A Prompts and Data Provenance · p. 13

B Human Review Details

The human-review sample was selected independently of judge scores and completion status, and reviewers were blinded to model identities and automated outputs. Blinding was not guaranteed, and the item-bootstrap intervals are not exact design-based intervals.

B Human Review Details

  • The primary review sample was frozen from generated responses without using judge scores or completion status, avoiding judge-conditioned convenience sampling.Packets omitted model identities, judge outputs, and confidence scores, although response text could reveal a model.Source: B Human Review Details · p. 13
  • Reviewers could mark responses unjudgeable rather than infer unfamiliar language content, but blinding was not guaranteed.A response could reveal its model through its text.Source: B Human Review Details · pp. 13–14
  • The item bootstrap preserves dependence among repeated configurations of an item, but its intervals are not exact design-based intervals.The automatic language rule agreed with reviewers on compliant-versus-other labels for 85.6% of 972 reviewed responses (κ = 0.69).Source: B Human Review Details · p. 14

C Robustness Details

Robustness analyses probe matching effects, judge errors, alignment, pooled metrics, and sensitivity to exclusions and prompt templates. Several headline patterns persist under alternative data treatments, while the direct confidence probe is sensitive to template mode.

C Robustness Details

  • Excluding score-3 and malformed judge outputs raises accuracy from 0.511 to 0.540 on LSQA and from 0.316 to 0.322 on TriviaQA, while AUROC changes by less than 0.003.Regime and group orderings remain unchanged, but the LSQA Q=R ECE coefficient falls to 0.009 and its interval includes zero.Source: C Robustness Details · p. 14
  • 0.197 [0.182, 0.213] is the MMMLU English–Hindi question-language accuracy difference without Qwen3-8B, and the English-response advantage remains 0.022 [0.019, 0.025] above the non-English mean.The Hindi native advantage on LSQA becomes −0.031 [−0.064, −0.0004].Source: C Robustness Details · pp. 14–15
  • 0.5232 to 0.5234 is the MMMLU accuracy change after excluding truncated prompts; AUROC, ECE, and matching-effect interval conclusions are effectively unchanged.The affected prompts involved 16 items and 357 responses; only 74.1% of LSQA, 63.2% of MMMLU, and 51.9% of TriviaQA responses were labeled as written in the requested language.Source: C Robustness Details · p. 15
  • On LSQA and TriviaQA, model-level tests confirm AUROC effects of Q=R and R=V, while the same effects are not distinguishable from zero on MMMLU.No LSQA or TriviaQA ECE effect of Q=R or R=V reaches significance; specified ECE effects occur for Q=V and three-way matching on TriviaQA, and R=V on MMMLU.Source: C Robustness Details · p. 15
  • 0.915 [0.858, 0.962] is judge agreement when question and response share a language, versus 0.845 [0.815, 0.876] otherwise.On LSQA the judge’s correct-rate difference from reviewers ranges from 0.000 for English and Korean requests to −0.140 for Hindi requests.Source: C Robustness Details · p. 15
  • 0.189 [0.127, 0.251] is the Korean/Hindi LSQA Q=R=s gain for Hindi, while English has a −0.050 [−0.088, −0.012] gain.The Korean and Hindi gains are positive for all ten models; the unadjusted native advantage is small there because unaligned conditions include accuracy-raising English questions and responses.Source: C Robustness Details · pp. 15–16
  • 0.797, 0.795, 0.783, 0.805, and 0.788 are the cell-level AUROCs for the five regimes in table order, rather than pooled-regime AUROCs.Accuracy is unchanged by this alternative averaging; the pooled regime metric can reward differences between conditions within a regime.Source: C Robustness Details · p. 16

D Thinking-Template Ablation

  • 0.502 accuracy remains unchanged under the Qwen3-8B thinking-enabled template, but median P(True) moves from 1.000 to 0.516 and AUROC drops from 0.862 to 0.572.ECE falls from 0.236 to 0.143 while Brier rises from 0.238 to 0.270; this is template-mode sensitivity of the direct probe, not evidence about reasoning.Source: D Thinking-Template Ablation · p. 16

E Additional Results

Additional analyses track model-rank changes across language settings and provide reviewer instructions and human-validated examples. The appendices specify how reviewers assess correctness, language, translation, and related response properties.

E Additional Results

  • Figures 1 and 5 follow a path that switches question, response, and verifier languages in sequence; Figure 5 reports benchmark-specific model ranks averaged over five target languages.The four colored models were selected after results were seen, and the supplementary data provide scores, ranks, and rank intervals for all ten models.Source: E Additional Results · pp. 16–17, Figure 5 · p. 18
  • Six of 39 significantly ordered accuracy pairs reverse between all-English and fully localized settings, while six ECE pairs reverse when only verifier language changes.One ECE and three Brier pairs also reverse at the verifier step, where accuracy cannot change.Source: E Additional Results · p. 16, Table 8 · p. 17
  • Single-configuration rankings depart from the all-English ranking more than all-English bootstrap sampling noise, with median Kendall’s τ ranging from 0.11–0.38 on LSQA, 0.47–0.56 on TriviaQA, and 0.60–0.73 on MMMLU.All-English resamples themselves yield mean τ of 0.70–0.95.Source: E Additional Results · p. 17

F Reviewer Instructions

  • Reviewer instructions require abstention when reviewers cannot assess the response language and caution that localized-question judgments do not validate equivalence to the original question.Reviewers assess translation preservation, answer validity, cultural context, and answerability jointly where applicable.Source: F Reviewer Instructions · p. 18, Response correctness and language · pp. 18–19

Response correctness and language

  • Correctness is judged overall, accepting valid paraphrases and aliases but not treating a correct name with a contradictory explanation as automatically correct.Language mismatch is recorded separately from factual correctness, and MMMLU option accuracy is distinguished from overall correctness and explanation accuracy.Source: Response correctness and language · pp. 18–19

G Human-Validated Examples

  • Table 9 presents three human-validated examples selected to illustrate phenomena, not estimate their prevalence.The first two examples are the first qualifying units in random review-ID order; the last has the widest P(True) range across verifier languages among qualifying reviewed units with agreeing labels.Source: Table 9 · p. 19, G Human-Validated Examples · p. 19

Response correctness and language

  • Optional author adjudication follows locked primary ratings and is not a second independent rating or evidence of inter-rater agreement.Unresolved cases remain unjudgeable, and automated fallback should not force uncertain human judgments to incorrect.Source: Response correctness and language · p. 19
Loading 2610.32144v1…