Multilingual Verifier Bias in RLVR: Benchmark, Rollout Diagnosis, and the Cross-Lingual Selection Bottleneck

Introduction

• Strict exact-match verification can reject semantically correct answers with currency symbols, markdown, brackets, decimal variants, full-width digits, CJK units, or language-specific suffixes, injecting language-dependent reward pressure into policy gradients.The introduction identifies verifier stability across training languages and answer formats as an implicit reliability assumption of RLVR.Source: §1 · p. 2
• The verifier language bias VLB(m) is the max-minus-min spread of a per-language metric; diagnosis spans three models and two model families, and the worst-affected language depends on model output distribution.Rollout diagnosis, the interface probe, selection analysis, and training audit are presented as connected measurements of the language-conditioned gap between proxy and trusted reward.Source: §1 · p. 2
• Across three models, exact-match reward noise is false-negative dominated; on same-split MGSM rollouts, a plain-numeric interface model drives exact reward-error VLB to 0.000 without changing the residual accuracy gap.The probe localizes the exact-match component to the final-answer interface, separating format effects from capability effects.Source: §1 · p. 3
• The reusable audit protocol fixes a common tuple and applies language-conditioned reward-error metrics unchanged across models, verifiers, and language sets.Its metrics include false-negative and false-positive rates, reward-error rate, expected policy pressure, and verifier language bias; the released JP/EN/CN suite provides exact-match, normalized-numeric, and hybrid rule verifiers.Source: §1 · p. 3
• On MGSM250, target-local label-free aggregation recovers 55–78% of the average and 63–88% of the JP local-majority selection gap, replicates on 483 MATH-500 problems, and over 95% of repairs need cross-lingual agreement.The authors characterize much of the language gap as a recoverable selection failure rather than a generation deficit.Source: §1 · p. 3

Verifier noise in RLVR

• Prior RLVR work studies global verifier noise and proxy-reward exploitation [15] [17] [6], while this work conditions false-negative-dominated exact-match reliability on language and links its spread to multilingual policy pressure.When FPR is approximately zero, J(l) ≈ TPR(l) = 1 − FNR(l), so a single global correction cannot track different language reward regimes.Source: §2 · pp. 3–4

Multilingual reasoning and judges

• Prior multilingual reasoning and judge studies [12] [9] [5] examine evaluation or reasoning behavior, whereas this work measures language-conditioned failures of a rule verifier used as a rollout reward.Source: §2 · p. 4

Cross-lingual consistency and aggregation

• Unlike cross-lingual voting [16] or self-consistency training [4], the aggregation procedure selects among target-language answer clusters without importing answers or training, to diagnose recoverable selection failure.Its purpose is to separate selection failure from generation deficit, not to maximize accuracy as a decoder.Source: §2 · p. 4

Verifier-free and self-verified rewards

• Verifier-free and self-verified rewards [14] are training-time approaches, while this work’s central contribution is an auditing protocol around inference-time aggregation.Source: §2 · p. 4

Math benchmarks across languages

• The suite builds on GSM8K and MATH [2] [7] and the multilingual-evaluation motivation of MATH-PT [13], but contributes a reward-auditing protocol rather than a new test set.Source: §2 · p. 4

Problem Formulation

• The trusted reward compares canonically normalized predictions and references, while the proxy reward is produced by a verifier such as exact string match.Canonicalization handles Arabic digits, CJK magnitude expressions, percentages, and signs.Source: §3 · p. 5
• FNR is the share of trusted-correct completions rejected by the verifier, while FPR is the share of trusted-incorrect completions it accepts.Err(l) is the probability that proxy and trusted rewards differ; for an FN-dominated verifier, it reduces to FNR(l)·acc(l), the withheld reward mass.Source: §3 · p. 5
• Because Err creates a negative reward signal on correct-but-rejected answers, high-FNR languages can receive systematic negative GRPO-style advantage pressure.The resulting gradient pressure can bias against formatting habits used in that language.Source: §3 · p. 5
• The audit samples k completions per language-prompt pair and reports per-language error profiles, VLB summaries, and the dominant failure mode; it flags languages with FNR > 0.3 and VLB(FNR) > 0.2.The protocol is specified by (π, {x_l}, k, v, r∗) and is model- and verifier-agnostic.Source: §3 · pp. 5–6, Figure 1 · p. 6
• VLB measures the max–min spread of a per-language metric: near-zero VLB indicates symmetry, whereas large VLB indicates language-specific gradient pressure.In MGSM, exact-match errors are FN-dominated, making VLB(FNR) the most informative single measure; rates are reported to three decimals and VLBs use displayed values.Source: §3 · p. 5

4 A Multilingual Verifier-Robustness Suite

• The fixed battery uses MGSM JP/EN/CN and multilingual MATH-500 records, generating correct, format-variant, and near-miss candidates across script, magnitude, punctuation, percentage, wrapper, and numeric forms.The release includes schemas, transformations, configs, scripts, and aggregate tables; upstream problem text remains under original dataset licenses.Source: §4 · pp. 6–7
• Canonical equivalence normalizes script, magnitude units, punctuation, and percentage forms in predictions and references; the suite tests exact match, normalized numeric equivalence, and exact-then-numeric hybrid verification.The proxy verifiers are deliberately weaker so their gap from the trusted reward exposes surface phenomena strict matching mishandles.Source: §4 · pp. 6–7
• Exact matching yields error around 0.71 in every language, while normalized numeric and hybrid verifiers reduce error to roughly 0.12–0.13; all are false-positive-near-zero and have VLB ≤0.013.The suite is a verifier-fragility baseline; its language spread is small, while language-conditioned bias is measured on live rollouts in Section 5.Source: Table 1 · p. 7, §4 · p. 7

• The rollout diagnosis sampled three models on MGSM EN/JP/CN, using 80 seed problems per language and k = 8 completions, for 1920 records per model.The trusted reward used canonical answer equivalence, while the primary proxy was the intentionally weaker exact-match verifier rule_exact_match_v1.Source: §5 · p. 7
• A fixed robustness suite showed language-symmetric exact-match brittleness (VLB ≤0.013), whereas live model-specific formatting exposed language-conditioned reward noise.Live policies develop different answer-formatting habits by language, so the interaction between verifier brittleness and those habits breaks the static suite’s symmetry.Source: §5 · p. 7
• Observed false-positive rates were 0 for every model and language, so the reward error consisted entirely of deserved reward withheld from correct answers.Concentrated false negatives depress the advantage of correct trajectories in one language, injecting a systematic language-specific negative signal into the policy gradient.Source: §5 · pp. 7–8
• Formatting explains most rejected-correct completions: 307 of 308 Qwen3-8B JP cases used angle brackets, and numeric normalization readmitted 95% of that model’s rejected-correct mass.Markdown or math delimiters dominated false negatives for Qwen3-4B on EN (98%) and Llama-3.1-8B-Instruct on EN (93%); numeric normalization readmitted 83% and 70%, respectively.Source: §5 · p. 8
• Qwen3-8B had trusted accuracy of 0.750 on JP despite an FNR of 0.642, while Llama-3.1-8B-Instruct had JP trusted accuracy of 0.469 and JP FNR of 0.090.The language with the largest reward error therefore need not be the model’s weakest-capability language; inspecting proxy reward by language distinguishes verifier penalties from low trusted accuracy.Source: §5 · p. 8
• False-negative-rate VLB reached 0.569 for Qwen3-8B, versus 0.234 for Qwen3-4B and 0.258 for Llama-3.1-8B-Instruct.The worst-affected language was EN for Qwen3-4B (FNR = 0.345) and Llama-3.1-8B-Instruct (0.342), but JP for Qwen3-8B (FNR = 0.642, versus 0.122 on EN and 0.073 on CN).Source: §5 · p. 8, Table 2 · p. 9, Figure 2 · p. 9

6 The Interface Mechanism

• The probe evaluates three Qwen3-8B variants on a leak-free, 20-identifier MGSM heldout split using a bare-numeric final-answer prompt.The variants are the base model, same-split rule-GRPO, and final-answer-interface SFT; each is evaluated on 240 records across three languages with k = 4, with no rollout failures.Source: §6 · pp. 9–10
• 0.000 exact reward-error VLB and exact false-negative-rate VLB in interface SFT remove measured exact-match verifier bias; base exact reward-error VLB is 0.013.A problem-level bootstrap gives the interface model’s exact VLB [0, 0], compared with Base [0, 0.05] and rule-GRPO [0, 0.14].Source: Table 3 · p. 10, §6 · p. 10
• 0.150 language accuracy gap persists for both rule-GRPO and interface SFT, compared with 0.213 for base, despite controlling the answer interface.Interface-model trusted accuracies are 0.875 on EN, 0.825 on CN, and 0.725 on JP; the 0.150 gap is EN–JP.Source: §6 · p. 10
• Controlling how the final answer is written and parsed removes the format component of verifier bias, but leaves a residual trusted-accuracy gap between languages.The probe isolates a policy capability difference rather than a verifier-format artifact; Section 7 examines how much of the remaining gap is recoverable at selection time.Source: §6 · p. 10

7 The Cross-Lingual Selection Bottleneck

• Average accuracy rises from 0.867 to 0.895 for Qwen3-8B and from 0.776 to 0.867 for Llama-3.1-8B-Instruct on MGSM250.The EN–JP gap falls from 0.172 to 0.124 and from 0.236 to 0.104, respectively; aggregation closes 55.3% and 78.2% of the average local-majority selection gap. It uses no trusted labels and selects only among answer clusters present in the target language.Source: Table 4 · p. 11, §7 · p. 11
• 23 of 24 Qwen3-8B fixes and 70 of 71 Llama fixes have target-language votes no greater than sibling support, ruling out within-language plurality as their source.Both sibling languages support the chosen cluster in 22/24 Qwen and 60/71 Llama fixes; borrowed count is 0 because the selector only chooses clusters present in the target language.Source: §7 · p. 11
• 123 corrections versus 14 regressions raise Qwen3-8B average accuracy on audited MATH-500 from 0.662 to 0.737, with the JP gain dominating.The audited set contains 483 problems after excluding 17 with Japanese rendering or translation artifacts; the EN–JP gap falls from 0.276 to 0.113, and borrowing stays at 0.Source: Figure 3 · p. 12, §7 · p. 12
• Average accuracy changes by ≤0.004 across α ∈{0.5, 1.0, 2.0}, while at least 91.7% of fixes are preserved.A conservative gate requiring support from both sibling languages retains 22 of 24 Qwen3-8B fixes and reduces regressions from 3 to 2 at the same gap reduction.Source: §7 · p. 12
• The section attributes recoverable language-gap headroom to selecting correct answers already generated, distinct from format artifacts and residual generation deficits.Sibling-language agreement supplies a label-free selection signal; the remaining pass@k ceiling not addressed by interface fixes or selection reflects a residual generation deficit.Source: §7 · p. 13

8 Training-Time Consequence

• Single-language Qwen3-4B has zero reward standard deviation and gradient norm 0.000 across five steps, while mixed-group Qwen3-8B reaches mean reward standard deviation 0.258 and gradient norm 0.172.Mixed-group sampling yields nonzero reward variance in 5 of 10 steps; constant rewards occur when correct answers are uniformly accepted or rejected within a group.Source: §8 · p. 13
• On a held 17-prompt mixed split (k=4), trusted accuracy rises from 0.647 to 0.735 while exact-match reward-error VLB worsens from 0.417 to 0.500.Accuracy gains occur on CN, EN, and JP, yet proxy optimization does not recover the selection headroom identified in Section 7.Source: §8 · p. 13

The workflow

• The workflow diagnoses language-conditioned reward error, probes answer-format effects, then uses cross-lingual selection to recover residual headroom.Skipping these steps can silently zero out the learning signal.Source: §9 · pp. 13–14
• Rule-GRPO raises trusted accuracy while exact-match reward-error VLB stays high, leaving language bias untouched.The training audit warns that optimizing against a verifier can silently zero out the learning signal.Source: Table 5 · p. 14, §9 · p. 14

An operational rule of thumb

• A practitioner trigger of one language above ∼0.3 false-negative rate and ∼0.2 false-negative-rate VLB prescribes interface repair before selection.This ordering held across all three models; sibling-language agreement enables label-free recovery of correct answers.Source: §9 · p. 14

Implications for verifier-free rewards

• Cross-lingual aggregation identifies the correct answer cluster in over 95% of repaired cases, supporting verifier-free training, intrinsic gradient signals [14], and soft self-verification.The proposed signal is most useful for selecting among answers already produced, rather than generating correct answers in a weak language.Source: §9 · p. 14

Reusability

• A shared tuple and model- and verifier-agnostic false-negative, false-positive, reward-error, and VLB metrics make multilingual verifier fairness a repeatable measurement.The protocol applies across a synthetic battery, three live policies, an interface probe, and a training run.Source: §9 · p. 14

10 Conclusion

• Exact-match verifier reward creates false-negative-dominated, language-conditioned pressure on real Qwen and Llama MGSM rollouts, with the worst language determined by model rather than benchmark.Source: §10 · p. 14
• A controlled plain-numeric probe localizes the exact-match component to the final-answer interface, separating a removable format effect from a residual capability gap.Source: §10 · pp. 14–15
• On fresh data, label-free cross-lingual selection closes 55–78% of the average local-majority selection gap and 63–88% of the JP gap, with repairs genuinely cross-lingual.The repair autopsy confirms that the gains require cross-lingual support.Source: §10 · p. 15
• The diagnosis spans two model families on MGSM JP/EN/CN and is corroborated on a manually audited 483-problem MATH-500 set.Cross-lingual aggregation runs at inference time on parallel multilingual samples; converting its selection signal into a training-time reward is a natural next step.Source: §10 · p. 15
• The results support auditing multilingual RLVR rewards by language and answer interface before optimizing against them.Source: §10 · p. 15