PaperlayerView exact revision on arXiv

Source-linked AI summary

Multilingual Verifier Bias in RLVR: Benchmark, Rollout Diagnosis, and the Cross-Lingual Selection Bottleneck

Chenyu Zhou, Qiliang Jiang, Xu Zhou

arXiv:2608.20362v1cs.CLcs.LG

TL;DR

Multilingual RLVR can turn answer-format variation into language-dependent false-negative reward noise, undermining the assumption that verifiers are language-neutral. The paper develops a reusable audit protocol and finds substantial rollout bias, a recoverable cross-lingual selection bottleneck, and persistent bias under training optimization.

  • Problem

    Exact-match verifiers can reject semantically correct multilingual answers, but the resulting language-conditioned reward noise is insufficiently characterized for RLVR.

  • Method

    The paper audits proxy versus trusted rewards through verifier-robustness tests, rollout diagnosis, interface probing, language-conditioned metrics, selection analysis, and training audits.

  • Results

    55–78% of the average selection gap is closed by label-free cross-lingual aggregation, while exact-match false-negative rates vary sharply, reaching 0.642 on Qwen3-8B JP.

  • Takeaways & Limitations

    Multilingual RLVR rewards should be audited by language and answer interface before optimization, because selection headroom remains recoverable while reward bias can persist during training.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) is a standard recipe for training large language models on mathematical reasoning, where an answer verifier serves as a language-neutral reward function. We show that this assumption fails in multilingual settings: an exact-match verifier turns format and script variation into language-dependent false-negative reward noise. We introduce a reusable protocol for auditing multilingual RLVR rewards: a verifier-robustness suite, a rollout-diagnosis procedure, and language-conditioned reward-error metrics for Japanese, English, and Chinese answers. On MGSM rollouts with k=8, the exact-match proxy rejects trusted-correct answers at sharply different rates by language across Qwen3-4B, Qwen3-8B, and Llama-3.1-8B-Instruct; for Qwen3-8B, the false-negative rate reaches 0.642 on JP against 0.122 on EN and 0.073 on CN. A plain-numeric probe localizes the mechanism to the final-answer interface: an interface model drives reward-error VLB to zero while the residual accuracy gap is unchanged. We then expose a cross-lingual selection bottleneck: on MGSM250 rollouts, a target-local aggregation rule using no trusted labels closes 55-78% of the average selection gap, and over 95% of repairs require genuine cross-lingual support. The bottleneck replicates on a 483-problem MATH-500 set. A controlled training audit shows that rule-GRPO raises trusted accuracy while the reward-error VLB stays high. The unifying message is operational: multilingual RLVR rewards should be audited by language and by answer interface before they are optimized.

1 Introduction

RLVR assumes verifiers provide language-neutral correctness signals, but exact matching converts multilingual answer-format variation into language-dependent reward pressure. The paper introduces a reusable audit protocol that traces this bias from measurement through mechanism, selection, and optimization.

  • RLVR optimizes mathematical reasoning with a binary verifier reward, making verifier reliability central to training.
  • Exact-match verification can reject semantically correct answers whose scripts, wrappers, punctuation, or numeric formats differ across languages.These false negatives become language-dependent reward pressure when their distribution varies by language.
  • The paper measures multilingual verifier bias through robustness tests, rollout diagnosis, interface probing, selection analysis, and training audits.The protocol compares weak proxy rewards with trusted canonical-equivalence rewards on model samples.
  • Verifier language bias VLB(m) summarizes the spread of any language-conditioned metric across languages and is reproducible across three models and two model families.The worst-affected language depends on the model’s output distribution rather than the benchmark alone.
  • The paper treats rollout diagnosis, interface probing, selection analysis, and training audits as four views of one language-conditioned proxy-versus-trusted reward gap.A shared tuple schema supports all analyses and connects verifier noise to downstream decision and optimization costs.
  • Its contributions are an auditable protocol, an interface-level diagnosis, and a label-free cross-lingual procedure targeting the selection bottleneck.The protocol includes language-conditioned error metrics and applies across models, verifiers, and language sets.

2 Related Work

Prior work studies verifier noise, multilingual evaluation, cross-lingual consistency, and verifier-free rewards from complementary perspectives. This paper distinguishes its focus on language-conditioned rule-verifier failures during RLVR rollouts and label-free inference-time selection.

  • Prior RLVR research treats verifier unreliability as noisy supervision and studies how false positives and false negatives affect optimization.The paper sharpens this framework by conditioning verifier reliability on language.
  • The paper’s central distinction is that multilingual exact-match errors are false-negative dominated, whereas reward-hacking work primarily examines false positives.These failure modes require different audits: trusted references for false positives, and language- and interface-conditioned audits for multilingual false negatives.
  • Multilingual judge studies evaluate language-dependent evaluator reliability, while this work studies rule verifiers used as rewards during model rollouts.
  • Cross-lingual consistency methods improve reasoning through multilingual agreement, but this paper’s aggregation targets selection failures rather than agreement-maximizing decoding.
  • Verifier-free and self-verified reward methods remove or soften external verification during training, whereas this work audits proxy rewards and uses inference-time aggregation.
  • The work complements multilingual math benchmark construction by proposing a reward-auditing protocol rather than a new test set.

3 Problem Formulation

The paper formalizes multilingual verifier error by comparing proxy and trusted rewards on language-conditioned rollouts. It reports per-language error profiles, their spread, and the resulting policy pressure through a reusable audit tuple.

  • A trusted reward uses canonical answer equivalence, while a proxy verifier may use exact string matching on extracted final answers.Canonicalization handles Arabic digits, CJK magnitude units, percentages, and signs before comparison.
  • A false negative is a trusted-correct completion rejected by the verifier, while a false positive is an incorrect completion incorrectly rewarded.
  • FNR(l) measures rejected trusted-correct completions, and Err(l) measures all proxy-versus-trusted reward disagreements for language l.When false positives are negligible, Err(l) reduces to FNR(l)·acc(l), the withheld reward mass.
  • The audit outputs per-language profiles, VLB summaries, and the dominant failure mode, then connects rollout and interface measurements to downstream selection and training consequences.
  • VLB summarizes the spread of a per-language metric as max_l m(l)−min_l m(l).The protocol uses the same model-, verifier-, and language-conditioned structure across analyses.
  • Reward-error VLB and false-negative-rate VLB are primary quantities because large spread indicates language-specific gradient pressure, especially when exact-match errors are false-negative dominated.
  • The algorithm samples k completions per multilingual prompt and scores each under both proxy and trusted rewards before computing language-conditioned error metrics.

4 A Multilingual Verifier-Robustness Suite

The verifier-robustness suite deliberately tests format and script variation against trusted canonical equivalence. It establishes exact-match fragility while showing that a fixed synthetic battery alone does not reveal live language-conditioned bias.

  • The suite constructs candidate answers spanning correct forms, formatting variants, and near misses across MGSM and multilingual MATH-500 records.Transformations include full-width digits, CJK forms, percentages, explanatory text, mixed scripts, and near-miss values.
  • Trusted canonical equivalence normalizes script, magnitude units, punctuation, and percentage forms, exposing the surface phenomena mishandled by weaker proxy verifiers.This design measures verifier fragility rather than model competence.
  • The synthetic battery is language-symmetric because every language receives the same format variants, so it serves as a fragility baseline rather than the live language-bias result.Live model formatting habits differ by language and break this symmetry.
  • Exact matching has error around 0.71 in every language, while normalized numeric and hybrid verifiers reduce error to roughly 0.12–0.13.All three verifiers are false-negative dominated, with language spread VLB ≤0.013 on the battery.

5 Rollout Reward Diagnosis

Exact-match reward noise on live MGSM rollouts is almost entirely false-negative and strongly language-conditioned, with the worst-affected language varying by model. The errors arise from model-specific formatting habits and can suppress deserved reward despite strong trusted accuracy.

  • 0 false-positive rate was observed across all models and languages, so exact-match noise only rejects trusted-correct answers.
  • 0.642 was Qwen3-8B’s JP false-negative rate, versus 0.122 on EN and 0.073 on CN.
  • 0.569 was Qwen3-8B’s false-negative-rate VLB, compared with 0.234 for Qwen3-4B and 0.258 for Llama-3.1-8B-Instruct.
  • 307 of 308 rejected-correct Qwen3-8B JP completions used angle brackets, and numeric normalization would readmit 94.8% of them.
  • Qwen3-8B’s trusted accuracy is highest on EN at 0.948 but remains 0.750 on JP, showing that reward noise can concentrate where problem-solving ability remains substantial.
  • The worst-affected language shifts across models: EN for Qwen3-4B and Llama-3.1-8B-Instruct, but JP for Qwen3-8B.

6 The Interface Mechanism

A controlled plain-numeric probe separates final-answer formatting from multilingual arithmetic ability. Fixing the interface eliminates exact-match verifier bias, while a smaller trusted-accuracy gap remains.

  • The probe evaluates three Qwen3-8B variants on a leak-free 20-identifier heldout split across three languages with k = 4.
  • 0.000 was the interface SFT model’s exact reward-error VLB and exact false-negative-rate VLB under the plain-numeric interface.
  • 0.150 was the language accuracy gap for both rule-GRPO and the interface model, versus 0.213 for the base model.
  • 0.875 EN, 0.825 CN, and 0.725 JP trusted accuracies leave an EN–JP difference after formatting is held constant.
  • The exact-match component is removed by controlling how the final answer is written and parsed, not by changing the RL objective.

7 The Cross-Lingual Selection Bottleneck

Target-local cross-lingual aggregation addresses a substantial selection bottleneck without trusted labels or imported answers. It improves accuracy, chiefly for JP, closes much of the selection gap, and replicates on audited MATH-500.

  • 55.3% and 78.2% of the average local-majority selection gap were closed for Qwen3-8B and Llama-3.1-8B-Instruct, respectively.
  • 0.867 to 0.895 was Qwen3-8B’s average-accuracy increase, while its EN–JP gap fell from 0.172 to 0.124.
  • 0.889 and 0.959 were the corrected fractions among changed decisions for Qwen3-8B and Llama-3.1-8B-Instruct.
  • 95.8% of Qwen3-8B fixes and 98.6% of Llama fixes had target-language votes no greater than sibling support, indicating genuine cross-lingual support.
  • The same target-local rule replicates on an audited 483-problem MATH-500 set after excluding 17 Japanese problems with answerability or translation issues.

8 Training-Time Consequence

The training audit shows that verifier bias can suppress RLVR’s learning signal, while proxy optimization can improve trusted accuracy without reducing language-conditioned reward error.

  • Zero reward standard deviation across five logged steps produced gradient norm 0.000 in a Qwen3-4B rule-GRPO run.Every sampled group received identical reward in the single-language split.
  • Trusted accuracy improved from 0.647 to 0.735 while exact-match reward-error VLB worsened from 0.417 to 0.500.The evaluation used a held 17-prompt mixed split with k = 4 and 68 records per model.

9 Discussion

The discussion turns the findings into an operational workflow: diagnose language-conditioned reward error, isolate interface effects, recover selection headroom, and avoid optimizing an unchecked verifier.

  • Workflow: Rollout diagnosis locates language-conditioned reward error, interface probing separates format effects from residual capability gaps, and selection recovers part of the remaining headroom.The workflow connects Sections 5–7 into a sequence from diagnosis to decision-time repair.
  • Training audit: Trusted accuracy can improve while exact-match reward-error VLB stays high, so proxy optimization does not by itself remove verifier bias.Table 5 evaluates a 17-prompt mixed split with k = 4 and 68 records per model.
  • Operational rule: A practical trigger is a single language with false-negative rate above ∼0.3 and false-negative-rate VLB above ∼0.2.The prescribed ordering is interface first, followed by label-free selection using sibling-language agreement.
  • Reusability: The protocol uses one tuple and model- and verifier-agnostic metrics to audit multilingual verifier fairness on a common output scale.The same measurement format spans synthetic tests, live policies, interface probes, and training runs.

10 Conclusion

The paper establishes multilingual verifier bias as a reproducible RLVR problem and traces it from language-conditioned false negatives to interface effects and cross-lingual selection repair.

  • Exact-match verifier rewards create false-negative-dominated, language-conditioned pressure across Qwen and Llama MGSM rollouts.The worst-affected language is determined by the model rather than the benchmark.
  • A plain-numeric probe localizes the exact-match component to the final-answer interface, separating a removable format effect from a residual capability gap.
  • A label-free cross-lingual selection rule closes 55–78% of the average local-majority selection gap and 63–88% of the JP gap.The repairs were confirmed to be genuinely cross-lingual, and the diagnosis was corroborated on a manually audited 483-problem MATH-500 set.
Loading 2608.20362v1…