Source-linked AI summary
Where the Verifier Fails: A Category-Level Audit of Reward Signals in RLVR
Esther Xin
TL;DR
Automatic verifiers underpin both RLVR rewards and benchmark accuracy, but aggregate reliability does not reveal which answer forms fail. The paper uses certified metamorphic testing to audit four verifiers across 307,420 verdicts, finding implementation-dependent reliability, concentrated string-hygiene failures, and systematic magnitude-dependent false positives.
Problem
Aggregate verifier reliability, including a published roughly 94% self-validation figure, does not identify which answer forms consume the error budget.
Method
The paper applies metamorphic testing to verifiers, generating semantics-preserving answer variants whose rejection certifies false negatives, and audits four implementations with contract and coverage accounting.
Results
Self-validation ranges from 53.8% to 95.2%, while whitespace and punctuation account for 93.0% of default-LaTeX in-contract failures; a reference cascade also shows a magnitude threshold for false positives.
Takeaways & Limitations
Verifier reliability is implementation-specific, dominated by answer-string normalization in one configuration, and requires coverage reporting to distinguish rejection from execution failure.
Takeaways & Limitations
The paper does not estimate effects on model benchmark scores or downstream RLVR training, and makes no claims about closed frontier systems.
Abstract
from arXiv · showhide
Reinforcement learning with verifiable rewards (RLVR) and standard benchmark evaluation both rely on an automatic verifier that turns a free text answer into a binary reward. Prior work reports that one evaluation harness accepts only about 94% of its own ground truth answers, blaming LaTeX parsing. That is an aggregate: it does not say which answer forms consume the error budget. We supply the decomposition. We apply metamorphic testing to the verifier rather than the model, generating certified equivalent answer variants, that is, rewrites that preserve mathematical meaning by construction, so that any rejection is a provable false negative needing no human adjudication. We then measure rejection per answer category across four widely used verifiers over 307,420 verdicts. We find three things. (1) Self validation ranges from 53.8% to 95.2% on identical inputs, a spread of 41.3 points. The published figure describes one implementation, not the task; two configurations of the same library disagree on 49.9% of pairs. (2) The residual is not spread across parsing categories but concentrated in whitespace and punctuation, which account for 93.0% of in contract failures for the default LaTeX configuration. A trailing period or newline dominates the budget. (3) Separating rejection from execution failure shows that verifiers with similar aggregate error fail for opposite reasons, and that a reference numeric cascade accepts off by one wrong answers as a step function of magnitude, from 0% below 10^4 to 100% at or above, because its relative tolerance is scale invariant.
1 Introduction
The paper audits automatic verifiers that convert mathematical answers into binary rewards, replacing aggregate reliability claims with certified, category-level evidence. Across implementations, failures vary sharply by configuration and concentrate in trivial string handling rather than mathematical parsing.
- Approach: Certified metamorphic testing constructs equivalent answers whose rejection is a provable false negative without human adjudication.Meaning-changing transformations similarly certify false positives when accepted.
- Findings: 93.0% of in-contract failures in the default LaTeX configuration come from whitespace and punctuation handling.A trailing period or newline can cause rejection despite preserving mathematical meaning.
- Findings: Separating rejection from execution failure shows that similar aggregate residuals can arise from opposite verifier behaviors.The audit therefore distinguishes whether a verifier rejects an answer or fails to return any verdict.
- Findings: SymPy-cascade accepts off-by-one adversarial answers as a magnitude step function, from 0% below 10^4 to 100% at or above 10^4.The pattern follows from scale-invariant relative tolerance and is invisible in aggregate error rates.
- Contributions: The audit contributes a certified protocol, per-category decomposition, contract matrix, coverage measure, and released verdict records.These artifacts support auditing implementation defects separately from specification ambiguity.
2 Method
The method constructs certified transformations over declared verifier contracts, then evaluates verdicts while distinguishing rejection from execution failure. It audits four implementations on corpus and synthetic answers using isolated, time-bounded executions.
- Certified equivalence: Semantics-preserving transformations make transformed gold answers correct by construction, so rejection certifies a false negative.Meaning-changing transformations provide the dual certificate for false positives when accepted.
- Transformation design: 43 transformations across 14 strata are partitioned into certified-equivalent, contract-dependent, and adversarial classes.Contract-dependent cases are reported as specification ambiguity rather than counted as defects.
- Contract handling: Contract assignments restrict defect scoring to strata each verifier claims to handle, while out-of-contract results remain reportable.The assignments are published because they involve judgement and can be contested.
- Coverage: A verifier can either reject a correct answer or fail to return a verdict through an exception, timeout, or crash.The audit reports all-input and evaluated rates, with zero-coverage cells marked N/A rather than 0.
- Coverage: Overall in-contract coverage is 96.5%, with the deficit concentrated in a single implementation.Coverage is defined as evaluated inputs divided by all inputs.
- Implementations: The audit compares LaTeX and plain-expression math-verify configurations, a DeepSeek-Math string normalizer, and a three-level numeric-symbolic cascade.The cascade uses exact string matching, numeric comparison with relative tolerance 10^-4, then SymPy.
- Data: The dataset contains 4,990 unique gold answers from GSM8K, MATH, Big-Math, and 2,000 synthetic answers.Synthetic answers strengthen sparse strata such as set and interval notation, while corpus-only results are reported separately.
- Execution: Each verification runs in an isolated subprocess with a five-second per-item budget, recording crashes and timeouts as distinct verdict classes.The pipeline uses five CPU nodes.
3 Results
Across identical inputs, self-validation varies substantially by implementation, while failures concentrate in whitespace and punctuation and can reflect either rejection or execution failure. Numeric tolerance also creates a sharp magnitude-dependent false-positive threshold.
- Self-validation: 41.3 points separate self-validation rates of 53.8% to 95.2% across implementations on identical inputs.The published ∼94% figure lies near the top of this range, characterising one harness rather than the task.
- Failure modes: 87.3% coverage with 100% correctness among judged inputs makes sympy-cascade’s residual entirely execution failure, unlike rejection-based defects.Separating coverage from judged accuracy distinguishes parse failures from rejected answers.
- Error decomposition: 93.0% of mv-latex failures and 74.1% of mv-expr failures arise from whitespace and punctuation.Representative pairs show acceptance of an expression followed by rejection when a period is appended.
- Scale-dependent false positives: sympy-cascade accepts off-by-one answers at or above 104 but rejects them below 104, whereas the other three verifiers reject them at every magnitude.A scale-invariant relative tolerance of 10^-4 accepts 10,001 against 10,000; the aggregate false-positive rate is 16.0%.
- Self-validation: 49.9% of certified-equivalent pairs receive different verdicts from two configurations of the same library.This disagreement shows that benchmark accuracy depends on the verifier configuration.
4 Discussion
Verifier evaluation should report configuration and coverage, because aggregate residuals can conceal different failure causes. The decomposition also identifies a concentrated formatting failure mode and a scale-dependent false-positive threshold.
- Failure decomposition: 93.0% of one verifier’s in-contract failures are whitespace and punctuation, making a trailing period or newline the dominant error source.The corresponding plain-expression configuration attributes 74.1% of in-contract failures to these categories.
- Configuration reporting: Verifier identity, version, extraction configuration, and runtime should accompany benchmark accuracy because configurations can disagree on 49.9% of cases.Acceptance on identical inputs ranges from 0%–100% by configuration in the discussion’s recommendation.
- Coverage: Two verifiers with similar aggregate residuals, 12.7% and 4.8%, fail for opposite reasons revealed only by separating coverage from accuracy.The paper recommends reporting coverage alongside accuracy.
- Measurement scope: Certified-transform measurements isolate formatting and equivalence false negatives and therefore provide a lower bound on false-negative rates measured on real model outputs.The two measurements are described as complementary because real outputs also contain mathematical non-equivalences.
5 Limitations
The study is limited to verifier-level measurements and does not establish downstream effects on model benchmark scores or RLVR training. Its evidence also comes from open implementations, includes synthetic answers, and uses contestable contract assignments.
- Verifier-level scope: The paper does not estimate effects on any model’s benchmark score because real outputs do not produce transform variants at equal rates.It also reports no training runs and does not measure downstream RLVR effects.
- System scope: The study covers open implementations only and makes no claims about closed frontier systems.This bounds the generality of its verifier conclusions.
- Contract scope: Contract assignment is a judgement call, and the published matrix is intended to make those assignments contestable.The limitation concerns which verifier behaviors count as in-contract defects.
6 Related Work
Related work frames verifier errors as reward-channel noise, systematic failure, and evaluation brittleness, while this paper audits the verifier directly with certified transformations and category-level decomposition. It also connects verifier auditing to broader benchmark and security-scanner evaluation practices.
- Verifier noise in RLVR: Cai et al. model verifier unreliability as asymmetric reward-channel noise, whereas this paper measures and decomposes the underlying false-negative and false-positive rates.The prior corrections require verifier error rates as inputs.
- Systematic error: Egashira et al. show that systematic verifier errors can drive plateaus or collapse, and this paper characterizes such patterns by answer category.The magnitude threshold is presented as a maximally systematic instance.
- Verifier auditing: The paper extends audits of rule-based and model-based verifiers with a false-negative category decomposition and certified adversarial probes.Its probes preserve or change meaning by construction to certify errors.
- Audit practice: Benchmark and security-scanner audits provide a broader precedent for corpus-scale checker evaluation, fault taxonomies, and released tooling.The paper adopts this auditing posture for natural-language answer verification.
- Metamorphic testing: Metamorphic-testing work typically perturbs model inputs, whereas this paper applies semantics-preserving transformations to answer strings and tests the grader.This reverses the usual system-under-test target.
- Evaluation brittleness: Prior evaluation work reports prompt sensitivity and heuristic-scoring artifacts, while this paper provides a verifier-side counterpart with Wilson intervals for every cell.This links verifier auditing to statistical rigor in evaluation reporting.
7 Conclusion
Verifier reliability varies substantially across widely used implementations, is dominated by whitespace handling, and includes deterministic false-positive mechanisms triggered by answer magnitude.
- 41 points separate verifier reliability across widely used implementations.
- Whitespace handling dominates verifier failures rather than mathematical parsing.
- Deterministic false positives can be triggered by answer magnitude.
A Supplementary Tables
Tables 5–8 document disagreement, contract-dependent acceptance, out-of-contract behavior, and certified-equivalent pairs rejected by production verifiers. They show that verifier behavior varies across implementations and answer forms, including sensitivity to trailing punctuation.
- Cross-verifier disagreement: 49.9% of certified-equivalent pairs receive different verdicts from V1 and V2, two configurations of the same library.The disagreement matrix is computed only where both verifiers return a verdict.
- Coverage: V4 has lower pair support because sympy-cascade does not return a verdict on every input.It is omitted from boxed-answer and scientific-notation acceptance rows for the same reason.
- Contract-dependent inputs: Acceptance spans 0–100% on identical inputs across contract-dependent classes, reflecting undocumented contract differences rather than defects.The classes include boxed answers, scientific notation, trailing units, the prefix “The answer is”, and percent signs.
- Out-of-contract behavior: Out-of-contract strata are reported for completeness and are not counted as defects.Table 7 separates unsupported inputs from failures within the claimed contract.
- Certified-equivalent rejections: A trailing period changes the verdict for a certified-equivalent pair rejected by production verifiers.The table presents representative equivalent variants, including a full fourth form with grouped factors.