TinyV: Reducing False Negatives in Verification Improves RL for LLM Reasoning

Zhangchen Xu; Yuetai Li; Fengqing Jiang; Bhaskar Ramasubramanian; Luyao Niu; Bill Yuchen Lin; Radha Poovendran

Source-linked AI summary

TinyV: Reducing False Negatives in Verification Improves RL for LLM Reasoning

Zhangchen Xu, Yuetai Li, Fengqing Jiang, Bhaskar Ramasubramanian, Luyao Niu, Bill Yuchen Lin, Radha Poovendran

arXiv:2505.14625v2May 20, 2025cs.LGcs.AIcs.CL

TL;DR

False negatives in rule-based verifiers can reject correct LLM reasoning outputs, weakening the reward signals used for RL. The paper analyzes their prevalence and training impact, then introduces TINYV, which improves reward accuracy and reports up to 10% higher pass rates with faster convergence. Its broader applicability remains open beyond mathematical reasoning and depends on the stated GRPO assumption.

Problem
False negatives in rule-based verifiers reject correct outputs, reducing informative gradient signals and slowing RL convergence for LLM reasoning.
Method
TINYV is a lightweight LLM-based verifier that augments Prime Verifier by re-evaluating negative decisions and correcting potential false negatives.
Results
TINYV improves final performance and convergence, with up to a 10% pass-rate improvement over baseline verifiers across evaluated mathematical reasoning benchmarks.
Takeaways & Limitations
More accurate reward signals from TINYV enhance RL training efficiency and final model performance on mathematical reasoning tasks.
Takeaways & Limitations
The paper leaves evaluation of false negatives in broader RL domains, including theorem proving, medicine, software engineering, and robotics, to future work.

Abstract

from arXiv · show

Reinforcement Learning (RL) has become a powerful tool for enhancing the reasoning abilities of large language models (LLMs) by optimizing their policies with reward signals. Yet, RL's success relies on the reliability of rewards, which are provided by verifiers. In this paper, we expose and analyze a widespread problem--false negatives--where verifiers wrongly reject correct model outputs. Our in-depth study of the Big-Math-RL-Verified dataset reveals that over 38% of model-generated responses suffer from false negatives, where the verifier fails to recognize correct answers. We show, both empirically and theoretically, that these false negatives severely impair RL training by depriving the model of informative gradient signals and slowing convergence. To mitigate this, we propose tinyV, a lightweight LLM-based verifier that augments existing rule-based methods, which dynamically identifies potential false negatives and recovers valid responses to produce more accurate reward estimates. Across multiple math-reasoning benchmarks, integrating TinyV boosts pass rates by up to 10% and accelerates convergence relative to the baseline. Our findings highlight the critical importance of addressing verifier false negatives and offer a practical approach to improve RL-based fine-tuning of LLMs. Our code is available at https://github.com/uw-nsl/TinyV.

1 Introduction

The paper identifies false negatives in rule-based verification as an underexplored threat to RL for LLM reasoning. It analyzes their training impact and introduces TINYV, reporting improved performance and convergence.

Rule-based verifiers can reject mathematically correct answers because string-matching and parsing criteria fail to recognize equivalent outputs.Figure 1 illustrates this failure for Prime Verifier and Math Verify.
High false-negative rates can exclude hard-to-verify problems from training and evaluation and introduce reward signals that mislead policy optimization.The paper highlights natural-language elements and complex LaTeX expressions as especially difficult verification cases.
False negatives reduce informative gradient signals, hinder learnability measured by reverse KL divergence, and slow RL convergence.The paper reports both empirical and theoretical evidence for this training impairment.
Up to 10% higher pass rates and faster convergence are reported for TINYV relative to baseline verifiers across the evaluated mathematical reasoning benchmarks.TINYV augments rule-based verification while maintaining computational efficiency.
TINYV is a lightweight LLM-based verifier that augments existing rule-based methods to correct false negatives and improve reward accuracy.

2 Preliminaries

The preliminaries formalize RL policy optimization, GRPO, and verifier-based binary rewards. These components define how verification outcomes become training signals.

RL optimizes policy parameters by maximizing expected rewards while constraining deviation from an initial policy through KL divergence.The reward function supplies the supervision signal for policy optimization.
Group Relative Policy Optimization (GRPO): GRPO computes policy gradients from group-based advantages, avoiding the parameterized value model used in PPO.
Group Relative Policy Optimization (GRPO): For each prompt, GRPO samples n responses and computes each response’s advantage from the group’s reward mean and variance.A smoothing constant ε prevents division by zero.
Verification and Reward Calculation in RL: The verifier assigns a binary reward of 1 when a generated response is equivalent to the ground truth and 0 otherwise.The response and reference are compared for equivalence, optionally using the input prompt.
Verification and Reward Calculation in RL: In practice, the system extracts answers from structured formats such as \boxed{} to simplify verification.

3 Discovering and Analyzing False Negatives from the Wild

The study measures false negatives in Big-Math-RL-Verified using LLM re-annotation and develops a taxonomy of their causes. It finds that false negatives are common and arise mainly from language, formatting, and notation differences.

The dataset study uses Big-Math-RL-Verified and generates four responses per problem with DEEPSEEK-R1-DISTILL-QWEN-7B to examine false negatives in open-domain settings.The dataset contains more than 250,000 mathematical problems with ground-truth solutions and pass-rate metadata.
Two LLM annotators re-evaluate Prime Verifier’s incorrect responses, and manual review of 200 sampled false negatives finds 99.5% annotation accuracy.The review identified one incorrectly marked response and three questions with incorrect ground-truth answers.
38.5% of 226K seemingly incorrect prompt-response pairs were correct responses misclassified by Prime Verifier, while 42.1% of 95K prompts lacked recognized correct responses.CN_K12 had the highest source-level false-negative rate, exceeding 50%.
The analysis organizes false negatives into seven broad error classes and 31 finer subcategories spanning formatting, notation, and semantic misunderstandings.GROK-3-MINI-HIGH automatically labels prompts exhibiting at least one false negative.
Language differences are the most prevalent false-negative source, followed by formatting issues and notation discrepancies.Examples include natural-language elements, missing whitespace or delimiter changes, and intervals versus inequalities.

4 Analysis of False Negatives and Their Impact on RL Training

False negatives accumulate during RL and reduce the informative reward variation available for policy updates, especially early in training. Theoretical analysis further shows that false-negative rewards lower step-wise learnability and slow convergence.

Empirical Analysis: Prompts with partially correct rollouts provide useful gradient signals, whereas all-correct or all-incorrect rollouts provide none for RL updates.Prompt efficiency is defined as the fraction of prompts with pass rates strictly between 0 and 1.
Empirical Analysis: 46.7% of prompts for QWEN2.5-7B and 50.5% for QWEN2.5-MATH-7B experience at least one false-negative rollout by training’s end.The false-negative ratio also reaches 20% of rollouts on average at individual training steps.
Empirical Analysis: A 15% early-training gap in prompts without correct rollouts reduces prompt efficiency under Prime Verifier relative to LLM annotations.Prime Verifier marks 50% of prompts as having no correct rollouts, compared with 35% under LLM annotations; the gap narrows later.
Theoretical Analysis: The theoretical framework assumes that average ground-truth reward scores improve over iterations and do not grow exponentially during training.These assumptions support the comparison between ground-truth and false-negative reward policies.
Theoretical Analysis: The theoretical learnability gap δk = Dk,GT−Dk,FN remains positive at every training step under the paper’s assumptions.Because larger reverse KL indicates greater policy improvement, the theorem implies greater step-wise learnability with ground-truth rewards than with false-negative rewards.

5 Improve RL by Detecting False Negatives with TINYV

TINYV is a lightweight verifier that augments Prime Verifier to recover false negatives, improve reward accuracy, and support more efficient RL training on hard-to-verify mathematics.

Curation and Deployment: The verifier is trained on a hybrid dataset of real and synthetic false-negative and true-negative examples to generalize across diverse verification patterns.Its compact Qwen2.5-1.5B-Instruct model uses the answer, ground truth, and problem context for binary classification.
Curation and Deployment: TINYV is queried only after Prime Verifier rejects a response, then re-evaluates that response to recover potential false negatives while minimizing computational overhead.This add-on deployment focuses computation on challenging cases rather than responses already accepted by the rule-based verifier.
HardVerify-Math Benchmark: HardVerify-Math contains 250 hard-to-verify answers selected from Olympiad, MATH, and Big-Math to evaluate scenarios prone to false negatives.The benchmark addresses the limited coverage of existing datasets, which often emphasize easily verifiable answers.
Experimental Results: Up to 10% higher pass rates are achieved by TINYV across HardVerify-Math, while training converges faster and final performance improves relative to Prime Verifier.The reported comparisons cover multiple mathematical reasoning benchmarks and identify faster convergence alongside stronger final performance.
Experimental Results: TINYV trained on Big-Math reaches 68.68% final accuracy on HardVerify-Math with Qwen2.5-7B, versus 58.64% for Prime Verifier and lower generalization from DeepScaleR.The advantage also extends to MATH500 and Olympiad Bench, where answer formats can be difficult to verify.
Experimental Results: TINYV incurs only 6% overhead while retaining its lightweight design.Additional experiments compare TINYV with TINYV-THINK, Math Verify, and Prime Verifier, and separately assess training costs.

6 Conclusion and Future Work

The work shows that addressing false negatives improves reward accuracy and RL outcomes for mathematical reasoning, while identifying broader domains for future study.

TINYV improves final performance and accelerates convergence compared with baseline verifiers while maintaining computational efficiency.
The study focuses on false negatives in mathematical reasoning and proposes TINYV to enhance reward accuracy during RL training.
Future work should examine false negatives in theorem proving, medical applications, software engineering, and robotics.

B Limitations and Broader Impacts

The paper’s scope is mathematical reasoning, with broader-domain generalization and full-output verification remaining open limitations.

Limitations: The study does not investigate false negatives in domains such as theorem proving, medical applications, or software engineering.
Limitations: The experiments and theory primarily use GRPO, while generalization to other online and offline RL methods remains empirically unvalidated.
Limitations: TINYV relies on Prime Verifier’s extraction within \boxed{} and evaluates the final answer rather than the full reasoning process.
Broader Impacts: The work reports no identified negative societal impacts and frames its broader impact as improving RL efficiency for mathematical reasoning.
False-negative taxonomy: The false-negative taxonomy includes formatting, notation, algebraic-form, numerical, language, collection-order, and symbol variations that preserve mathematical correctness.

C.2 Mathematical Notation Variations

The paper catalogs mathematically equivalent answer representations and formalizes step-wise learnability using reverse KL divergence between consecutive policies.

Mathematical notation variations: Equivalent answers may differ in interval, ratio, solution-set, vector, delimiter, decimal, or regional notation while preserving meaning.
Mathematical notation variations: Equivalent algebraic forms include reordered, factored, simplified, radical, exponent, logarithmic, and trigonometric representations.
Mathematical notation variations: Numerical equivalence can involve exact versus decimal or percentage forms, scientific notation, rounding, and mixed versus improper fractions.
Mathematical notation variations: Language and collection variations include explanatory text, explicit assignments, phrasing, separators, element order, delimiters, and redundant nesting.
Step-wise learnability: The analysis represents the learnability gap through partial derivatives and proves positivity under the stated domain condition b < a < 2b.
Step-wise learnability: Step-wise learnability is defined as reverse KL divergence between policies at consecutive optimization steps, comparing ground-truth and false-negative rewards.

D.3 Proof of Theorem 1

The proof establishes that ground-truth rewards yield greater step-wise learnability than rewards containing false negatives, implying slower learning under false-negative feedback.

Proof of Theorem 1: The proof expresses the learnability gap as an integral and shows its integrand is positive throughout the integration domain.
Proof of Theorem 1: δk = Dk,GT − Dk,FN is positive for every training step k under Lemma 1 and Assumption 1.
Proof of Theorem 1: The result implies that false-negative rewards reduce step-wise policy improvement and can lead to slower convergence and suboptimal performance.

E.1 Experimental Setups for Zero RL

The experiments combine real and synthetic false-negative examples to train TINYV, with supervised fine-tuning and zero-RL evaluations using specified infrastructure.

Zero RL Training: Zero-RL experiments use VERL on 8 A100 GPUs with hyper-parameters listed in Table 2.
Real Example Generation: Real examples pair Prime Verifier rejections with LLM-generated correctness labels, while retaining intermediate analyses for TINYV-THINK training.
Synthetic Example Generation: Synthetic examples expand coverage and balance labels by varying formatting, numerical approximations, and mathematically equivalent expressions.
Supervised Fine-Tuning: TINYV supervised fine-tuning uses Llama Factory on 8 A100 GPUs, with training and inference specified by Figure 8 and hyper-parameters in Table 3.

F HardVerify-Math Benchmark

HardVerify-Math Bench targets answers that are difficult for verifiers because of complex formats and false-negative-prone cases. Its examples illustrate mathematically equivalent, semantically matching, and differently formatted model answers, while benchmark comparisons evaluate TINYV against existing verifiers.

Benchmark Construction: HardVerify-Math Bench contains 250 hard-to-verify answers spanning the paper’s false-negative categories, including 115 Olympiad, 10 MATH, and 125 Big-Math questions.
False-Negative Identification: The labeling task asks whether a model answer is correct despite appearing different from the ground truth.
False-Negative Examples: The benchmark includes cases where correct answers differ from references through equivalent expressions, alternative formatting, or semantically matching wording.
Benchmark Evaluation: Figure 11 compares TINYV, TINYV-THINK, Math Verify, and Prime Verifier on AMC, MATH, Olympiad, and HardVerify-Math Bench using QWEN2.5-MATH-7B.

G.2 Training Cost Analysis

TINYV provides performance comparable to TINYV-THINK while reducing training time and maintaining near-baseline computational cost during GRPO training. The section also documents prompts and examples used to label and generate false negatives.

Training Cost: TINYV adds only 6% computational cost per GRPO training step compared with Prime Verifier.The average time per step remains comparable, with checkpoint saving producing the peak time cost.
Prompt Design: The appendix provides prompt templates for labeling false-negative responses and categories, generating synthetic examples, and training or inferring with TINYV.
False-Negative Taxonomy: False-negative categories include mathematically equivalent expressions and set or list presentation differences that preserve correctness.