Source-linked AI summary
RLPR: Extrapolating RLVR to General Domains without Verifiers
Tianyu Yu, Bo Ji, Shouli Wang, Shu Yao, Zefan Wang, Ganqu Cui, Lifan Yuan, Ning Ding, Yuan Yao, Zhiyuan Liu, Maosong Sun, Tat-Seng Chua
TL;DR
RLVR is limited beyond mathematics and code because it depends on complex, domain-specific verifiers. RLPR replaces those verifiers with intrinsic reference-answer token probabilities and stabilizing methods, improving reasoning across general and mathematical benchmarks without external verifiers.
Problem
RLVR remains largely confined to mathematics and code because domain-specific verifiers require substantial engineering and restrict application to new domains.
Method
RLPR uses average decoding probabilities of reference-answer tokens as an intrinsic reward, with debiasing and adaptive standard-deviation filtering for training stability.
Results
RLPR substantially improves reasoning across seven benchmarks without external verifiers, including 7.6 points on TheoremQA and 7.5 on Minerva over VeriFree.
Takeaways & Limitations
Intrinsic probability rewards allow RLPR to extend RLVR-style training to general domains while outperforming naive likelihood and verifier-model approaches.
Abstract
from arXiv · showhide
Reinforcement Learning with Verifiable Rewards (RLVR) demonstrates promising potential in advancing the reasoning capabilities of LLMs. However, its success remains largely confined to mathematical and code domains. This primary limitation stems from the heavy reliance on domain-specific verifiers, which results in prohibitive complexity and limited scalability. To address the challenge, our key observation is that LLM's intrinsic probability of generating a correct free-form answer directly indicates its own evaluation of the reasoning reward (i.e., how well the reasoning process leads to the correct answer). Building on this insight, we propose RLPR, a simple verifier-free framework that extrapolates RLVR to broader general domains. RLPR uses the LLM's own token probability scores for reference answers as the reward signal and maximizes the expected reward during training. We find that addressing the high variance of this noisy probability reward is crucial to make it work, and propose prob-to-reward and stabilizing methods to ensure a precise and stable reward from LLM intrinsic probabilities. Comprehensive experiments in four general-domain benchmarks and three mathematical benchmarks show that RLPR consistently improves reasoning capabilities in both areas for Gemma, Llama, and Qwen based models. Notably, RLPR outperforms concurrent VeriFree by 7.6 points on TheoremQA and 7.5 points on Minerva, and even surpasses strong verifier-model-dependent approaches General-Reasoner by 1.6 average points across seven benchmarks.
RLPR: EXTRAPOLATING RLVR TO GENERAL DOMAINS WITHOUT VERIFIERS
Figure 1 compares RLPR with other methods on general-domain and mathematical reasoning benchmarks, while reporting average accuracy and verifier requirements.
- RLPR is compared on both general-domain and mathematical reasoning benchmarks.
- Average denotes the average accuracy across five benchmarks.
- Verifier requirements for different methods are listed in parentheses.
1 INTRODUCTION
RLPR addresses RLVR’s confinement to mathematics and code by replacing domain-specific verifiers with intrinsic probability rewards and stabilization methods. Experiments across models and benchmarks report improvements in general and mathematical reasoning without external verifiers.
- RLVR methods remain largely confined to mathematics and code because domain-specific verifiers require costly engineering and limit scalability.
- RLPR uses the LLM’s intrinsic probability of generating a correct answer as a reward for reasoning.
- Probability-based reward averages decoding probabilities of reference-answer tokens and is debiased against text-related reward bias.
- Adaptive standard-deviation filtering removes prompts with low reward variation to stabilize training and improve final performance.
- RLPR substantially improves reasoning across seven mathematical and general-domain benchmarks without external verifiers.
- 7.6 points on TheoremQA and 7.5 on Minerva are reported improvements over VeriFree, while gains also appear across Qwen, Llama, and Gemma models.
2 RLPR
RLPR replaces domain-specific verifiers with intrinsic reference-answer probabilities as a reward, then debiases and stabilizes that signal for reasoning optimization. The framework addresses variance and latent-factor bias while extending verifier-based reinforcement learning beyond mathematics and code.
- RLVR background: RLVR optimizes a rule-based verifier score for generated reasoning and answers, but handcrafted verifier design restricts extension to new domains.Task-specific symbolic verifiers and sandboxed execution are cited as common implementations.
- Probability reward: RLPR uses the model’s per-token probabilities for a training reference answer as a reward, avoiding external domain-specific verifiers.The generated answer is replaced with the reference answer, and the resulting sequence is scored by the policy model.
- Probability reward: Average token probabilities provide a more robust reward than sequence likelihood, whose product is highly sensitive to small token-probability differences.The paper reports stronger correlation with answer quality for the adopted aggregation and notes length normalization may matter for some algorithms.
- Reward debiasing: RLPR debiases probability rewards by subtracting a reference-only base score and clipping the result to [0, 1].This models the reward as the probability improvement attributable to generated reasoning and is reported to stabilize training and improve robustness.
- Training stabilization: Because continuous probability rewards make universal accuracy thresholds difficult, RLPR filters prompts with low reward standard deviation to stabilize training.Low standard deviation identifies prompts producing consistently high or low bounded rewards, analogous to excluding too-easy or too-difficult prompts.
3 EXPERIMENTS
Experiments evaluate RLPR across seven reasoning benchmarks, reward-quality analyses, ablations, and verifiable mathematical domains. RLPR improves reasoning across general and mathematical settings while producing robust probability-based rewards without external verifiers.
- Experimental Setup: Experiments cover four general-domain and three mathematical reasoning benchmarks using Gemma2, Llama3.1, and Qwen2.5 models.Unless otherwise specified, experiments use Qwen2.5-7B-Base; evaluations include MMLU-Pro, GPQA, TheoremQA, WebInstruct, MATH-500, Minerva, and AIME24.
- Main Results: 24.9% higher average performance across four general-domain benchmarks is achieved by RLPR on Qwen2.5-7B without an external verifier.RLPR also exceeds the RLVR baseline on Gemma, Llama, and Qwen, with gains of 1.4, 3.9, and 1.4 average points respectively.
- Main Results: 1.6 average points over General Reasoner are achieved across seven benchmarks, despite General Reasoner using a trained 1.5B-parameter verifier model.RLPR also improves over VeriFree by 7.6 points on TheoremQA and 7.5 points on Minerva.
- Probability-Based Reward Analysis: PR discriminates correct responses better than the rule-based verifier on general data, which achieves only 0.61 ROC-AUC there.PR consistently provides high-quality rewards across mathematical and general domains, while the rule-based verifier struggles with natural-language complexity.
- Probability-Based Reward Analysis: PR outperforms verifier models across both domains, improving by at least 2% on mathematical data and 20% on general-domain data.PR computes rewards in a single forward pass and eliminates text post-processing.
- Probability-Based Reward Analysis: PR remains effective with Qwen2.5-0.5B and shows negligible correlation with response length or decoding entropy.The average Spearman correlations are -0.060 for length and 0.059 for entropy, with only 8% of prompts having p-values below 0.05.
- RLPR on Verifiable Domains: General-domain data improves TheoremQA by 1.9 points and Minerva by 4.3 points, while rule-based verifiers show diminished performance on that data.Combining PR with rule-based rewards also improves data utilization in verifiable mathematical domains by providing finer discrimination among responses with identical correctness labels.
4 RELATED WORKS
Prior work relies on verifiers or self-rewarding, while RLPR uses reference-answer decoding probabilities to extend verifier-based reinforcement learning to general domains without length constraints.
- Reinforcement Learning with Verifiable Rewards: RLVR commonly advances reasoning on mathematics and code through binary verifiable rewards and domain-specific verifiers.Examples include Math-Verify, SandboxFusion, and custom implementations.
- Reasoning in General Domains: General-domain reasoning research includes generative reward models and verifier or judge models that assess rollout quality.These approaches use another generative model as the evaluator or improve the judge model's reasoning ability.
- RLPR: RLPR uses the decoding probability of the reference answer as a reward signal for general-domain reinforcement learning.The approach relies on robust per-token probabilities and removes the short-answer limitation of a concurrent method restricted to answers under 7 tokens.
- Self-Reward Optimization: Unlike self-rewarding methods that raise the probability of consistent answers and may restrict exploration, RLPR builds rewards from the reference answer while maintaining healthy token entropy.The paper attributes this entropy behavior to its clip-high trick.
5 CONCLUSION
RLPR extends RLVR to broader domains without external verifiers and improves reasoning on both general and mathematical tasks across several model families.
- 5 CONCLUSION: RLPR extends reinforcement learning with verifiable rewards to broader general domains without using external verifiers.The framework introduces probability rewards and reward debiasing to improve reward quality.
- 5 CONCLUSION: RLPR improves reasoning on both general and mathematical tasks for Gemma, Llama, and Qwen models.The experiments cover seven benchmarks and report significant improvement across both domains.
- 5 CONCLUSION: A standard-deviation filtering strategy stabilizes training by removing samples with low reward standard deviation.The conclusion also identifies multimodal understanding and larger models as future directions.
A.1 EXPERIMENTAL DETAILS
The experiments post-train Qwen2.5-7B directly with GRPO and use prompt changes to make thoughts and answers extractable during training and validation.
- A.1 EXPERIMENTAL DETAILS: Experiments use Qwen2.5-7B as the default base model and directly post-train it without supervised fine-tuning.GRPO is the default training algorithm.
- A.1 EXPERIMENTAL DETAILS: The training and validation prompt templates are changed to control response structure and produce extractable thoughts and answers.The adopted training prompt is the R1 prompt, shown in Table 5.
- A.1 EXPERIMENTAL DETAILS: GRPO is used by default alongside direct post-training on the base model.The setup forgoes the supervised fine-tuning process.
A.1.1 PARAMETER SETTINGS
The reported parameter settings use substantial GPU resources, an entropy penalty, no KL penalty, and a small policy learning rate.
- A.1.1 PARAMETER SETTINGS: Each experiment is trained on 32 NVIDIA A100 GPUs.This is the stated hardware allocation for each experiment.
- A.1.1 PARAMETER SETTINGS: The policy model uses a learning rate of 5e-7.This is the reported learning rate for policy optimization.
A.1.2 TRAINING LOGS
RLPR training on Qwen2.5-7B shows increasing response lengths, rapid structural learning, and stable entropy without degeneration or collapse. These dynamics support a balance between exploration and exploitation.
- Training dynamics: Response length steadily increases during training, allowing more profound reasoning behaviors without signs of degeneration.The response-length trend is shown in Fig. 6a.
- Training dynamics: The policy model quickly learns to follow the intended response structure.This behavior is reported in Fig. 6b.
- Training dynamics: Training entropy neither collapses from the clip-high trick nor increases abruptly, preserving exploration–exploitation balance.The entropy trend is shown in Fig. 6c.
A.2 PASS@K EVALUATION
RLPR is evaluated using Pass@k curves against standard RLVR and General Reasoner. It achieves comparable or better Pass@k accuracy, indicating that improved Pass@1 does not reduce reasoning potential.
- Pass@k evaluation: RLPR achieves comparable or better Pass@k accuracy than standard RLVR and General Reasoner.The comparison covers various tasks shown in Fig. 7.
- Pass@k evaluation: RLPR does not trade reasoning potential for Pass@1 improvements.The authors draw this conclusion from the Pass@k comparison with standard RLVR and General Reasoner.
- Pass@k evaluation: Figure 7 presents Pass@k curves comparing RLPR with baseline methods.The figure is used to assess the method’s potential reasoning boundary across various tasks.