Source-linked AI summary
Reinforcing General Reasoning without Verifiers
Xiangxin Zhou, Zichen Liu, Anya Sims, Haonan Wang, Tianyu Pang, Chongxuan Li, Liang Wang, Min Lin, Chao Du
TL;DR
R1-Zero-style reinforcement learning is limited when answers cannot be verified by rules, while model-based verifiers add dependence, reward-hacking risk, and computational overhead. VeriFree instead maximizes reference-answer likelihood through verifier-free policy optimization. It matches or surpasses verifier-based methods across general reasoning benchmarks while reducing practical burdens.
Problem
R1-Zero-style training is largely restricted to domains with feasible rule-based verification, and model-based verifier workarounds require strong verifier models, risk reward hacking, and add memory and computation costs.
Method
VeriFree generates a reasoning trace and uses the likelihood of the dataset reference answer as both a policy-gradient reward and a supervised-training weight.
Results
VeriFree matches or surpasses verifier-based alternatives across diverse reasoning tasks, including MMLU-Pro and SuperGPQA, while improving base-model accuracy by 12% to 40% on MMLU-Pro.
Takeaways & Limitations
VeriFree provides a practical verifier-free approach for training general-purpose reasoners where rule-based verifiers are infeasible and model-based verifiers are expensive or vulnerable to reward hacking.
Takeaways & Limitations
The objective’s equivalence to verifier-based RL is derived under a single exact-match correct-answer assumption.
Abstract
from arXiv · showhide
The recent paradigm shift towards training large language models (LLMs) using DeepSeek-R1-Zero-style reinforcement learning (RL) on verifiable rewards has led to impressive advancements in code and mathematical reasoning. However, this methodology is limited to tasks where rule-based answer verification is possible and does not naturally extend to real-world domains such as chemistry, healthcare, engineering, law, biology, business, and economics. Current practical workarounds use an additional LLM as a model-based verifier; however, this introduces issues such as reliance on a strong verifier LLM, susceptibility to reward hacking, and the practical burden of maintaining the verifier model in memory during training. To address this and extend DeepSeek-R1-Zero-style training to general reasoning domains, we propose a verifier-free method (VeriFree) that bypasses answer verification and instead uses RL to directly maximize the probability of generating the reference answer. We compare VeriFree with verifier-based methods and demonstrate that, in addition to its significant practical benefits and reduced compute requirements, VeriFree matches and even surpasses verifier-based methods on extensive evaluations across MMLU-Pro, GPQA, SuperGPQA, and math-related benchmarks. Moreover, we provide insights into this method from multiple perspectives: as an elegant integration of training both the policy and implicit verifier in a unified model, and as a variational optimization approach. Code is available at https://github.com/sail-sg/VeriFree.
1 Introduction
R1-Zero-style reinforcement learning has improved mathematical reasoning but remains difficult to extend beyond domains with rule-based answer verification. VeriFree addresses this gap by using reference-answer likelihoods instead of explicit verifiers, while retaining strong practical and empirical performance.
- Rule-based reinforcement learning has produced strong improvements in mathematical reasoning, but applies mainly where answer verification is feasible.
- Model-based verifiers extend the paradigm beyond rule-verifiable tasks but require a strong verifier, risk reward hacking, and add training-time memory and computation costs.
- VeriFree generates a reasoning trace, appends the dataset reference answer, and uses the reference-answer likelihood as both a policy-gradient reward and supervised-learning weight.
- Under a unique-answer setting, VeriFree is expectation-equivalent to RL with verifiable rewards while providing lower-variance estimation; with multiple valid answers, one reference still supplies a sufficient learning signal empirically.
- Across diverse reasoning tasks, VeriFree matches or often outperforms verifier-based alternatives while being simpler, faster, less memory-intensive, and more robust.
2 Methodology
The methodology replaces answer-verifier rewards with reference-answer likelihoods while preserving the verifier-based objective under a unique-answer assumption. Its derivation marginalizes over sampled answers, reducing gradient variance, and its implementation addresses tokenization and reasoning-trace extraction challenges.
- Verifier-based reinforcement learning: R1-Zero-style RL parses each response into reasoning trace z and answer y, then uses a verifier to assign a correctness reward against reference answer y⋆.
- Verifier-based reinforcement learning: Because correctness verification is difficult in general reasoning tasks, the paper proposes a verifier-free extension of the training paradigm.
- VeriFree policy optimization: Under a single exact-match correct answer, VeriFree derives an equivalent objective by using the probability of y⋆ conditioned on x and sampled reasoning trace z.
- VeriFree policy optimization: VeriFree’s objective and gradient estimator are expectation-equivalent to verifier-based counterparts, combining a reasoning policy-gradient term with a reward-weighted reference-answer term.
- VeriFree policy optimization: Analytically marginalizing the sampled answer removes one source of Monte Carlo randomness, yielding lower variance through Rao-Blackwellization.
- Variance reduction and implementation: The estimator supports RLOO, GRPO normalization, and PPO clipping, with multiple responses sampled per prompt and corrected response-length normalization.
- Comparison to existing approaches: Unlike JLB and LaTRO, VeriFree weights the reference-answer term by answer likelihood, down-weighting low-quality reasoning traces instead of reinforcing mismatched answers.
- Variance reduction and implementation: Tokenization-aware splitting ends z at “<answer” rather than “<answer>”, maintaining token-space alignment and avoiding optimization instability from inconsistent boundaries.
3 Experiments
The experiments evaluate VeriFree across general reasoning benchmarks, training settings, and ablations. VeriFree improves accuracy, learning efficiency, confidence alignment, and cross-domain transfer while exposing sensitivity to tokenization and answer-equivalence handling.
- Setup: VeriFree is evaluated using Qwen3 base models at 1.7B, 4B, and 8B parameters, with direct RL fine-tuning that skips supervised fine-tuning.The training pipeline uses Oat and the paper’s specified prompt template.
- Setup: The evaluation covers MMLU-Pro, SuperGPQA, and GPQA, spanning general and graduate-level reasoning across diverse disciplines.MMLU-Pro evaluates multiple domains, SuperGPQA spans 285 disciplines, and GPQA targets graduate-level questions resistant to shallow matching.
- Main Results: 12% to 40% average-accuracy gains are obtained over base models on MMLU-Pro after applying VeriFree.The reported gains are measured across model scales and domains in Table 1.
- Main Results: VeriFree reaches performance comparable to or higher than the instruct model in thinking mode and the Verifier baseline on SuperGPQA without explicit verification signals.The comparison is reported across base-model evaluations on SuperGPQA.
- Main Results: VeriFree improves learning efficiency by achieving higher accuracy with fewer training steps than the model-based-verifier baseline.The paper attributes this to continuous reward signals and the RLOO objective, which reduce gradient variance and stabilize policy gradients.
- Main Results: MMLU-Pro accuracy correlates positively with average confidence πθ(y⋆|x, z), with ρ = 0.82 during Qwen3-8B training.The authors use this self-estimated confidence as a proxy for emergent reasoning capability.
- Main Results: Training without math examples still improves general benchmarks and transfers to math-specific tasks.This experiment evaluates whether reasoning learned from non-math data extends across domains.
- Ablation Study: Removing RLOO lowers final accuracy by more than 3% and causes premature convergence during MMLU-Pro training.The ablation compares Qwen3-1.7B models with and without RLOO.
4 Related Works
Prior R1-Zero-style work applies RL with verifiable rewards primarily to mathematics and coding. Research on broader reasoning domains instead uses self-improvement signals or additional LLMs as proxy verifiers or reward models.
- DeepSeek-R1-Zero-like reinforcement learning: R1-Zero-style training has reproduced strong reasoning improvements at smaller scales but remains concentrated in mathematics and coding, where rewards are readily verifiable.The related work describes rule-based verification as the common basis for these settings.
- Reasoning beyond verifiable domains: General-reasoning approaches commonly use a second LLM as a proxy verifier or reward model when verifiable rewards are unavailable.Examples include NaturalReasoning, Su et al., General-Reasoner, and X-Reasoner.
- Self-improving language models: Other self-improving methods derive training signals from model-generated judgments, implicit DPO rewards, or majority voting.These approaches use the model’s own outputs for ranking, preference learning, or self-labeled RL rewards.
5 Conclusions
The paper introduces VeriFree, a verifier-free approach that rethinks RLVR and eliminates explicit verification. It is designed for general reasoning tasks where rule-based verifiers are infeasible and model-based verifiers are costly or vulnerable to reward hacking.
- VeriFree rethinks RLVR from a new perspective by deriving an optimization objective that eliminates explicit verification.The derivation leverages gradient equivalence under the unique-answer assumption.
- VeriFree is suited to general reasoning tasks where rule-based verifiers are infeasible and model-based verifiers are expensive and vulnerable to reward hacking.The authors report extensive experiments and ablations across general reasoning benchmarks.
A.1 Derivation of Gradient Estimators
This appendix derives the gradient estimators for VeriFree and the standard verifier-based objective. It presents the derivations as complementary formulations for comparison.
- It also derives the corresponding standard verifier-based gradient estimator for completeness.
- The appendix derives the gradient estimator for VeriFree.
A.2 Proof of Lower Variance
The proof establishes that VeriFree has policy-gradient estimator variance no greater than the verifier-based estimator. The argument uses conditional expectation and the law of total variance, and the method also supports PPO-style clipping for reused data.
- VeriFree’s policy-gradient estimator has variance less than or equal to the verifier-based estimator for any given input and reference answer.The same relationship holds for global objectives averaged over the data distribution.
- The lower-variance result follows because the VeriFree estimator is the conditional expectation of the verifier-based estimator over sampled answers.The conditioning fixes the question, reference answer, and reasoning trace while averaging over the answer sampling.
- The proof applies the law of total variance, whose non-negative conditional-variance term yields the lower-variance relationship.
- VeriFree remains compatible with PPO-style gradient clipping when data is reused to improve sample efficiency.The off-policy estimator uses the sampling policy and PPO clipping operation.
C Dataset Details
The dataset details section visualizes the category distribution of WebData, the training dataset, in Figure 7.
- Figure 7 visualizes the category distribution in WebData, the training data.
D Extended Empirical Results
The paper reports benchmark evaluations across broad-domain reasoning and mathematics, using domain breakdowns for MMLU-Pro and SuperGPQA and accuracy comparisons for math benchmarks and GPQA.
- MMLU-Pro results are organized by fourteen domains, including computer science, mathematics, chemistry, engineering, law, biology, health, business, and economics.
- SuperGPQA results are organized across thirteen domains, including engineering, medicine, science, philosophy, economics, management, law, and education.
- Table 3 compares accuracy across six math evaluation benchmarks and GPQA.
- VeriFree-trained models demonstrate consistent and significant improvements over base models on the math evaluation suite and GPQA.
E Case Study
The case studies illustrate VeriFree-generated reasoning on astronomy, psychology, and hazardous-materials questions, including answers that match the provided references in the examples.
- For the skull-and-crossbones question, the model identifies Class 6 hazardous materials, specifically Division 6.1 for toxic substances.
- For an astronomy question, the model identifies Cepheid variables as the objects used with the period-luminosity relation to determine distances.The example explains that Cepheid luminosity variation periods relate to intrinsic luminosity and supports distance estimation.
- The astronomy example describes measuring a Cepheid’s period, determining intrinsic luminosity, comparing apparent luminosity, and calculating distance.
- For the Kohlberg question, the model selects Stage 2’s personal-needs explanation because this stage involves exchange and self-interest.
- The broader-impact discussion cautions that improved reasoning can amplify beneficial or harmful behavior depending on user intent and oversight.
- The paper encourages safeguards for misuse prevention and performance guarantees under distributional shifts as reasoning models enter real-world workflows.