PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcing General Reasoning without Verifiers

Xiangxin Zhou, Zichen Liu, Anya Sims, Haonan Wang, Tianyu Pang, Chongxuan Li, Liang Wang, Min Lin, Chao Du

arXiv:2505.21493v1cs.LGcs.CL

TL;DR

R1-Zero-style reinforcement learning is limited when answers cannot be verified by rules, while model-based verifiers add dependence, reward-hacking risk, and computational overhead. VeriFree instead maximizes reference-answer likelihood through verifier-free policy optimization. It matches or surpasses verifier-based methods across general reasoning benchmarks while reducing practical burdens.

  • Problem

    R1-Zero-style training is largely restricted to domains with feasible rule-based verification, and model-based verifier workarounds require strong verifier models, risk reward hacking, and add memory and computation costs.

    Source: §1 · p. 2, §2.1 · p. 3

  • Method

    VeriFree generates a reasoning trace and uses the likelihood of the dataset reference answer as both a policy-gradient reward and a supervised-training weight.

    Source: §1 · p. 2

  • Results

    VeriFree matches or surpasses verifier-based alternatives across diverse reasoning tasks, including MMLU-Pro and SuperGPQA, while improving base-model accuracy by 12% to 40% on MMLU-Pro.

    Source: §3.2 · p. 7, §5 · p. 10

  • Takeaways & Limitations

    VeriFree provides a practical verifier-free approach for training general-purpose reasoners where rule-based verifiers are infeasible and model-based verifiers are expensive or vulnerable to reward hacking.

    Source: §5 · p. 10

    The objective’s equivalence to verifier-based RL is derived under a single exact-match correct-answer assumption.

    Source: §2.2 · p. 3

Abstract

from arXiv · show

The recent paradigm shift towards training large language models (LLMs) using DeepSeek-R1-Zero-style reinforcement learning (RL) on verifiable rewards has led to impressive advancements in code and mathematical reasoning. However, this methodology is limited to tasks where rule-based answer verification is possible and does not naturally extend to real-world domains such as chemistry, healthcare, engineering, law, biology, business, and economics. Current practical workarounds use an additional LLM as a model-based verifier; however, this introduces issues such as reliance on a strong verifier LLM, susceptibility to reward hacking, and the practical burden of maintaining the verifier model in memory during training. To address this and extend DeepSeek-R1-Zero-style training to general reasoning domains, we propose a verifier-free method (VeriFree) that bypasses answer verification and instead uses RL to directly maximize the probability of generating the reference answer. We compare VeriFree with verifier-based methods and demonstrate that, in addition to its significant practical benefits and reduced compute requirements, VeriFree matches and even surpasses verifier-based methods on extensive evaluations across MMLU-Pro, GPQA, SuperGPQA, and math-related benchmarks. Moreover, we provide insights into this method from multiple perspectives: as an elegant integration of training both the policy and implicit verifier in a unified model, and as a variational optimization approach. Code is available at https://github.com/sail-sg/VeriFree.

1 Introduction

R1-Zero-style reinforcement learning has improved mathematical reasoning but remains difficult to extend beyond domains with rule-based answer verification. VeriFree addresses this gap by using reference-answer likelihoods instead of explicit verifiers, while retaining strong practical and empirical performance.

  • Rule-based reinforcement learning has produced strong improvements in mathematical reasoning, but applies mainly where answer verification is feasible.Source: §1 · p. 2
  • Model-based verifiers extend the paradigm beyond rule-verifiable tasks but require a strong verifier, risk reward hacking, and add training-time memory and computation costs.Source: §1 · p. 2
  • VeriFree generates a reasoning trace, appends the dataset reference answer, and uses the reference-answer likelihood as both a policy-gradient reward and supervised-learning weight.Source: §1 · p. 2
  • Under a unique-answer setting, VeriFree is expectation-equivalent to RL with verifiable rewards while providing lower-variance estimation; with multiple valid answers, one reference still supplies a sufficient learning signal empirically.Source: §1 · p. 2
  • Across diverse reasoning tasks, VeriFree matches or often outperforms verifier-based alternatives while being simpler, faster, less memory-intensive, and more robust.Source: §1 · p. 2

2 Methodology

The methodology replaces answer-verifier rewards with reference-answer likelihoods while preserving the verifier-based objective under a unique-answer assumption. Its derivation marginalizes over sampled answers, reducing gradient variance, and its implementation addresses tokenization and reasoning-trace extraction challenges.

Verifier-based reinforcement learning

  • R1-Zero-style RL parses each response into reasoning trace z and answer y, then uses a verifier to assign a correctness reward against reference answer y⋆.Source: §2.1 · pp. 2–3
  • Because correctness verification is difficult in general reasoning tasks, the paper proposes a verifier-free extension of the training paradigm.Source: §2.1 · p. 3

VeriFree policy optimization

  • Under a single exact-match correct answer, VeriFree derives an equivalent objective by using the probability of y⋆ conditioned on x and sampled reasoning trace z.Source: §2.2 · pp. 3–4
  • VeriFree’s objective and gradient estimator are expectation-equivalent to verifier-based counterparts, combining a reasoning policy-gradient term with a reward-weighted reference-answer term.Source: §2.2 · p. 4
  • Analytically marginalizing the sampled answer removes one source of Monte Carlo randomness, yielding lower variance through Rao-Blackwellization.Source: §2.2 · p. 4

Variance reduction and implementation

  • The estimator supports RLOO, GRPO normalization, and PPO clipping, with multiple responses sampled per prompt and corrected response-length normalization.Source: §2.2 · p. 4

Comparison to existing approaches

  • Unlike JLB and LaTRO, VeriFree weights the reference-answer term by answer likelihood, down-weighting low-quality reasoning traces instead of reinforcing mismatched answers.Source: §2.3 · p. 5

Variance reduction and implementation

  • Tokenization-aware splitting ends z at “<answer” rather than “<answer>”, maintaining token-space alignment and avoiding optimization instability from inconsistent boundaries.Source: §2.4 · p. 5

3 Experiments

The experiments evaluate VeriFree across general reasoning benchmarks, training settings, and ablations. VeriFree improves accuracy, learning efficiency, confidence alignment, and cross-domain transfer while exposing sensitivity to tokenization and answer-equivalence handling.

Setup

  • VeriFree is evaluated using Qwen3 base models at 1.7B, 4B, and 8B parameters, with direct RL fine-tuning that skips supervised fine-tuning.The training pipeline uses Oat and the paper’s specified prompt template.Source: §3.1 · p. 5
  • The evaluation covers MMLU-Pro, SuperGPQA, and GPQA, spanning general and graduate-level reasoning across diverse disciplines.MMLU-Pro evaluates multiple domains, SuperGPQA spans 285 disciplines, and GPQA targets graduate-level questions resistant to shallow matching.Source: §3.1 · p. 6

Main Results

  • 12% to 40% average-accuracy gains are obtained over base models on MMLU-Pro after applying VeriFree.The reported gains are measured across model scales and domains in Table 1.Source: §3.2 · p. 7
  • VeriFree reaches performance comparable to or higher than the instruct model in thinking mode and the Verifier baseline on SuperGPQA without explicit verification signals.The comparison is reported across base-model evaluations on SuperGPQA.Source: §3.2 · p. 7
  • VeriFree improves learning efficiency by achieving higher accuracy with fewer training steps than the model-based-verifier baseline.The paper attributes this to continuous reward signals and the RLOO objective, which reduce gradient variance and stabilize policy gradients.Source: §3.2 · p. 7
  • MMLU-Pro accuracy correlates positively with average confidence πθ(y⋆|x, z), with ρ = 0.82 during Qwen3-8B training.The authors use this self-estimated confidence as a proxy for emergent reasoning capability.Source: §3.2 · p. 8
  • Training without math examples still improves general benchmarks and transfers to math-specific tasks.This experiment evaluates whether reasoning learned from non-math data extends across domains.Source: §3.2 · p. 8

Ablation Study

  • Removing RLOO lowers final accuracy by more than 3% and causes premature convergence during MMLU-Pro training.The ablation compares Qwen3-1.7B models with and without RLOO.Source: §3.3 · p. 8

4 Related Works

Prior R1-Zero-style work applies RL with verifiable rewards primarily to mathematics and coding. Research on broader reasoning domains instead uses self-improvement signals or additional LLMs as proxy verifiers or reward models.

DeepSeek-R1-Zero-like reinforcement learning

  • R1-Zero-style training has reproduced strong reasoning improvements at smaller scales but remains concentrated in mathematics and coding, where rewards are readily verifiable.The related work describes rule-based verification as the common basis for these settings.Source: §4 · p. 9

Reasoning beyond verifiable domains

  • General-reasoning approaches commonly use a second LLM as a proxy verifier or reward model when verifiable rewards are unavailable.Examples include NaturalReasoning, Su et al., General-Reasoner, and X-Reasoner.Source: §4 · p. 9

Self-improving language models

  • Other self-improving methods derive training signals from model-generated judgments, implicit DPO rewards, or majority voting.These approaches use the model’s own outputs for ranking, preference learning, or self-labeled RL rewards.Source: §4 · p. 9

5 Conclusions

The paper introduces VeriFree, a verifier-free approach that rethinks RLVR and eliminates explicit verification. It is designed for general reasoning tasks where rule-based verifiers are infeasible and model-based verifiers are costly or vulnerable to reward hacking.

  • VeriFree rethinks RLVR from a new perspective by deriving an optimization objective that eliminates explicit verification.The derivation leverages gradient equivalence under the unique-answer assumption.Source: §5 · p. 9
  • VeriFree is suited to general reasoning tasks where rule-based verifiers are infeasible and model-based verifiers are expensive and vulnerable to reward hacking.The authors report extensive experiments and ablations across general reasoning benchmarks.Source: §5 · p. 10

A.1 Derivation of Gradient Estimators

This appendix derives the gradient estimators for VeriFree and the standard verifier-based objective. It presents the derivations as complementary formulations for comparison.

  • It also derives the corresponding standard verifier-based gradient estimator for completeness.Source: §A.1 · p. 14
  • The appendix derives the gradient estimator for VeriFree.Source: §A.1 · p. 15

A.2 Proof of Lower Variance

The proof establishes that VeriFree has policy-gradient estimator variance no greater than the verifier-based estimator. The argument uses conditional expectation and the law of total variance, and the method also supports PPO-style clipping for reused data.

  • VeriFree’s policy-gradient estimator has variance less than or equal to the verifier-based estimator for any given input and reference answer.The same relationship holds for global objectives averaged over the data distribution.Source: §A.2 · pp. 15–16
  • The lower-variance result follows because the VeriFree estimator is the conditional expectation of the verifier-based estimator over sampled answers.The conditioning fixes the question, reference answer, and reasoning trace while averaging over the answer sampling.Source: §A.2 · pp. 15–16
  • The proof applies the law of total variance, whose non-negative conditional-variance term yields the lower-variance relationship.Source: §A.2 · pp. 15–16
  • VeriFree remains compatible with PPO-style gradient clipping when data is reused to improve sample efficiency.The off-policy estimator uses the sampling policy and PPO clipping operation.Source: B Off-policy Gradient Estimators · p. 16

C Dataset Details

The dataset details section visualizes the category distribution of WebData, the training dataset, in Figure 7.

D Extended Empirical Results

The paper reports benchmark evaluations across broad-domain reasoning and mathematics, using domain breakdowns for MMLU-Pro and SuperGPQA and accuracy comparisons for math benchmarks and GPQA.

E Case Study

The case studies illustrate VeriFree-generated reasoning on astronomy, psychology, and hazardous-materials questions, including answers that match the provided references in the examples.

  • For the skull-and-crossbones question, the model identifies Class 6 hazardous materials, specifically Division 6.1 for toxic substances.Source: E Case Study · p. 18
  • For an astronomy question, the model identifies Cepheid variables as the objects used with the period-luminosity relation to determine distances.The example explains that Cepheid luminosity variation periods relate to intrinsic luminosity and supports distance estimation.Source: E Case Study · p. 19
  • The astronomy example describes measuring a Cepheid’s period, determining intrinsic luminosity, comparing apparent luminosity, and calculating distance.Source: E Case Study · p. 19
  • For the Kohlberg question, the model selects Stage 2’s personal-needs explanation because this stage involves exchange and self-interest.Source: E Case Study · pp. 20–21
  • The broader-impact discussion cautions that improved reasoning can amplify beneficial or harmful behavior depending on user intent and oversight.Source: F Broader Impact · p. 21
  • The paper encourages safeguards for misuse prevention and performance guarantees under distributional shifts as reasoning models enter real-world workflows.Source: F Broader Impact · p. 21

Related papers

Loading 2505.21493v1…