PaperlayerView exact revision on arXiv

Source-linked AI summary

NOVER: Incentive Training for Language Models via Verifier-Free Reinforcement Learning

Wei Liu, Siya Qi, Xinyu Wang, Chen Qian, Yali Du, Yulan He

arXiv:2505.16022v2cs.CLcs.AIcs.LG

TL;DR

Existing incentive training depends on external verifiers, limiting its use on difficult-to-grade text-to-text tasks. NOVER replaces those verifiers with a model-based perplexity reward using standard SFT data, enabling broad incentive training and reported advantages including stronger performance and inverse incentive training.

  • Problem

    External verifiers are hard to construct for contextual, free-form tasks, while verifier models require costly training and substantial computational resources.

    Source: §1 · p. 1

  • Method

    NOVER performs verifier-free incentive training using standard SFT data and a model-based perplexity reward derived from the model’s reasoning process.

    Source: §1 · p. 2, §4.1 · p. 3

  • Results

    NOVER demonstrates strong performance, stable training, reduced reward hacking, shaped reasoning patterns, and adaptation to diverse tasks.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    NOVER extends incentive learning to arbitrary text-to-text tasks and enables inverse incentive training as a new optimization paradigm.

    Source: §1 · p. 2, §7 · p. 9

    NOVER’s effectiveness depends on base models possessing capabilities such as partial CoT reasoning and instruction-following, with free-form reasoning requiring stronger capabilities than structured-answer tasks.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Recent advances such as DeepSeek R1-Zero highlight the effectiveness of incentive training, a reinforcement learning paradigm that computes rewards solely based on the final answer part of a language model's output, thereby encouraging the generation of intermediate reasoning steps. However, these methods fundamentally rely on external verifiers, which limits their applicability to domains like mathematics and coding where such verifiers are readily available. Although reward models can serve as verifiers, they require high-quality annotated data and are costly to train. In this work, we propose NOVER, NO-VERifier Reinforcement Learning, a general reinforcement learning framework that requires only standard supervised fine-tuning data with no need for an external verifier. NOVER enables incentive training across a wide range of text-to-text tasks and outperforms the model of the same size distilled from large reasoning models such as DeepSeek R1 671B by 7.7 percent. Moreover, the flexibility of NOVER enables new possibilities for optimizing large language models, such as inverse incentive training.

1 Introduction

Incentive training rewards only final answers to elicit intermediate reasoning, but external verifiers limit it on open-ended tasks. NOVER removes that dependency using model-based perplexity rewards and extends incentive training to broad text-to-text settings.

1 Introduction

  • Incentive training computes rewards only on final answers, encouraging models to generate intermediate reasoning steps autonomously.Source: §1 · p. 1
  • External verifiers are difficult to construct for contextual tasks such as social behavior analysis and creative writing, while verifier models require costly training and substantial computation.Source: §1 · p. 1
  • NOVER supports open-ended questions such as “Discuss”, “Why”, and “Create”, whose free-form answers are difficult to verify with clear-cut labels.Source: Figure 2 · p. 2, §1 · p. 2
  • NOVER is a verifier-free incentive-training framework that uses standard SFT data and a model-built reward proxy for lightweight reinforcement learning.Its perplexity-based reward is computed from the model’s reasoning process, avoiding an external verifier.Source: §1 · p. 2
  • NOVER’s reported benefits include stable training, reduced reward hacking, shaped reasoning patterns, and adaptation to diverse tasks.Source: §1 · p. 2
  • Inverse incentive training uses NOVER’s flexibility to teach models how to fish rather than simply giving them a fish, surpassing standard incentive training on creativity-demanding tasks.Source: §1 · p. 2

2 Related Work

Related work develops language-model reasoning through prompting and incentive training. Recent incentive-training studies investigate optimization practices and extensions beyond mathematics and coding.

Language Model Reasoning

  • Prompting methods evolved from Chain-of-Thought and Zero-Shot CoT to structured workflows including Tree of Thoughts, Least-to-Most, and self-reflection.Source: §2 · p. 2

Incentive Training

  • DeepSeek-R1 introduced incentive training as a simpler alternative to earlier reasoning-prompting methods.Source: §2 · p. 2
  • Logic-RL, DAPO, SimpleRL, OpenReasoner, Dr.GRPO, and VAPO studied exploration-exploitation, “aha” moments, and task-difficulty variance.Source: §2 · p. 3
  • Several studies attempted to extend incentive training beyond mathematics and coding.Source: §2 · p. 3

3 Background

Rule-based incentive training assigns sparse outcome rewards to final answers and thereby encourages intermediate reasoning, while GRPO uses grouped completions to optimize the policy. Format rewards help ensure outputs are parseable.

Rule-based Reward

  • A rule-based verifier computes the outcome reward solely from the final answer, while intermediate tokens represent reasoning steps generated by the policy.These steps may include task decomposition or self-reflection.Source: §3 · p. 3
  • Sparse but accurate outcome rewards encourage models to generate intermediate tokens that can lead to better final answers.Source: §3 · p. 3
  • A regular-expression format check requires reasoning and answer spans to use designated tags, making outputs easier for the verifier to parse.Source: §3 · p. 3
  • The combined reward adds rule-based and format rewards with task-specific weights.Source: §3 · p. 3

GRPO

  • GRPO samples a group of completions for each prompt, normalizes their rewards into relative advantages, and updates the policy with a clipped objective.The objective includes a clip ratio and KL penalty weight.Source: §3 · p. 3

4 Method

NOVER replaces external verification with reasoning perplexity computed on standard supervised fine-tuning data, using the policy model as a synchronized proxy. Its reward design combines ranked reasoning quality, efficiency, and tag-format compliance for GRPO training.

4.1 Reasoning Perplexity as Proxy

  • NOVER uses reasoning perplexity of the ground-truth answer, conditioned on a model’s reasoning trajectory, as a verifier-free reward proxy.The proxy is computed with teacher forcing and requires only about 5% of total training time.Source: §4.1 · pp. 3–4, §4 · p. 3

4.2 Policy-Proxy Synchronization

  • The policy model itself serves as the proxy, while synchronization lets the proxy adapt gradually to policy improvements through shared LoRA adapters.The LoRA adapter comprises about 0.4% of the full model parameters.Source: §4.2 · p. 4

4.3 Verifier-Free Reward

  • Reasoning perplexity is discretized into rank-based rewards to reduce variance and stabilize training.The best completion alone can receive reward 1.0 for objectively correct tasks, whereas normalized ranks suit subjective or open-ended tasks.Source: §4.3 · p. 4
  • An efficiency reward favors completions that achieve lower reasoning perplexity with fewer reasoning tokens.This targets effective reasoning length rather than rewarding longer trajectories by default.Source: §4.3 · p. 4
  • NOVER conditionally combines tag-format, reasoning, and efficiency rewards for GRPO, calculating the latter two only when the required format is correct.The reward weights control the relative importance of the components, and GRPO is used for efficient training.Source: §4.3 · pp. 4–5

5 Experimental Setup

NOVER is evaluated with 3B and 7B Qwen 2.5 models across general reasoning and creative-writing tasks, against vanilla, chain-of-thought, and supervised fine-tuning baselines. The setup also analyzes task distributions and uses Figure 4 to examine performance across reasoning-task characteristics.

Dataset

  • The evaluation spans general reasoning and creative writing, including datasets requiring factually grounded, multi-step answers and long-form narrative planning.The general-reasoning category includes Natural Reasoning, General Thought, and WebInstruct.Source: §5 · p. 5

Baselines

  • Experiments compare 3B and 7B Qwen 2.5 models with vanilla, chain-of-thought, and matched supervised fine-tuning baselines.The matched SFT baseline uses the same pretrained checkpoints and each dataset’s training split.Source: §5 · p. 5

Training and Evaluation

  • Training uses Hugging Face TRL with LoRA adapters, and a Qwen2.5-7B run for 1,000 steps takes about 2 hours on two H100 GPUs.Training time depends mainly on rollout efficiency, maximum generation length, and task.Source: §5 · p. 5
  • Figure 4 analyzes NOVER accuracy across three general reasoning tasks by question format and other distributional aspects.The supplied caption identifies the figure’s scope, while the setup frames distribution analysis as part of understanding when NOVER performs well.Source: Figure 4 · p. 6

6 Results and Analysis

NOVER consistently outperforms baselines across model sizes, backends, and task types while enabling verifier-free incentive training. Analyses show how task structure, reward design, and inverse incentive training shape its effectiveness and limitations.

6.1 Overall Performance

  • NOVER consistently outperforms all baselines for 3B and 7B models, with the 3B model approaching the much larger R1-Distill-Qwen-7B on General Thoughts.NOVER reaches nearly 60% accuracy on General Thoughts and improves especially on EmoBench and OPUS.Source: §6.1 · p. 6, Table 2 · p. 6
  • NOVER gains transfer across Mistral and LLaMA backends and both pretrained and instruct-tuned checkpoints.The results show consistent gains across different model families and checkpoint types.Source: §6.1 · p. 6

6.2 When and Where for Effective Reasoning

  • Accuracy is higher for multiple-choice and clearly directed questions, while flexible prompts such as “analyze” are weaker.Candidate options reduce the search space during reinforcement learning, lowering task difficulty.Source: §6.2 · p. 7
  • NOVER maintains stable accuracy for 60–240 reasoning tokens and adaptively adjusts reasoning length to question difficulty.Shorter questions, references, and answers generally improve accuracy, whereas reasoning length is comparatively stable in this range.Source: §6.2 · p. 7
  • On false-premise FANToM questions, NOVER underperforms SFT because it relies on pretrained responses and lacks learned refusal behavior.SFT can memorize refusal patterns, while NOVER selectively reinforces candidate responses generated by the pretrained model.Source: §6.2 · p. 7, Table 3 · p. 7

6.3 Curse of Proxy

  • Full NOVER stabilizes proxy-based training, whereas incomplete reward designs can cause reasoning explosion, collapse, or inefficient recovery.Proxy-policy synchronization reduces bias, and the efficiency reward helps invalid completions receive zero reward and be resampled.Source: §6.3 · pp. 7–8, Figure 5 · p. 7

6.4 Verifier, Free, or Not

  • Model-based verifiers are unstable: lenient judges encourage reward hacking, strict judges produce sparse rewards, and dedicated verifiers can be misled by policy outputs.These findings motivate NOVER’s verifier-free design for difficult-to-verify reasoning tasks.Source: §6.4 · p. 8, Table 4 · p. 8

6.5 Reasoning Patterns

  • Training shifts reasoning from task decomposition toward direct reasoning, including skipping redundant steps when intermediate conclusions are available.The shift is associated with the efficiency reward during NOVER training.Source: §6.5 · p. 8, Figure 6 · p. 8

7 Conclusion

NOVER is a verifier-free incentive-training framework that extends incentive learning to any text-to-text task. Its flexibility also enables novel paradigms such as inverse incentive training.

  • NOVER is a verifier-free framework for incentive training across any text-to-text task.Source: §7 · p. 9
  • NOVER’s flexibility enables novel learning paradigms such as inverse incentive training.Source: §7 · p. 9

Limitations

NOVER’s effectiveness depends on base models possessing capabilities such as partial chain-of-thought reasoning and instruction-following. The authors therefore conduct experiments on Qwen models, which meet these requirements among open-source models considered.

  • Incentive training requires base models with capabilities such as partial CoT reasoning and instruction-following.Source: Limitations · p. 9
  • General free-form reasoning tasks demand stronger base-model capabilities than structured-answer tasks such as mathematics or coding.Source: Limitations · p. 9
  • The experiments use Qwen because it is described as a suitable open-source model meeting these requirements.Source: Limitations · p. 9

A Dataset Construction Pipeline & Statistics

The dataset pipeline assembles recent, diverse reasoning and text-generation datasets, filters and reformats examples, and creates distributionally consistent splits with capped sample sizes for very large datasets.

B Profiling

NOVER profiling shows that rollout dominates training time, while reward computation is comparatively small. The experiments use task-specific completion limits, early stopping, minimal prompts, and several evaluation-judge prompts; examples illustrate improved reasoning quality over the base model.

Profiling

Training settings

  • General reasoning and SS-GEN use 2,048-token completion limits, while other tasks use 512 tokens; training runs up to 5,000 steps with validation-based early stopping.Source: C Hyperparameters · pp. 14–15

Prompts

Judgment prompts

Reasoning cases

  • In a translation example, the base model produces confident but incorrect self-verification, whereas NOVER identifies and translates omitted descriptive phrases such as the woman’s appearance.Source: F Reasoning Hallucination Case · p. 16
  • A medical example shows that lower reasoning perplexity is associated with superior physiological mechanisms, terminology, dietary-factor discussion, and overall scientific reasoning.Source: G Reasoning Perplexity Example · p. 16

H False Premise Example

NOVER can underperform SFT on false-premise questions because it may not recognize that a question is unanswerable. The example illustrates how reliance on the base model’s initial answer constrains reasoning in counterintuitive cases.

False Premise Example

  • NOVER remains constrained by the flawed assumption that a false-premise question is answerable, so it reasons toward an incorrect response.In the FANToM example, the model fails to recognize that one dialogue participant is unaware of the topic and therefore cannot answer the question.Source: H False Premise Example · p. 19
  • SFT can outperform NOVER on counterintuitive false-premise tasks by memorizing response patterns such as stating that someone is unaware.NOVER instead reinforces the base model’s generated answer, which may omit the appropriate unanswerability pattern.Source: H False Premise Example · p. 19
  • The false-premise example involves higher-order theory-of-mind reasoning in a multi-party dialogue, where the correct response reflects a participant’s lack of awareness.The task requires inferring what one person would think from another person’s perspective while recognizing that the premise may be invalid.Source: H False Premise Example · p. 19
  • The paper suggests a hybrid approach that first uses SFT to expose appropriate response patterns before applying NOVER-style reinforcement.This proposed combination targets cases where the base model does not spontaneously detect that a question is unanswerable.Source: H False Premise Example · p. 19

Related papers

Loading 2505.16022v2…