PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Reason without External Rewards

Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, Dawn Song

arXiv:2505.19590v5cs.LGcs.CL

TL;DR

RLVR supports complex reasoning but relies on costly, domain-specific supervision and verification. The paper introduces RLIF and INTUITOR, which optimize a model’s self-certainty through GRPO instead of external rewards. INTUITOR matches GRPO on mathematical reasoning and reports stronger generalization to code and other out-of-domain tasks, while remaining limited by small-scale experiments and offline-training degradation.

  • Problem

    RLVR improves reasoning but requires domain-specific verifiers, gold solutions, or test infrastructure, limiting use where external verification is unavailable.

  • Method

    INTUITOR is an RLIF method that replaces GRPO’s external reward with the model’s self-certainty as its sole intrinsic reward.

  • Results

    INTUITOR matches GRPO on mathematical reasoning and achieves competitive, sometimes better, generalization to code generation and instruction following without gold-standard solutions.

  • Takeaways & Limitations

    Intrinsic self-certainty can support effective learning across domains without external supervision or verifiable rewards.

  • Takeaways & Limitations

    Experiments use relatively compact models and small unsupervised corpora, while purely offline INTUITOR training caused performance degradation over time.

Abstract

from arXiv · show

Training large language models (LLMs) for complex reasoning via Reinforcement Learning with Verifiable Rewards (RLVR) is effective but limited by reliance on costly, domain-specific supervision. We explore Reinforcement Learning from Internal Feedback (RLIF), a framework that enables LLMs to learn from intrinsic signals without external rewards or labeled data. We propose Intuitor, an RLIF method that uses a model's own confidence-termed self-certainty-as its sole reward signal. Intuitor replaces external rewards in Group Relative Policy Optimization (GRPO) with self-certainty scores, enabling fully unsupervised learning. Experiments demonstrate that Intuitor matches GRPO's performance on mathematical benchmarks while achieving better generalization to out-of-domain tasks like code generation, without requiring gold solutions or test cases. Our findings show that intrinsic model signals can drive effective learning across domains, offering a scalable alternative to RLVR for autonomous AI systems where verifiable rewards are unavailable. Code is available at https://github.com/sunblaze-ucb/Intuitor

1 INTRODUCTION

RLVR improves reasoning but depends on costly, domain-specific verification, motivating RLIF, which trains models from intrinsic feedback without external supervision. INTUITOR implements this idea using self-certainty as its reward and reports competitive in-domain performance with stronger out-of-domain generalization.

  • Motivation: RLHF requires extensive human annotation, while RLVR requires domain-specific verifiers and gold-standard solutions or test infrastructure.These requirements limit application to carefully curated domains and complicate open-ended deployment.
  • RLIF: RLIF enables language models to improve reasoning by optimizing intrinsic, self-generated feedback without external rewards or labeled data.The paradigm is motivated partly by settings where humans may be unable to evaluate increasingly capable models directly.
  • INTUITOR: INTUITOR uses a model’s own confidence, termed self-certainty, as the sole intrinsic reward within an RLIF framework.It replaces the verifiable reward in GRPO with self-certainty scores while retaining the same policy-gradient algorithm.
  • Results: On MATH, INTUITOR matches GRPO without gold answers and generalizes to code tasks more effectively than GRPO.For Qwen2.5-3B, it yields 65% relative improvement on LiveCodeBench and 76% on CRUXEval-O, versus 0% and 44% for GRPO, respectively.
  • Results: INTUITOR matches supervised reinforcement learning on in-domain tasks and achieves competitive, sometimes better, out-of-domain generalization.The reported contributions also include emergent structured reasoning and enhanced instruction following.

2 RELATED WORK

Prior work progresses from human-preference rewards to verifiable rewards and then intrinsic-feedback approaches. These methods differ in their reliance on annotation, domain-specific correctness checks, and model-generated signals.

  • RLHF: RLHF aligns model outputs with human values using reward models trained on human preference data, but extensive annotation makes it resource-intensive.The related-work discussion presents RLHF as an early influential approach to refining language models.
  • RLVR: RLVR uses rule-based verification, such as exact answer matching, for tasks with clear correctness criteria including mathematics and code generation.This avoids the complexities of learned reward models but remains tied to verifiable task outcomes.
  • Intrinsic signals: Intrinsic-reward and self-play methods use models themselves to provide feedback or evaluate outputs during optimization.Examples include SPIN, Self-Rewarding LMs, STaR, and concurrent methods exploring intrinsic signals and procedural generalization.

3 METHOD

The method replaces external verification with intrinsic feedback and instantiates RLIF through GRPO optimization of self-certainty. It samples output groups, computes relative intrinsic rewards, and updates the policy toward higher-confidence generations.

  • RLIF: RLIF uses the model’s own assessment of generated outputs as task-agnostic feedback instead of external evaluation.The objective defines an intrinsic signal u(q, o) derived from the model’s internal state or computation.
  • INTUITOR: INTUITOR replaces external rewards with the model’s confidence as the sole intrinsic reward signal.The approach is motivated by observations that confidence is often lower on unfamiliar or difficult tasks and higher confidence frequently correlates with correctness.
  • Self-certainty: Self-certainty is the average KL divergence between a uniform vocabulary distribution and the model’s next-token distribution.Higher self-certainty values indicate greater confidence, and the metric is reported as less biased toward longer generations than perplexity or entropy measures.
  • GRPO: GRPO samples multiple candidate outputs for each query and uses their relative rewards to estimate advantages for policy updates.Importance-ratio clipping and a KL penalty control update stability and deviation from the reference policy.
  • Training pipeline: INTUITOR computes self-certainty for each candidate, estimates group-relative advantages, and updates the policy to favor high-confidence outputs without external supervision.This creates a self-reinforcing training loop based on sampled candidate groups.

4 EXPERIMENTAL SETUP

The experiments train GRPO and INTUITOR on MATH using Qwen2.5 backbones and evaluate mathematical, code, and instruction-following benchmarks. Code-generation training uses a separate INTUITOR-Code setup with altered sampling and optimization hyperparameters.

  • Training setup: Both methods are trained on the 7,500-problem MATH training split using Qwen2.5-1.5B and Qwen2.5-3B backbones.Training uses the Open-R1 framework and chat-based prompting throughout.
  • Training setup: The training setup processes 128 problems per update and does not require models to separate intermediate reasoning from final answers.This accommodates the backbones’ initially weak instruction-following abilities.
  • Code generation: INTUITOR-Code applies INTUITOR to Codeforces code generation with 14 sampled completions per problem, a 1 × 10^-5 learning rate, and β = 0.01.The run is limited to 50 steps and uses 3,200 training examples.
  • Evaluation: Evaluation covers mathematical reasoning, code reasoning, and instruction-following benchmarks using greedy decoding.MMLU-Pro uses its original prompt format, while other evaluations generally retain chat-style prompting.

5 RESULTS AND ANALYSIS

The results compare INTUITOR with supervised and alternative reward methods across performance, instruction following, reasoning behavior, generalization, and reward robustness. INTUITOR matches GRPO in-domain while showing faster learning, emergent structured reasoning, and strong transfer to code-related tasks.

  • Overall Performance: INTUITOR and GRPO-PV achieve performance comparable to GRPO on in-domain MATH and GSM8K without golden answers.GRPO uses golden answers, whereas INTUITOR and GRPO-PV are golden-answer-free.
  • Learning to Follow Instructions: INTUITOR significantly enhances instruction-following by reducing gibberish, decreasing completion lengths, and improving Length Control Win Rate on AlpacaEval beyond GRPO under identical settings.These gains occur for both Qwen2.5-1.5B and Qwen2.5-3B models trained on MATH.
  • Learning to Follow Instructions: After 10 training steps, INTUITOR consistently achieves higher GSM8K and MATH accuracy than GRPO across both Qwen2.5-1.5B and Qwen2.5-3B.The GRPO baseline uses ground-truth rewards.
  • Cross-Task Generalization: LiveCodeBench gains emerge after initial MATH500 improvements and continue after MATH500 plateaus, indicating later cross-task generalization from MATH training.This trajectory is observed for both INTUITOR and GRPO.
  • Fostering Structured Reasoning: INTUITOR-trained models develop prereasoning before JSON or code outputs, with free-form or natural-language reasoning emerging despite prompts requiring structured answers.The paper suggests this emergent prereasoning may contribute to performance on CRUXEval-O and LiveCodeBench.
  • Understanding Emergent Long-Form Reasoning: On LiveCodeBench, models first improve valid-code generation and accuracy, then add explicit reasoning and progressively more elaborate self-explanatory traces.Figure 6 analyzes ten randomly selected questions across successive training checkpoints.
  • Online Self-Certainty Prevents Reward Exploitation: Offline self-certainty reward exploitation causes response length to spike while validation accuracy collapses, whereas the online annotator prevents this exploitation.The policy inflates offline reward by appending an already-solved auxiliary problem.
  • Online Self-Certainty Prevents Reward Exploitation: Both GRPO and INTUITOR assign significantly higher average self-certainty scores to correct than incorrect MATH500 responses.Mann–Whitney U tests are used to assess the separation between correct and incorrect responses.

6 DISCUSSION AND FUTURE RESEARCH

The paper identifies open questions around scaling, theoretical foundations, and combining internal and external reward signals. Offline training can degrade over time, while the optimal RLIF reward and benefits of combined signals remain unresolved.

  • Scalability and Generalization: Offline INTUITOR training caused performance degradation over time, motivating periodic online self-certainty updates or hybrid offline-online schedules.These approaches are proposed to maintain calibration when scaling the method.
  • Scalability and Generalization: Experiments used relatively compact models and small unsupervised corpora because of computational constraints, leaving larger models and diverse real-world datasets for future work.The authors specifically identify foundation models with hundreds of billions of parameters as a scaling direction.
  • Theoretical Analysis of RLIF: The theoretically optimal RLIF reward signal and the fundamental reasoning boundaries of LLMs remain open problems requiring further theoretical research.The paper positions existing sharpening analyses as related but incomplete theoretical foundations for RLIF.
  • Combining Reward Signals: The paper evaluates a single reward signal to compare self-certainty with golden-answer rewards, leaving combinations with external and formatting rewards for future work.Potential combinations include summation, alternating by answer availability, or integration with RLHF and RLVR.

7 CONCLUSION

The paper introduces INTUITOR, which uses intrinsic self-certainty as its sole reward to eliminate external supervision and gold solutions. It matches supervised RLVR on mathematical reasoning and shows competitive or better out-of-domain generalization, supporting RLIF as a path toward autonomous self-improvement.

  • 7 CONCLUSION: INTUITOR uses a model’s intrinsic self-certainty as its sole reward signal, eliminating external supervision and gold-standard solutions.It is presented as an instantiation of Reinforcement Learning from Internal Feedback.
  • 7 CONCLUSION: INTUITOR matches supervised RLVR methods like GRPO on mathematical reasoning and achieves competitive, sometimes better generalization to code generation and instruction following.The conclusion also reports structured reasoning and online self-certainty as safeguards against reward exploitation.
  • 7 CONCLUSION: The findings support AI systems that improve through introspection and may acquire skills autonomously in novel domains.The authors frame scalable self-improvement as a future direction as systems approach or surpass human oversight limits.

ETHICS STATEMENT

The research uses publicly available datasets and open-source language models, addressing concerns related to private data and human subjects. Its stated goal is self-supervised improvement of language-model reasoning toward more transparent and robust AI systems.

  • ETHICS STATEMENT: The study uses publicly available datasets and open-source language models, mitigating concerns related to private data or human subjects.The authors report no direct negative societal impacts or ethical concerns from the work.
  • ETHICS STATEMENT: The stated goal is to enhance language-model reasoning through self-supervision as a step toward more transparent and robust AI systems.The authors also make their code publicly available for scrutiny of the methods and findings.

REPRODUCIBILITY STATEMENT

The paper provides source code, training configurations, prompts, and evaluation details to support complete replication and validation of its experiments.

  • REPRODUCIBILITY STATEMENT: Source code and training configurations are provided at the Intuitor repository to support reproducibility.The resources include hyperparameters, software versions, and evaluation setups.
  • REPRODUCIBILITY STATEMENT: Appendix C.1 provides the exact prompts used during training and evaluation.The paper states that these materials enable complete replication and validation of the findings.

LLM USAGE STATEMENT

The paper reviews external-reward reinforcement-learning paradigms and their limitations, then presents evidence that INTUITOR can train across model families and settings.

  • FROM EXTERNAL SUPERVISION TO INTERNAL FEEDBACK: RLHF optimizes models using learned human-preference rewards, while RLVR uses automatically verifiable, task-specific signals.RLHF introduces reward-model fragility; RLVR relies on gold answers or test execution.
  • FROM EXTERNAL SUPERVISION TO INTERNAL FEEDBACK: RLVR still depends on costly, domain-specific supervision and struggles with ambiguous or subjective tasks.Mathematics requires gold-standard answers, while code generation requires test executions.
  • KL-PENALTY ANALYSIS: INTUITOR’s KL penalty is a critical regularizer because its out-of-domain accuracy is highly sensitive to this hyperparameter.The penalty limits policy drift from the initial model distribution and safeguards against degeneration.
  • GENERALIZATION ACROSS MODEL FAMILIES: INTUITOR improves Llama3.2-3B-Instruct throughout training, with steady gains in both accuracy and response length.The experiment compares INTUITOR with GRPO on the MATH dataset.
  • GENERALIZATION ACROSS MODEL FAMILIES: INTUITOR provides consistent training improvements on OLMo-2-1124-7B-SFT, extending effectiveness beyond the Qwen model family.The model’s open-source training data and code also address evaluation-data-contamination concerns.

B.4 COMPARISON WITH ALTERNATIVE REWARD SIGNALS

Alternative intrinsic reward signals can destabilize reinforcement learning through length bias, repetition, or reward hacking, whereas INTUITOR provides more robust training behavior. Larger rollout groups and staged INTUITOR-to-GRPO training further improve performance, while naive reward mixing does not.

  • Entropy minimization: Negative entropy rewards can improve reasoning without labels but risk repetition and collapse because low entropy often correlates with repetitive loops.Entropy minimization caused severe degeneration, including producing the same character regardless of the prompt, while INTUITOR avoided collapse.
  • Random reward: Random rewards severely degrade chat-style reinforcement learning, showing that INTUITOR’s performance gains are non-trivial.The random baseline assigned each response a reward of 0 or 1 with equal probability.
  • Log-probability rewards: Raw log probability rapidly decreases completion length and reward through short-length bias, while normalized log probability encourages overly long completions and degraded performance.INTUITOR consistently improves accuracy on both models and demonstrates substantially stronger robustness.
  • Self-certainty optimization: Directly optimizing self-certainty initially raises accuracy but ultimately collapses, whereas INTUITOR uses self-certainty as a relative weighting factor for stable, superior training.The relative weighting formulation mitigates the reward-hacking incentive to inflate certainty without genuine task improvement.
  • Rollout size: Increasing rollout size improves GSM8K and MATH500 validation accuracy by reducing self-certainty advantage variance, improving training stability and generalization.The cited experiment compares a rollout size of 14 with previous settings.
  • Combining reward signals: A naive weighted combination of INTUITOR and GRPO performs worse than GRPO alone, whereas one INTUITOR epoch followed by one GRPO epoch outperforms two GRPO epochs.Prior INTUITOR training may establish more confident and coherent reasoning trajectories for subsequent GRPO reinforcement.
  • Cross-domain training: Codeforces pretraining followed by INTUITOR training on MATH improves GSM8K and MATH500 validation performance rather than hindering later-domain training.The result suggests that INTUITOR training on one domain does not impede subsequent training on another.
  • Completion uncertainty: INTUITOR and GRPO show similar completion-uncertainty trends, providing no clear evidence that INTUITOR further reduces response variance.Overall correctness standard deviation remains largely stable, while within-group standard deviation decreases under both procedures.

B.10 FAILURE CASE ANALYSIS

INTUITOR depends on sufficient pretrained or supervised domain knowledge and can fail when that prior knowledge is inadequate. Its weaker correctness signal also makes training sensitive to hyperparameter choices, particularly the learning rate.

  • Dependence on prior knowledge: INTUITOR assumes sufficient domain knowledge from pretraining or supervised fine-tuning; inadequate prior knowledge can cause immediate or delayed collapse.Llama3.2-3B-Base on MATH failed to raise its near-zero accuracy, likely because it was not aligned with the chat template.
  • Signal reliability: Self-certainty provides no guarantee of true correctness, making INTUITOR more sensitive to hyperparameter choices.The passage identifies this weaker learning signal as a direct reason for increased sensitivity.
  • Learning-rate sensitivity: A 3 × 10−6 learning rate degraded Qwen2.5-7B performance after a brief initial climb, whereas 1×10−6 stabilized training and prevented collapse.This example shows a concrete learning-rate boundary for the reported setup.

C PROMPTS AND MODEL COMPLETIONS

The appendix specifies model prompts and presents examples showing that INTUITOR-trained models develop more explicit reasoning and often outperform GRPO-tuned behavior on code-generation tasks. It also traces how reasoning and code quality evolve during training on MATH and code data.

  • C PROMPTS AND MODEL COMPLETIONS: The appendix uses Qwen2.5-3B and MATH as the default model and training-dataset settings.It also lists prompts for Qwen2.5-1.5B, 7B, and 14B.
  • C.1 TRAINING PROMPTS: Training prompts request step-by-step reasoning, with some MATH prompts requiring boxed final answers and immediate stopping after the box.The code prompt instead asks the model to generate Python for competitive programming problems.
  • C.2 EXAMPLE FROM LIVECODEBENCH CODE GENERATION: INTUITOR-trained models tend to reason before producing code, whereas GRPO-tuned models typically output Python directly under the same prompt.The example problem asks models to count passengers older than 60 from encoded strings.
  • C.3 EXAMPLE FROM LIVECODEBENCH CODE GENERATION: On a card-ordering task, INTUITOR usually interprets instructions and produces correct Python, while GRPO may misinterpret the instructions or decline to respond.The task permits at most one swap and requires YES or NO for each permutation.
  • C.4 EXAMPLE FROM CRUXEVAL-O: INTUITOR-trained models commonly explain their reasoning before returning the required formatted response, while GRPO-tuned models often output the JSON immediately.This comparison comes from a CRUXEval-O example.
  • C.5 ROLLOUT EXAMPLE OF INTUITOR DURING TRAINING ON CODE CORPUS: During INTUITOR training on CodeContest, responses shift from concise reasoning to detailed step-by-step explanations that reinforce interim outputs.The passage describes progressively richer explanations during reinforcement-learning rollouts.
  • C.6 EVOLUTION OF QWEN2.5-3B’S RESPONSES ON LIVECODEBENCH CODE GENERATION TRAINED WITH INTUITOR ON MATH: During INTUITOR training on MATH, code generation progresses from syntactically valid Python to clearer formatting and ultimately correct solutions.The examples concern LiveCodeBench code generation after training on MATH.
Loading 2505.19590v5…