PaperlayerView exact revision on arXiv

Source-linked AI summary

LaSeR: Reinforcement Learning with Last-Token Self-Rewarding

Wenkai Yang, Weijie Liu, Ruobing Xie, Yiju Guo, Lulu Wu, Saiyong Yang, Yankai Lin

arXiv:2510.14943v1cs.CLcs.AIcs.LG

TL;DR

Standard RLVR lacks verification signals when test-time ground-truth answers are unavailable. LaSeR aligns a last-token self-rewarding score with verifier-based rewards, improving reasoning and self-verification while requiring little extra inference computation.

  • Problem

    Standard RLVR cannot provide verification signals when ground-truth answers are unavailable at test time, motivating models that can assess their own outputs.

    Source: §1 · p. 1, §3.2.1 · p. 4

  • Method

    LaSeR adds an MSE loss aligning verifier-based reasoning rewards with a last-token score derived from the policy model’s next-token probability and a pre-calculated constant.

    Source: §1 · p. 2, §6 · p. 10

  • Results

    Around 80% self-verification F1; LaSeR improves reasoning and self-rewarding accuracy and outperforms equally sized state-of-the-art external verifiers on model-generated solutions.

    Source: §1 · p. 2, §4.2 · p. 8

  • Takeaways & Limitations

    The score can be computed without forwarding the solution through the reference model, making self-rewarding lightweight to calculate.

    Source: §3.3 · p. 5

    The discussed implicit reward is length-biased, and directly aligning it with verifier rewards can degrade the policy model’s generation ability.

    Source: §3.2.1 · p. 4

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as a core paradigm for enhancing the reasoning capabilities of Large Language Models (LLMs). To address the lack of verification signals at test time, prior studies incorporate the training of model's self-verification capability into the standard RLVR process, thereby unifying reasoning and verification capabilities within a single LLM. However, previous practice requires the LLM to sequentially generate solutions and self-verifications using two separate prompt templates, which significantly reduces efficiency. In this work, we theoretically reveal that the closed-form solution to the RL objective of self-verification can be reduced to a remarkably simple form: the true reasoning reward of a solution is equal to its last-token self-rewarding score, which is computed as the difference between the policy model's next-token log-probability assigned to any pre-specified token at the solution's last token and a pre-calculated constant, scaled by the KL coefficient. Based on this insight, we propose LaSeR (Reinforcement Learning with Last-Token Self-Rewarding), an algorithm that simply augments the original RLVR loss with a MSE loss that aligns the last-token self-rewarding scores with verifier-based reasoning rewards, jointly optimizing the reasoning and self-rewarding capabilities of LLMs. The optimized self-rewarding scores can be utilized in both training and testing to enhance model performance. Notably, our algorithm derives these scores from the predicted next-token probability distribution of the last token immediately after generation, incurring only the minimal extra cost of one additional token inference. Experiments show that our method not only improves the model's reasoning performance but also equips it with remarkable self-rewarding capability, thereby boosting its inference-time scaling performance.

1 Introduction

RLVR improves LLM reasoning, but standard verification is unavailable when test-time ground truth is missing, and existing verification approaches can be inefficient. LaSeR derives a self-rewarding score from the solution’s final token and trains it alongside RLVR, with experiments reporting improved reasoning and self-rewarding capability.

1 Introduction

  • Test-time verification is unavailable without ground-truth answers, while external verifiers require separate training and prior joint methods generate solutions and verifications separately.The latter doubles per-sample inference cost and reduces generation efficiency.Source: §1 · p. 1
  • LaSeR’s closed-form insight equates the verifier’s true reasoning reward with a last-token score based on the policy model’s special-token log-probability, a reference-model constant, and the KL coefficient.The reference-model log-probability for a randomly chosen special token is practically constant across problems and solutions, enabling a policy-only score with a pre-calculated constant.Source: §1 · pp. 1–2
  • LaSeR adds an MSE loss aligning last-token self-rewarding scores with verifier rewards to the standard RLVR loss, jointly optimizing reasoning and self-rewarding.The score is computed after solution generation, with at most one additional token inference and no extra generation.Source: §1 · p. 2
  • Experiments across LLaMA and Qwen architectures and pre-trained, mid-trained, and reinforced variants report improved reasoning, self-rewarding accuracy, confidence calibration, and inference-time scaling.The paper reports these results on broader math reasoning tasks.Source: §1 · p. 2

2 Related Work

Related work covers RLVR methods, external verifiers, and efforts to combine generation and verification in one model. LaSeR instead derives its self-rewarding signal from the generated solution’s final-token distribution.

2 Related Work

  • RLVR research spans algorithm efficiency [Schulman et al., 2017; Shao et al., 2024], broader reasoning and agent domains [Ma et al., 2025; Wang et al., 2025b], verifiable datasets, and mechanism analysis.The passages also list work on dataset collection and RLVR mechanisms (Hu et al., 2025; He et al., 2025; Mukherjee et al., 2025; Yue et al., 2025a).Source: §2 · pp. 2–3
  • External verifiers include ORMs that evaluate final answers, PRMs that assess intermediate reasoning, and generative verifiers that critique solutions before judging outcomes.The cited work reports stronger verification performance for generative verifiers than scalar reward models.Source: §2 · p. 3
  • Prior self-verification methods unify generator and verifier roles in one policy model but sequentially generate solutions and verifications; LaSeR derives its signal from the final-token distribution.The trained self-verification capability can be used during RL training and inference-time scaling.Source: §2 · p. 3

3 Methodology

LaSeR turns verifier-based correctness rewards into a last-token self-rewarding signal and combines that signal with RLVR to train reasoning and self-verification. Its practical techniques simplify score computation, stage training, and support use of the scores during inference.

3.1 Preliminaries

  • The preliminaries describe RLVR as using verifier rewards, often binary, and policy-gradient updates that weight token log-probabilities by advantages.GRPO estimates a baseline from the average reward among sampled solutions for the same problem.Source: §3.1 · p. 3

3.2.1 Formal Formulation

  • The true verification reward for a correct solution reduces to the policy model's next-token log-probability ratio against the reference model, scaled by the KL coefficient.The paper approximates the partition-function term as negligible because reference-model probabilities for verification tokens are extremely small.Source: §3.2.1 · pp. 4–5
  • LaSeR trains self-verification by aligning a last-token score with verifier-based reasoning rewards through an MSE loss added to the RLVR objective.The verification reward scores whether the model's verification matches the solution's true correctness; the formulation focuses on two single-token labels for correct and incorrect solutions.Source: §3.2.1 · pp. 4–5

3.3 Other Techniques

  • A pre-calculated reference log-probability constant avoids forwarding each solution through the reference model, and an unused special token helps keep that probability near zero.At inference, the score can determine a verification outcome using a 0.5 threshold or support weighted majority voting.Source: §3.3 · pp. 5–6
  • Class-level loss re-weighting addresses changing imbalances between correct and incorrect solutions in each training batch.The authors identify solutions with a deterministic verifier and reweight the loss to support more balanced self-verification.Source: §3.3 · p. 5
  • Training separates reasoning and self-rewarding warm-up phases before integrating verifier-based and self-rewarding-based advantages.For base models, standard RLVR first warms up reasoning; self-rewarding loss is introduced later, followed by advantage integration.Source: §3.3 · pp. 5–6

3.4 Brief Discussion

  • LaSeR derives self-rewarding scores from the final solution token instead of generating a separate verification, requiring only one additional token inference at test time.The paper says training incurs no additional computation because token log-probabilities are computed after generation.Source: §3.4 · p. 6
  • Unlike SFT, the last-token loss does not push the correct-verification token probability to 1, which the authors say may interfere with reasoning optimization.The loss instead targets a probability scaled from the reference-model probability by exp(1/βv).Source: §3.4 · p. 6

4 Experiments

Across model variants, LaSeR improves reasoning performance and substantially strengthens self-verification. Its optimized self-rewarding scores also improve inference-time scaling performance.

4.1 Experimental Settings

  • Experiments use DeepMath-103K for RL training and evaluate reasoning and self-verification on five mathematical benchmarks.Evaluation reports Pass@1 and self-verification F1; OctoThinker-based results on AIME24–25 are omitted because correct solutions were too few for reliable evaluation.Source: §4.1 · pp. 6–7, Table 1 · p. 7

4.2 Main Results and Analysis

  • Reasoning accuracy improves in most settings, with higher average accuracy across all three base models than the baseline.The authors attribute the improvement to better confidence calibration and finer-grained advantage estimation.Source: §4.2 · p. 7
  • Around 80% self-verification F1; LaSeR outperforms equally sized external verifiers and matches a 72B reward model.Table 2 compares verification F1 on solutions generated by different policy models.Source: §4.2 · pp. 7–8, Table 2 · p. 8

4.3 Inference-Time Scaling Results

  • Weighted majority voting with optimized self-rewarding scores improves the models’ inference-time scaling performance.Table 3 compares reasoning and self-verification with and without reference log-probability simplification for Open-Reasoner-Zero-7B.Source: §4.3 · p. 8, Table 3 · p. 8

5 Analysis and Discussion

LaSeR’s reference-log-probability simplification preserves comparable reasoning and self-rewarding performance while halving the computation for that score. On general reasoning tasks, reasoning accuracy remains comparable to the baseline, but self-rewarding is less accurate; the section also discusses ways to reduce or increase inference cost.

5.1 The Impact of Simplifying The Reference Log-Probabilities to A Constant

  • With reference log-probabilities replaced by a constant, reasoning and self-rewarding performance remain comparable while calculating the score costs half as much.The comparison uses the Open-Reasoner-Zero-7B checkpoint after 200 optimization steps.Source: §5.1 · pp. 8–9

5.2 The Generalizability of LaSeR to General Reasoning Domain

  • On general reasoning benchmarks, LaSeR achieves comparable average reasoning accuracy to the baseline, but its self-rewarding scores overlap more on GPQA-Diamond and still improve weighted majority voting.The authors report the score distinction diminishes on the more challenging GPQA-Diamond benchmark.Source: Figure 3 · p. 9, §5.2 · p. 10

5.3 Further Reduction or Increase of Self-Rewarding Cost

  • A proposed EOS-position score could eliminate extra inference, but the generator may emit the selected token at sequence end and destabilize training.The authors suggest top p below 1.0 as one possible remedy and leave more principled strategies for future work.Source: §5.3 · p. 10
  • An alternative is to compute the score over multiple additional inference tokens, with the benefit to inference-time scaling left for future study.This trades increased inference cost for potentially stronger self-rewarding capability.Source: §5.3 · p. 10

6 Conclusion

LaSeR derives a last-token score equivalent to the verifier’s reasoning reward and adds an MSE objective to standard RLVR. The resulting scores support training and test-time solution ranking and selection.

6 Conclusion

  • LaSeR’s closed-form result equates the verifier’s true reasoning reward with a last-token self-rewarding score based on next-token log-probability, a pre-calculated constant, and the KL coefficient.The score uses the policy model’s next-token probability for a pre-specified token at the final response token.Source: §6 · p. 10
  • LaSeR adds an MSE loss aligning last-token self-rewarding scores with verifier-based rewards to the standard RLVR process.Source: §6 · p. 10
  • The optimized scores can be used in training and at test time to support verification, solution ranking, and selection.Source: §6 · p. 10

A The Length Bias in Implicit Reward

The appendix reports that implicit reward rises with trajectory length, making it unreliable for comparing reasoning paths, and that reference-model log-probabilities can be treated as nearly constant.

A The Length Bias in Implicit Reward

  • Across 32 sampled reasoning trajectories, implicit reward positively correlates with token count, and longer trajectories tend to receive higher final scores.Source: A The Length Bias in Implicit Reward · p. 15
  • Because incorrect solutions are generally longer than correct ones in reasoning tasks, implicit reward is not a reliable test-time indicator of relative reasoning-path quality.Source: A The Length Bias in Implicit Reward · p. 15
  • Across the evaluated inputs, reference-model log-probability for the pre-specified token remains nearly constant with low standard deviation, supporting its replacement by a constant.The reported analysis covers 300 input-output pairs and different reference-model and token combinations.Source: B Statistics of log πre f (zc|x, y) · p. 15, Figure 5 · p. 15
  • Treating that reference log-probability as constant reduces the computation cost of calculating last-token self-rewarding scores by half.Source: B Statistics of log πre f (zc|x, y) · p. 15

C Detailed Training Settings

The training settings use verl with binary correctness-and-format rewards and warm-up phases that vary by model.

C Detailed Training Settings

D Ablation Studies on Self-Rewarding Hyper-Parameters

The ablations examine how self-rewarding loss weight α and coefficient βv affect training, and how class-level loss re-weighting changes self-verification balance. The main experiments use (βv, α) = (0.1, 0.1).

D Ablation Studies on Self-Rewarding Hyper-Parameters

  • Larger α more detrimentally affects reasoning, while βv has little effect on self-rewarding optimization within 0.1–0.5.Very small βv can impair reasoning; the main experiments use βv = 0.1 and α = 0.1.Source: D Ablation Studies on Self-Rewarding… · p. 16
  • At βv = 0.05, πθ(zc|x, y) is about 0.05 under πref(zc|x, y) = e^-23, which the authors say interferes with reasoning optimization.The stated verifier reward is rv(x, y) = 1.Source: D Ablation Studies on Self-Rewarding… · p. 16

E The Effect of Class-Level Re-Weighting on The Balanced Self-Verification Capability

F Comparison between Last-Token Self-Rewarding Loss and Supervised Fine-Tuning Loss

The comparison finds that directly using SFT to optimize self-rewarding interferes with reasoning training, whereas last-token self-rewarding loss fits a less demanding target with less interference.

F Comparison between Last-Token Self-Rewarding Loss and Supervised Fine-Tuning Loss

G Detailed Self-Verification Results

Across benchmarks, the reported results show that LaSeR improves self-rewarding and self-verification capabilities on models’ self-generated solutions, with minimal additional computational cost.

G Detailed Self-Verification Results

H Training and Evaluation Settings in General Reasoning Experiments

The general-reasoning settings specify 800 optimization steps and a reference log-probability constant of −23.0 for WebInstruct-verified, without advantage integration. The appendix also presents training and evaluation prompt templates for several model setups, alongside self-verification prompts and training-dynamics comparisons.

H Training and Evaluation Settings in General Reasoning Experiments

  • For WebInstruct-verified, training uses 800 optimization steps and cref = −23.0, with no advantage integration because integration degraded performance when self-rewarding capability was limited.Source: H Training and Evaluation Settings in… · p. 16

I Prompt Templates

  • Training-dynamics figures plot rewards and self-verification F1 across hyperparameter combinations, with and without class-level re-weighting, and compare last-token self-rewarding loss against SFT loss.Source: Figure 6 · p. 17, Figure 7 · p. 18, Figure 8 · p. 19
  • The cited prompt examples vary: one asks the assistant to reason step by step and put its final answer within \boxed{}, while others require boxed extraction, with one also enclosing reasoning and answer in <think>/<answer> tags.Source: I Prompt Templates · p. 21
  • The self-verification template asks the model to evaluate a tentative response and assign a boxed rating of 0 or 1.Source: I Prompt Templates · p. 22

Related papers

Loading 2510.14943v1…