PaperlayerView exact revision on arXiv

Source-linked AI summary

Cliff: Learning Process Rewards from the First Mistake

Peixuan Han, Runhui Wang, Ketan Ramaneti, Jie Hao, Gerald Friedland, Chris Kong

arXiv:2609.02817v1cs.LG

TL;DR

RLVR’s coarse outcome rewards provide limited guidance on intermediate reasoning. Cliff identifies the first mistake in each rollout and assigns different token-level advantages before and after it, achieving a 7% average gain over vanilla GRPO across evaluated scenarios.

  • Problem

    RLVR’s coarse outcome rewards evaluate only final results, providing limited guidance for distinguishing reasoning processes of different quality.

  • Method

    Cliff uses an LLM teacher to identify the first mistake in each rollout and assigns separate token-level advantages to the valid prefix and subsequent suffix.

  • Results

    7% average performance gain over vanilla GRPO is reported across 12 scenarios, with Cliff outperforming other methods across evaluated settings.

  • Takeaways & Limitations

    Cliff provides a simple, general reward-shaping framework that works with teachers of different capabilities for improving RLVR supervision.

  • Takeaways & Limitations

    For domains with non-binary rewards, Cliff requires thresholding or within-group comparison to obtain binary reward signals.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for large language model (LLM) post-training, but its reliance on coarse outcome rewards leads to limited guidance on intermediate reasoning processes. Existing approaches such as process reward modeling and on-policy distillation introduce additional constraints, such as reliance on a specialized reward model or assuming identical reasoning patterns between teacher and student. Nevertheless, we observe that once a reasoning process first goes wrong, evaluating the subsequent reasoning provides limited additional information, as it is already conditioned on an invalid prefix. Therefore, we propose Cliff, a reward shaping strategy that utilizes an off-the-shelf LLM as a teacher to identify the first mistake in each rollout. As a result, the rollout is naturally decomposed into two parts: a correct prefix and an incorrect suffix. Cliff then converts this signal into token-level advantages, assigning positive advantages for the correct prefix and negative feedback afterward. Experiments across 12 different scenarios demonstrate that Cliff consistently improves reasoning performance, outperforming on-policy distillation by 15% and standard GRPO by 7%, even with teachers of modest capability. Furthermore, we analyse the role of ``ground truth'' in Cliff and investigate its training dynamics. These results establish Cliff as a simple, general and effective approach for improving RLVR with richer, fine-grained supervision.

1 INTRODUCTION

Cliff addresses the coarse outcome supervision of RLVR by identifying the first reasoning mistake, separating each rollout into a correct prefix and incorrect suffix, and converting this boundary into token-level advantages. Across 12 scenarios, it improves reasoning performance over on-policy distillation and GRPO, while supporting analysis of judge-model ground truth and training dynamics.

  • Motivation: RLVR evaluates only final reasoning outcomes, providing coarse-grained supervision for intermediate reasoning processes.RLVR remains scalable and avoids human annotations through automatically verifiable outcomes.
  • Method: Cliff uses a teacher model to locate the first mistake in a student rollout rather than evaluate every subsequent reasoning step.The method treats later reasoning as conditioned on an invalid prefix and therefore limits supervision to the first error boundary.
  • Method: Cliff assigns relatively higher advantages before the first mistake and negative feedback afterward, decomposing rollouts into correct prefixes and incorrect suffixes.This preserves the simple RLVR formulation while adding finer-grained token-level feedback.
  • Results: 15% over on-policy distillation and 7% over GRPO across 12 different scenarios are the reported improvements for Cliff.The paper also reports that ground truth helps weaker judge models and analyzes Cliff’s training dynamics.

2 RELATED WORK

Related work positions RLVR as a scalable paradigm for LLM reasoning post-training, while highlighting distillation and reward modeling as alternative knowledge-transfer approaches with important limitations. Existing methods can suffer from reward hacking or degeneration to outcome rewards.

  • Reinforcement Learning with Verifiable Rewards (RLVR): RLVR uses reinforcement learning for scalable LLM post-training on reasoning tasks, with PPO and GRPO among its common algorithms.RL has been widely used by state-of-the-art LLMs.
  • LLM Distillation and Reward Modeling: On-policy distillation trains students on their own trajectories and has shown strong performance as a recent distillation approach.Knowledge distillation generally trains a student to imitate a teacher’s behavior.
  • LLM Distillation and Reward Modeling: Reward models can produce scalar scores or textual judgments and support process rewards for reasoning tasks.They are a key component of LLM post-training and alignment, although commonly applied to unverifiable tasks.
  • LLM Distillation and Reward Modeling: Existing approaches often suffer from reward hacking or degeneration to outcome rewards.These limitations motivate richer supervision beyond coarse outcome signals.

3 METHODOLOGY

Cliff extends GRPO with teacher-guided process supervision by locating the first reasoning mistake in each student rollout. This Pitfall Step divides the rollout into a valid prefix and erroneous suffix, which are assigned refined token-level advantages.

  • Cliff Overview: Cliff extends GRPO to provide universal, fine-grained process supervision by decomposing each student rollout into a correct prefix and incorrect suffix.The method refines GRPO’s outcome-level supervision using process-level feedback.
  • Teacher Feedback: A stronger teacher independently solves each query, and only groups with teacher-verified correct solutions receive teacher guidance; otherwise, Cliff falls back to vanilla GRPO.An automatic verifier evaluates the teacher’s reference solution before its feedback is used.
  • Pitfall Step: The Pitfall Step p(a) is the first point where reasoning becomes incorrect and separates each rollout into a valid prefix before p(a) and problematic suffix at or after it.The judging prompt is designed to identify genuine reasoning errors rather than harmless typos or alternative valid strategies.
  • Advantage Normalization: λ = 0 is used empirically to avoid length hacking, while the offset b recenters token-level advantages to have zero mean across the group.λ controls positive reinforcement for valid prefixes, and b is the average token-level advantage before recentering.
  • Token-Level Advantages: Cliff assigns higher advantages to valid-prefix tokens while retaining the negative outcome-level advantage for suffix tokens responsible for an incorrect result.This converts the teacher’s Pitfall Step judgment into token-level advantages within GRPO.

4 JUDGE QUALITY OF CLIFF

Cliff’s teacher judges are evaluated against human annotations for solution correctness and first-error localization. Stronger models judge more reliably than they solve, while verified correct reference solutions improve judgment accuracy and Pitfall Step localization.

  • The evaluation compares teacher judgments with human annotations for solution correctness and first-mistake localization on a balanced dataset of 50 correct and 50 incorrect rollouts.Experts identify the first reasoning step leading to an incorrect conclusion, called the Pitfall Step.
  • SOTA achieves the highest problem-solving accuracy and the most reliable Pitfall Step judgments, showing strong agreement with human annotations.The result links stronger problem-solving ability with more reliable process judgment.
  • Qwen3-32B and Gemma3-27B judge Pitfall Steps reliably despite noticeably lower problem-solving accuracy than SOTA.The authors conclude that identifying mistakes may be easier than solving problems from scratch.
  • False negatives occur in approximately 10% of cases, while judges rarely reject correct student solutions incorrectly.Some apparent false negatives may reflect automatic verifiers accepting correct final answers produced through guessed answers or flawed reasoning.
  • Under the “Ground Truth” setting, judgment accuracy exceeds 90% and average p-dis is only around 3 sentences when the reference solution is verified correct.Incorrect reference solutions degrade judgment accuracy and increase p-dis, motivating reference-solution filtering.

5 EXPERIMENTS

Experiments evaluate Cliff across math-reasoning and coding tasks with multiple student and teacher models. Cliff achieves the best performance in every evaluated setting, while remaining effective with moderately capable teachers because its gains come from fine-grained credit assignment rather than merely adding a teacher.

  • Experimental setup: Cliff is evaluated on math reasoning and coding using Qwen3-4B-Base and Phi-4-mini-Instruct as students, with Qwen3-32B and Gemma3-27B as teachers.Qwen3-4B-Base receives supervised finetuning on OpenThoughts before reinforcement learning, and coding uses a binary reward requiring all test cases to pass.
  • Baselines: The comparison includes GRPO, GRPO with Teacher, supervised distillation, and on-policy distillation, with OPD applied only to open-source teachers.GRPO uses sequence-level verifier signals; GRPO with Teacher applies one teacher-derived advantage to the entire rollout, while OPD provides token-level signals from the teacher’s distribution.
  • Main results: GRPO provides a stable baseline and improves over SFT-style distillation and OPD in most experimental settings.This establishes that the observed advantage of Cliff is measured against a competitive reinforcement-learning baseline rather than only supervised methods.
  • Main results: Cliff achieves the best performance under every evaluated setting across different student models and benchmark domains.The results are reported for math and coding benchmarks, and the gains are described as both effective and general across the evaluated settings.
  • Teacher capability: Cliff does not require a near-perfect teacher, because even moderately capable teachers can identify reasoning pitfalls and provide useful training signals.Stronger teachers generally yield better student performance, but the performance gap is described as relatively modest.
  • Ablation: Cliff’s larger gains come from its two-fold credit assignment rather than merely introducing a teacher, since GRPO with Teacher yields only marginal improvements over GRPO.The ablation isolates the teacher’s contribution from Cliff’s credit-assignment mechanism.

6 ANALYSIS

The analysis shows that Cliff remains useful without reliable ground-truth solutions, while λ = 0 avoids the length-hacking behavior associated with larger values. Training dynamics reveal longer reasoning, imperfect teacher–verifier agreement, and stabilization of valid-prefix lengths.

  • Ground-Truth Analysis: Open-source teachers without ground-truth filtering often outperform vanilla GRPO, showing that Cliff can provide process-level supervision without verified solutions.The paper attributes this to judging student rollouts being easier than solving problems from scratch.
  • Effect of λ: λ = 0 provides the best performance with reasonable generation length, while λ = 1.0 produces substantially longer responses and degraded reasoning.The paper attributes the degradation to length hacking caused by stronger positive advantage on valid prefixes.
  • Effect of λ: λ = 0 assigns the correct prefix of an incorrect rollout an advantage of A = −b without additional positive reinforcement and is used in the main experiments.Theoretical analysis likewise characterizes λ = 0 as free from length hacking, whereas sufficiently large λ may promote longer responses.
  • Training Dynamics: Response lengths gradually increase during training, with Cliff generally longer than vanilla GRPO and showing an early spike before stabilization.This pattern suggests that Cliff initially encourages exploration of longer reasoning trajectories.
  • Training Dynamics: 85% to 90% consistency is observed between teacher judgments and the automatic verifier, with lower consistency for the Gemma teacher.Teacher judgments can reject responses accepted by the verifier because teachers assess reasoning validity while the verifier primarily assesses final answers.
  • Training Dynamics: Around 50 steps, the Pitfall Step’s relative position increases drastically to 0.5 before gradually stabilizing.The trend indicates that early training develops longer valid prefixes for difficult problems, followed by more consistent reasoning.

7 CONCLUSION

Cliff identifies the first mistake in each rollout and assigns different token advantages before and after it. The method is task- and model-agnostic, works with teachers of varying capabilities, and yields a 7% average gain over vanilla GRPO.

  • 7 CONCLUSION: Cliff identifies the first mistake, or Pitfall Step, in a rollout and assigns different advantages to tokens before and after it.This reward-shaping algorithm provides a learning signal without additional reward models or restrictive assumptions about reasoning trajectories.
  • 7 CONCLUSION: 7% average performance gain over vanilla GRPO demonstrates Cliff’s effectiveness across teachers with different capabilities.The passage characterizes Cliff as task-agnostic and model-agnostic.
  • 7 CONCLUSION: Consistency is the fraction of rollout outcome rewards judged identically by the teacher and automatic verifier.The Pitfall Step position measures the average fraction of tokens in the valid prefix before the first mistake of an incorrect rollout.

A DATASETS

The datasets section describes separate evaluation pipelines for math and coding tasks, using raw-question prompts without system prompts and task-specific answer verification. It also illustrates the domains with DAPO-Math and DeepCoder examples.

  • Math datasets: Math tasks use only the raw question, and evaluation compares the parsed final integer, fraction, or boxed answer with ground truth.No system prompt is used for math datasets.
  • Coding datasets: Coding tasks combine the raw question, an input-output example, and coding instructions, then evaluate extracted code in a VolcEngine7 sandbox.The setup uses single-turn tasks because models cannot run code in advance.
  • Coding datasets: Coding evaluation uses 10 cases per question, a 5-second time limit, and awards a score of 1 only when the model passes all cases.
  • Dataset examples: The section provides representative examples from the DAPO-Math and DeepCoder datasets.The examples cover a regular nonagon labeling problem and a permutation-based counting problem.

B TRAINING DETAILS … C.3 LENGTH MASS OF CLIFF

The paper specifies Cliff’s practical training configuration and analyzes how GRPO and Cliff shape rollout length. Cliff makes lengthening easier than GRPO under nonzero valid-prefix mass, while λ = 0 prevents length-only reward hacking and truncation requires a hard cap.

  • B TRAINING DETAILS: Cliff uses λ = 0 and teacher rollout temperature 0.6, skipping teacher judging for unanimously verified prompt groups with zero reward variance.The skip loses signal only when the judge disagrees with the verifier, which is described as very rare for such groups.
  • C LENGTH DYNAMICS OF CLIFF: GRPO encourages lengthening if and only if correct rollouts are longer than incorrect rollouts.Its length mass has the same sign as L_cor − L_inc, so lengthening cannot improve the objective without a genuine correctness-length correlation.
  • C.3 LENGTH MASS OF CLIFF: When valid-prefix mass s̄ > 0, Cliff makes lengthening easier than GRPO, and increasing λ strengthens the lengthening trend.The formulation gives f(λ) > 0 and df(λ)/dλ > 0.
  • C LENGTH DYNAMICS OF CLIFF: λ = 0 eliminates reward hacking by lengthening alone, while λ < L_cor/L_max is sufficient in most conditions to keep incorrect-rollout advantage below correct-rollout advantage.These conditions constrain Cliff’s length dynamics.
  • C.3 LENGTH MASS OF CLIFF: Hard-capping p(a) = 0 for overlength rollouts is necessary for stable Cliff training because long truncated rollouts may contain valid reasoning prefixes without finishing.The analysis contrasts this with GRPO, where overlength rollouts almost always receive zero reward and negative advantage.
  • C.1 NOTATIONS: The paper analyzes length dynamics through EOS probabilities: each untruncated rollout contributes |a_i| − 1 non-EOS tokens and one EOS token.This motivates measuring length pressure through length mass.
  • C.2 LENGTH MASS (Φ) AND LENGTH BIAS OF GRPO: GRPO’s lengthening condition is ρ > 1, and lengthening cannot increase its objective unless longer responses are genuinely more often correct.Thus, the analysis finds no exploitable length channel in GRPO.

C.4 CLIFF’S ROBUSTNESS TO REWARD HACKING · D PROMPTS · E CASE STUDIES

Cliff avoids reward hacking under explicit advantage conditions, with λ < 0.3 sufficient for one condition and λ = 0 used because the second requires a very small λ. Its prompts direct judges to identify only the first genuine, answer-invalidating mistake while accepting valid alternatives and corrected attempts.

  • C.4 CLIFF’S ROBUSTNESS TO REWARD HACKING: Cliff excludes length hacking when correct rollouts outrank incorrect ones and incorrect rollouts cannot increase total advantage by lengthening valid prefixes.These are conditions (C1) and (C2).
  • C.4 CLIFF’S ROBUSTNESS TO REWARD HACKING: λ < 0.3 is safe for condition (C1) given the empirical value Lcor ≈0.3Lmax.The bound follows from the sufficient condition derived for (C1).
  • C.4 CLIFF’S ROBUSTNESS TO REWARD HACKING: λ = 0 is used in the main experiments because b ≈0.05 makes condition (C2) require a very small λ.Extending a valid prefix is hacking only when added tokens are padding or repetition; genuine reasoning before an error should be rewarded.
  • D PROMPTS: Stage 2 prompts ask judges to compare student reasoning or code with a reference and identify the FIRST mistake only when the solution is genuinely incorrect.Stage 1 has the judge solve the problem independently, while Stage 2 performs mistake identification.
  • D PROMPTS: Judges should accept different valid approaches, corrected failed attempts, harmless slips, and equivalent notation, while treating uncertainty in the student’s favor.The prompt also rejects minor or unnecessary differences that do not invalidate the answer.
  • D PROMPTS: For coding, judges flag incorrect algorithms, result-changing index or boundary errors, crashes, nontermination, and repetitive or incoherent degeneration.Rare boundary-only bugs, excessive-but-terminating complexity, and later-corrected attempts are not flagged.

E.1 CASE 1: THE TEACHER IDENTIFIES THE STUDENT’S MISTAKE

The teacher identifies the student’s first mistake as incorrectly including 5 among primes less than or equal to 4. This error invalidates the complementary-set definition and leads to an incorrect final answer.

  • Teacher Judgment: The teacher reports that the student’s final answer is 26, whereas the reference answer is 80.
  • First Mistake Identification: The first mistake occurs in sentence [4], where the student includes 5 among primes ≤4.The primes less than or equal to 4 are only 2 and 3; 5 is greater than 4.
  • Why the Approach Fails: The student’s inclusion-exclusion approach counts numbers divisible by at least one of 2, 3, or 5, not numbers whose only prime factors are in that set.For example, 10 is divisible by 2 but also has 5 as a factor.

E.2 CASE 2: THE TEACHER CATCHES AN INCORRECT SOLUTION THAT PASSES THE AUTOMATIC VERIFIER · E.3 CASE 3: A FALSE CASE WHERE THE TEACHER’S PITFALL STEP IS INCONSISTENT WITH HUMAN EXPERTS

The teacher catches a concrete counting error despite the student’s correct final answer, but in the circular-sequence case it incorrectly labels a valid period-20 statement as the first mistake, whereas the human expert identifies a later error.

  • E.2 CASE 2: THE TEACHER CATCHES AN INCORRECT SOLUTION THAT PASSES THE AUTOMATIC VERIFIER: The student’s final answer is 450, matching the reference answer, but the solution contains an undeniable counting error.The teacher therefore marks the solution incorrect and identifies sentence [4] as the first mistake.
  • E.2 CASE 2: THE TEACHER CATCHES AN INCORRECT SOLUTION THAT PASSES THE AUTOMATIC VERIFIER: In sentence [4], the hundreds place has 4 even-digit choices, so the calculation should be 4 × 5 × 5 = 100, not 5 × 5 × 5 = 125.The valid even hundreds digits are 2, 4, 6, and 8.
  • E.2 CASE 2: THE TEACHER CATCHES AN INCORRECT SOLUTION THAT PASSES THE AUTOMATIC VERIFIER: The student also reverses the counts in another subcase, writing 4 × 5 × 5 = 100 where the correct count is 5 × 5 × 5 = 125.The hundreds place is odd and the tens place is even, with 5 choices in each position.
  • E.2 CASE 2: THE TEACHER CATCHES AN INCORRECT SOLUTION THAT PASSES THE AUTOMATIC VERIFIER: Several counting errors compensate, producing the correct final answer of 450 despite incorrect intermediate calculations.The reported subcase total is 325 instead of the correct 350, while a later all-even case introduces an offsetting error.
  • E.3 CASE 3: A FALSE CASE WHERE THE TEACHER’S PITFALL STEP IS INCONSISTENT WITH HUMAN EXPERTS: In the circular-sequence problem, the reference answer is −1, while the student answers 4.The teacher traces the discrepancy to the student’s period argument.
  • E.3 CASE 3: A FALSE CASE WHERE THE TEACHER’S PITFALL STEP IS INCONSISTENT WITH HUMAN EXPERTS: The teacher labels sentence [5] as the first mistake because it assumes period 20 directly, whereas circularity requires the true period to divide gcd(20, 268) = 4.The teacher’s formal judgment is that the sequence has period 4, not period 20.
  • E.3 CASE 3: A FALSE CASE WHERE THE TEACHER’S PITFALL STEP IS INCONSISTENT WITH HUMAN EXPERTS: The human annotator argues that sentence [5] is correct: period 20 is valid for the ring, although insufficient to solve the problem.The missing step is discovering that 4 is also a period.
  • E.3 CASE 3: A FALSE CASE WHERE THE TEACHER’S PITFALL STEP IS INCONSISTENT WITH HUMAN EXPERTS: The human expert assigns the actual pitfall to sentence [25], not to the teacher’s sentence-[5] diagnosis.This creates an explicit disagreement between the teacher’s identified pitfall and human expert judgment.
Loading 2609.02817v1…