PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards

Mengjie Ren, Jie Lou, Boxi Cao, Xueru Wen, Hongyu Lin, Xianpei Han, Le Sun, Xing Yu, Yaojie Lu

arXiv:2605.14539v1cs.CL

TL;DR

RLVR often treats diverse failed trajectories as uniformly negative feedback, weakening credit assignment. CIPO turns on-policy failures into correction supervision and outperforms strong baselines across reasoning and correction benchmarks.

  • Problem

    Sparse binary rewards and uniform penalties for heterogeneous failures leave RLVR with ambiguous optimization signals and underused supervision.

  • Method

    CIPO jointly optimizes standard RLVR with correction samples generated by refining the model’s own on-policy failed trajectories, without external annotations.

  • Results

    Across 11 benchmarks, CIPO consistently outperforms strong baselines in reasoning and correction, improving average accuracy by 17.56% across six mathematical benchmarks and surpassing GRPO by 4.55%.

  • Takeaways & Limitations

    Higher pass@K suggests CIPO expands intrinsic reasoning capacity rather than merely redistributing probability mass over existing correct answers.

  • Takeaways & Limitations

    1PRIME exhibits early-training instability, so comparisons use its best-performing checkpoints rather than a stable 500-step optimization trajectory.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective paradigm for improving the reasoning capabilities of large language models. However, RLVR training is often hindered by sparse binary rewards and weak credit assignment, resulting in ambiguous optimization signals and underutilization of the useful information embedded in failed trajectories. To address this challenge, we propose Correction-Oriented Policy Optimization (CIPO), a simple and effective extension to RLVR that converts on-policy failed trajectories into correction-oriented supervision, without relying on any external signals. By jointly optimizing correction samples derived from the model's own failed attempts together with the standard RLVR objective, CIPO improves learning effectiveness while explicitly enhancing the model's ability to correct its own errors. Extensive experiments across 11 benchmarks spanning mathematical reasoning and code generation demonstrate that CIPO consistently and significantly outperforms strong baselines in both reasoning and correction performance. Moreover, CIPO yields stronger pass@K gains, indicating that it improves the model's intrinsic reasoning capacity rather than merely redistributing probability mass over existing correct answers.

1 Introduction

RLVR’s sparse binary rewards uniformly penalize failed trajectories, discarding heterogeneous and potentially useful signals. CIPO instead converts on-policy failures into correction-oriented supervision without external information and improves reasoning and error correction across 11 benchmarks.

  • Motivation: Existing RLVR methods reinforce successful trajectories while uniformly penalizing failures, producing ambiguous signals that overlook heterogeneous failure quality.The binary, sparse reward structure does not distinguish failures by their logical proximity to the ground truth.
  • Motivation: Discarding intermediate structures in failed trajectories wastes useful supervision and may hinder exploration and generalization.
  • CIPO: CIPO transforms on-policy failed trajectories from penalty targets into correction-oriented supervisory signals without requiring external information or additional annotations.It constructs correction pairs by conditioning on the original prompt and the model’s erroneous output.
  • CIPO: CIPO differentiates failure modes by sampling locally around erroneous trajectories, making near-miss attempts more likely than fundamentally flawed ones to yield correct refinements.The passage gives simple final-step calculation errors as an example of near-miss failures.
  • Results: 7.63% gain on DebugBench was achieved by Seed-Coder-8B trained with CIPO, reaching performance comparable to Claude-4-sonnet and surpassing GRPO.Experiments span 11 benchmarks in mathematical reasoning and code generation, with improvements in both reasoning and error-correction performance.

2 Preliminaries

RLVR trains language models on reasoning tasks whose outputs can be automatically verified, typically using sparse binary, sequence-level rewards. GRPO stabilizes this optimization without a value model, but uniformly penalizes failed trajectories without accounting for their error modes or partial correctness.

  • RLVR: RLVR targets LLM reasoning tasks whose outputs can be automatically verified, such as mathematical answers or functional code execution.Verification determines whether generated outputs are valid.
  • RLVR: Given a prompt x ∼D, the policy πθ generates rollout y and receives a binary reward R(x, y) ∈{0, 1}.RLVR maximizes expected reward under this binary-feedback formulation.
  • RLVR: Sparse, sequence-level verifiable rewards make RLVR policy optimization reliant on sampling-based gradient estimators.The reward structure creates limited direct credit assignment across generated sequences.
  • GRPO: GRPO stabilizes training under sparse binary rewards without a value model by sampling and evaluating groups of N trajectories for each prompt.It computes a normalized relative advantage within each group.
  • GRPO: Failed trajectories receive uniformly negative advantages whenever successful trajectories exist, regardless of specific error modes or potential partial correctness.GRPO reinforces positive-advantage trajectories and suppresses negative-advantage trajectories relative to the group mean.

3 Correction-Oriented Policy Optimization

CIPO extends RLVR by converting failed on-policy trajectories into correction-oriented supervision through iterative generation and replay. It combines base and correction rollouts with adaptive replay, risk-averse shaping, and difficulty-aware preference to improve stability, efficiency, and correction guidance.

  • 3 Correction-Oriented Policy Optimization: CIPO transforms on-policy failed trajectories from penalty targets into exploitable supervisory signals within an iterative generation-and-correction replay cycle.Training uses base on-policy rollouts and correction rollouts conditioned on the original query and a previous trajectory.
  • 3 Correction-Oriented Policy Optimization: By refining a specific failure into a correct solution, CIPO establishes a gradient path from the failure mode to the goal instead of uniform negative suppression.This converts ambiguous failure penalties into directional guidance for improvement.
  • 3 Correction-Oriented Policy Optimization: CIPO stabilizes learning with an adaptive replay ratio that balances failed and successful trajectories according to recent retention performance.The replay fraction of successful trajectories increases when performance degrades and decreases when performance remains stable and high.
  • 3 Correction-Oriented Policy Optimization: Risk-averse shaping penalizes cases where a correct conditioning trajectory produces an incorrect response, explicitly suppressing capability regressions.The penalty preserves existing correct behaviors while allowing acquisition of new ones, and complements macro-level adaptive replay with micro-level regression guidance.
  • 3 Correction-Oriented Policy Optimization: Difficulty-aware trajectory preference prioritizes prompts with moderate empirical pass rates, while falling back to the full distribution when too few medium-difficulty prompts exist.Consistently solved or consistently failed prompts may waste computation or provide zero gradient signals.
  • 3 Correction-Oriented Policy Optimization: The joint objective combines base and correction rollouts, computes advantages separately within each group, and weights correction rollouts using λ > 0.Correction rewards incorporate risk-averse shaping; m and n are the sampled response counts for base and correction rollouts.

4 Experiments

Across mathematical reasoning and code generation benchmarks, CIPO consistently outperforms GRPO and improves both intrinsic reasoning and error-correction capabilities. Ablations further identify on-policy replay, adaptive control, risk-averse reward shaping, and difficulty-aware preference as important design choices.

  • Reasoning and code generation: 64.38% overall mathematical-reasoning accuracy surpasses GRPO by 4.55%, while CIPO also delivers stable code-generation improvements.Under matched computational budgets, CIPO outperforms GRPO (BS=256) by 4.72%.
  • Intrinsic reasoning capacity: CIPO expands intrinsic reasoning capacity rather than merely reshuffling existing solutions through sampling.This conclusion is supported by pass@32 evaluation on competition-style mathematical benchmarks and training-dynamics analysis on code-generation tasks.
  • Correction performance: 7.74% correction-rate improvement on CriticBench (Math) surpasses GRPO by 4.67%, while DebugBench gains reach 4.20%.CIPO improves error detection and rectification, significantly outperforming GRPO across CriticBench and DebugBench.
  • Cross-scenario generalization: Math-trained CIPO generalizes correction capabilities to unseen symbolic and algorithmic tasks despite training solely on mathematical data.The reported results describe substantial out-of-domain correction gains.
  • Ablation study: 3.91% additional improvement over offline replay establishes on-policy replay as a key driver of performance gains.The comparison uses a “step-0 replay” variant based on offline trajectories.
  • Ablation study: Removing risk-averse reward shaping causes the largest degradation, with overall accuracy plummeting by 6.97%; fixed replay and removed difficulty-aware preference also reduce performance.The fixed 1:1 replay ratio trails full CIPO by 4.19%, while difficulty-aware preference removal causes consistent declines across benchmarks.

5 Related Works

Prior work addresses sparse RLVR rewards through process supervision, learns from failed trajectories, and improves language-model self-refinement. These approaches provide denser feedback or correction data but may rely on external signals or compromise first-pass performance.

  • Process Supervision: Process supervision densifies RLVR feedback by evaluating intermediate steps, commonly through process reward models, LLM critics, or environment feedback.Process reward models assess step-level correctness for reinforcement learning or search guidance.
  • Learning from Failure: Learning-from-failure methods treat failed trajectories as informative signals, including hindsight experience replay and post-hoc rewriting into training data.LLM approaches often use external rewriters to correct invalid responses.
  • Self-refinement: Self-refinement methods use multi-turn generation to iteratively improve outputs, but may sacrifice first-pass performance and often incorporate external feedback.Most such approaches operate on a single refinement trajectory per failure.

6 Conclusion

CIPO turns on-policy failed trajectories into supervisory signals for RLVR training, using correction to extract richer learning signals from failures. Across 11 benchmarks, it improves both reasoning and correction performance, with pass@K gains indicating genuine expansion.

  • CIPO transforms on-policy failed trajectories into exploitable supervisory signals for RLVR training.The method jointly uses information from failures as training supervision.
  • Correction differentiates failure modes, and near-miss attempts are more likely to produce correct solutions during refinement.This makes near-miss failures especially useful for generating richer learning signals.
  • Across 11 benchmarks, CIPO significantly improves both reasoning and correction tasks.
  • Pass@K gains indicate genuine expansion from CIPO.

A Details about Method · A.1 Algorithm of CIPO

CIPO alternates base rollouts, correction rollouts, joint policy updates, and adaptive ratio updates within each training step. Its algorithm is parameterized by group size, replay fraction, difficulty range, risk penalty, and target reward.

  • A.1 Algorithm of CIPO: Correction rollouts are based on the previous step, enabling parallel sampling instead of waiting sequentially for base rollouts to complete and receive rewards.
  • A.1 Algorithm of CIPO: The algorithm requires an initial policy πθ, prompt set D, and reward function R(·).
  • A.1 Algorithm of CIPO: CIPO uses group size G, replay fraction γ, difficulty range [δlow, δhigh], risk penalty λrisk, and target reward R∗.
  • A.1 Algorithm of CIPO: Each training step samples prompts from D, generates G base responses per prompt, computes rewards, and collects base trajectories Bbase.
  • A.1 Algorithm of CIPO: Correction rollouts replay selected base trajectories, augment their prompts with correction responses, and sample G new responses through Eq. 1.
  • A.1 Algorithm of CIPO: The policy update jointly trains on the union Bbase ∪ Bcor of base and correction trajectories.
  • A.1 Algorithm of CIPO: CIPO updates its correction ratio using rewards from correction trajectories conditioned on successful trajectories, then carries the updated ratio into the next step.

A.2 Algorithm of RolloutReplay and UpdateRatio

RolloutReplay prioritizes medium-difficulty prompts, then selects successful and failed trajectories under a controlled mixture ratio. UpdateRatio adapts replay of successful trajectories using retention performance, deterioration, and persistent underperformance signals.

  • RolloutReplay: RolloutReplay prioritizes prompts with empirical pass rates in [δlow, δhigh], falling back to remaining prompts when necessary.The selected trajectories are then split into successful and failed groups.
  • RolloutReplay: The target replay set uses ρ to determine positive samples, fills remaining capacity with failures, and backfills from successes if needed.Specifically, N+ = min(⌊ρN⌋, |B+|) and N− = min(N − N+, |B−|).
  • UpdateRatio: UpdateRatio adapts ρt within [ρmin, ρmax] according to the model’s retention performance on replayed successful samples.Rt is the average shaped reward on replayed corrections, while R∗ is the target retention level.
  • UpdateRatio: The replay-ratio update combines the current target gap, recent retention deterioration, and a capped consecutive-underperformance signal.The consecutive-underperformance contribution is capped at 3, and the resulting ratio is clipped to [ρmin, ρmax].

A.3 Correction Prompt Construction.

CIPO constructs correction prompts from selected rollouts by presenting the original candidate solution without revealing whether it is correct. This turns both failed and successful attempts into supervision for recognizing, fixing, or reinforcing reasoning patterns.

  • A.3 Correction Prompt Construction.: Correction prompts present a selected candidate solution and ask the model to provide its own solution.The prompt conditions on the original problem and the model’s previous attempt.
  • A.3 Correction Prompt Construction.: The prompt states that the candidate’s correctness is unknown, avoiding reward-information leakage and preserving the RL objective.
  • A.3 Correction Prompt Construction.: This construction teaches the model to fix incorrect responses or reinforce correct reasoning instead of merely suppressing failure likelihood.

B Hyperparameters

This section presents the full hyperparameter configuration used for CIPO, documented in Table 6.

  • B Hyperparameters: Table 6 details the full hyperparameter configuration for CIPO.
  • B Hyperparameters: The configuration is specifically identified as CIPO’s hyperparameter setup.
Loading 2605.14539v1…