Source-linked AI summary
Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubrics
Leheng Sheng, Wenchang Ma, Ruixin Hong, Xiang Wang, An Zhang, Tat-Seng Chua
TL;DR
CoT is important for LLM reasoning but is weakly supervised in outcome-centric RLVR, while direct reward modeling is costly and vulnerable to distribution changes. RLCER addresses this by using self-proposed, self-evolving rubrics to reward CoTs, and the paper reports that these rubrics provide reliable supervision and improve performance over outcome-only RLVR.
Problem
Outcome-centric RLVR largely ignores direct CoT supervision, while reward-model-based supervision requires costly annotation and can struggle with evolving CoT distributions.
Method
RLCER uses a policy as both reasoner and rubricator, generating natural-language rubrics that supervise CoTs and evolve during training without human annotation.
Results
RLCER consistently outperforms outcome-only RLVR across datasets and LLM backbones, while self-proposed rubrics provide meaningful CoT supervision even without outcome rewards.
Takeaways & Limitations
Self-proposed and self-evolving rubrics can provide autonomous CoT supervision within RLVR and support stronger LLM reasoning.
Takeaways & Limitations
The rubricator increases rollout burden and training time, and effectiveness beyond verifiable RLVR domains remains unknown.
Abstract
from arXiv · showhide
Despite chain-of-thought (CoT) playing crucial roles in LLM reasoning, directly rewarding it is difficult: training a reward model demands heavy human labeling efforts, and static RMs struggle with evolving CoT distributions and reward hacking. These challenges motivate us to seek an autonomous CoT rewarding approach that requires no human annotation efforts and can evolve gradually. Inspired by recent self-evolving training methods, we propose \textbf{RLCER} (\textbf{R}einforcement \textbf{L}earning with \textbf{C}oT Supervision via Self-\textbf{E}volving \textbf{R}ubrics), which enhances the outcome-centric RLVR by rewarding CoTs with self-proposed and self-evolving rubrics. We show that self-proposed and self-evolving rubrics provide reliable CoT supervision signals even without outcome rewards, enabling RLCER to outperform outcome-centric RLVR. Moreover, when used as in-prompt hints, these self-proposed rubrics further improve inference-time performance.
1 Introduction
RLCER addresses underconstrained CoT supervision in outcome-centric RLVR by having one policy generate and evolve rubrics that reward reasoning quality alongside answer correctness. Experiments indicate these rubrics provide useful supervision without human annotation and improve RLVR performance.
- Motivation: Outcome-centric RLVR gives identical rewards to distinct reasoning trajectories with the same answer, allowing shortcut or brittle strategies.This underconstrained signal can lead to sub-optimal reasoning behavior and reduced robustness.
- Motivation: Direct CoT supervision is difficult because reward-model training requires fine-grained annotation and policy distribution shifts create non-stationary supervision.The paper frames self-proposed, self-evolving rubrics as a way to avoid human annotations while adapting during training.
- RLCER: RLCER uses one policy as both reasoner and rubricator, generating natural-language criteria whose satisfaction rewards desirable CoT properties.The rubricator is also trained to improve rubric generation by rewarding the fraction of valid proposals.
- RLCER: A rubric is treated as valid when its satisfaction correlates with final-answer correctness across sampled rollouts.The method uses this relationship to align CoT supervision with answer correctness.
- Findings: Self-proposed rubrics alone yield reliable, learnable CoT rewards, while random rubric rewards fail to improve reasoning and can cause performance drops.The result supports using rubric-based rewards even without outcome rewards.
- Findings: RLCER outperforms vanilla RLVR across datasets and model sizes, generalizes from math training to GPQA reasoning tasks, and benefits larger models more.The generated rubrics can also serve as in-prompt hints to improve inference-time reasoning.
2 Related Works
Prior work establishes RLVR, self-evolving single-policy training, and rubric-based rewards, but leaves self-generated CoT supervision and evolving rubrics insufficiently explored. RLCER combines these directions to continually adapt rubric criteria during training.
- RL and self-evolving training: RLVR rewards final-answer correctness on verifiable tasks such as mathematics, puzzles, and code execution.Its task-specific verifiers make these settings comparatively resistant to reward hacking.
- RL and self-evolving training: Self-evolving training lets one policy generate data or reward signals and adopt multiple prompted roles with little human intervention.This multi-role setup provides a basis for self-improving learning signals.
- Rubric-based reinforcement learning: Rubrics are structured, checkable criteria that decompose high-level objectives into fine-grained supervision signals.They have been used to extend reinforcement learning toward non-verifiable and open-ended domains.
- Rubric-based reinforcement learning: Existing rubric methods mainly score response quality and often rely on static criteria that may become mismatched as model behavior evolves.RLCER addresses these gaps by self-proposing and continually evolving rubrics for CoT supervision.
3 Preliminaries
The preliminaries describe RL training as optimizing a policy from rollout rewards, distinguish answer-only RLVR from direct CoT rewards, and motivate multi-role learning under one policy. This setup supports assigning separate roles and rewards within a self-evolving system.
- Incentivizing LLM reasoning with RL: The policy generates a question response containing a chain of thought followed by a final answer.This rollout structure separates the reasoning trajectory from the answer used for evaluation.
- Incentivizing LLM reasoning with RL: RL assigns rewards after rollout generation based on response quality, which may depend on either the CoT, the answer, or both.In standard RLVR, the reward typically depends solely on predicted-answer correctness.
- Incentivizing LLM reasoning with RL: RLVR maps an answer-equivalence indicator to a binary reward, while leaving the CoT itself rarely rewarded directly.The missing direct CoT supervision may induce sub-optimal reasoning strategies.
- Multi-role RL under a single policy: A single policy can learn multiple roles by receiving role-specific rewards under role-specific prompts.The same underlying policy is instantiated differently for each role.
- Multi-role RL under a single policy: Multi-role single-policy training can use one model as both reasoner and data generator, supporting RL from little or no human-labeled data.Different roles can compete or collaborate to improve learning signals and behaviors.
4 Methodology
RLCER extends outcome-centric RLVR by rewarding both final-answer correctness and CoT quality with self-proposed rubrics that evolve according to their alignment with correctness. A shared policy performs reasoning and rubric generation, while role-specific rewards jointly optimize it.
- 4.1 Key Idea: RLCER explicitly rewards how LLMs think by using self-proposed, self-evolving rubrics alongside final-answer correctness.The method is designed to provide autonomous CoT supervision without human annotation.
- 4.2 Two Roles in One Policy: Reasoner and Rubricator: A single policy model serves as both reasoner and rubricator under different prompts, with the rubricator generating natural-language criteria for desirable CoT properties.An external verifier judges whether generated CoTs satisfy the proposed rubrics.
- 4.3 Rewarding How to Think via Self-Proposed Rubrics: The CoT reward aggregates verifier scores over valid rubrics and min–max normalizes the result to [0, 1].The reasoner’s total reward combines this CoT reward with a binary outcome reward based on final-answer equivalence to the ground truth.
- 4.3 Rewarding How to Think via Self-Proposed Rubrics: A rubric is valid when its satisfaction is positively correlated with final-answer correctness and varies across sampled CoTs.The validity conditions are corr(v_k, z) > α and std(v_k) > 0; α is set to 0.2 by default.
- 4.4 Rubrics Self-Evolving for Better Supervision: The rubricator receives a validity-based quality reward and a format reward, encouraging informative, discriminative rubrics while maintaining parseable outputs.Validity rewards are based on the fraction of generated rubrics satisfying the correlation and variation criteria.
- 4.5 Two-Role Optimization under a Single Policy: Role-specific rewards produce separate advantages, whose gradients are aggregated to update the shared policy parameters end to end.This unified optimization lets the model learn from feedback associated with both reasoning and rubric-generation roles.
5 Experiments •
Experiments evaluate whether self-proposed rubrics provide useful CoT rewards, whether RLCER improves reasoning over outcome-only RLVR, how self-evolving affects training, and whether rubrics help as inference-time hints.
- 5.2 (RQ1) Reliability of Self-Proposed Rubrics: Self-proposed rubrics yielded meaningful CoT rewards: rewarding only rubric satisfaction improved reasoning consistently, whereas random rubric rewards failed to improve performance.The random-reward baseline also showed sudden performance drops around 200 training steps.
- 5.3 (RQ2) Effectiveness of RLCER: RLCER outperformed vanilla outcome-centric RLVR across multiple datasets and model sizes on average.The gains were more pronounced for 8B models than for 4B models.
- 5.3 (RQ2) Effectiveness of RLCER: RLCER also generalized from math training to GPQA-series general reasoning tasks.Both methods were trained using the same SFT checkpoint in the comparison.
- 5.4 (RQ3) Mechanism of Rubrics Self-Evolving: Self-evolving rubrics produced a more stable learning curve and gradually outperformed RLCER without the evolving reward.Both RLCER variants outperformed naive RLVR, which quickly converged to sub-optimal performance.
- 5.4 (RQ3) Mechanism of Rubrics Self-Evolving: During training, rubric satisfaction correlation with final-answer correctness increased with self-evolving but remained unchanged in the ablation baseline.The result indicates that self-evolving helps the policy propose increasingly informative rubrics.
- 5.5 (RQ4) Effectiveness of Rubrics as In-prompt Reasoning Hints: Rubrics used as in-prompt hints further improved Best-of-N performance on AIME datasets, suggesting increased exploration.The supplied passage reports this inference-time effect without providing the figure’s numerical values.
6 Limitations
The method increases rollout burden and training time, and its effectiveness outside RLVR remains unknown.
- RLCER requires more training time because adding the rubricator role increases rollout burden.
- The method remains limited to RLVR, so its effectiveness on non-verifiable domains is unknown.
7 Conclusion
The paper proposes RLCER to add self-proposed, self-evolving CoT supervision to outcome-centric RLVR and reports consistent improvements over outcome-only training.
- RLCER rewards CoTs with self-proposed and self-evolving rubrics without human annotation.
- RLCER consistently outperforms outcome-only RLVR across datasets and LLM backbones.
- The authors identify generalization to non-verifiable domains as future work.
A Implementation Details
Implementation uses cold-start supervised fine-tuning followed by RL training on DAPO-Math-17K, with experiments conducted using Qwen models and fixed training settings.
- Cold-start: Cold-start training uses 40k teacher-generated trajectories for math reasoning and rubricator behavior.The data are collected by reject sampling and used for full-parameter supervised fine-tuning.
- Training and evaluation: RL experiments use a maximum of 1500 training steps.
B Prompts
The prompts assign separate reasoner, rubricator, and verifier roles to generate, evaluate, and improve chain-of-thought rubrics. The rubricator is instructed to identify response merits and room for improvement, while the verifier scores responses against binary criteria.
- Reasoner: The reasoner prompt defines the model’s role in producing step-by-step solutions.
- Rubric evolution: The rubricator prompt was modified to encourage criteria whose current response score is below the middle value, triggering refinement during reasoning.
- Rubricator: The rubricator prompt asks for comprehensive evaluation criteria conditioned on a question–answer pair and its improvement opportunities.
- Verifier: The verifier prompt evaluates a mathematical response using binary rubrics and computes a final score.
C Case Study
The case study shows the rubricator identifying concrete calculation errors and proposing more efficient reasoning strategies for a consecutive-digit year problem. These rubrics guide the reasoner as auxiliary supervision.
- C Case Study: The rubricator identified ad-hoc manual listing for small digit sets as the source of miscalculation for d = 0 and d = 1, producing final error 131.
- C Case Study: It also suggested categorizing sets with minimum permutation > 2013 upfront to reduce redundant per-set analysis.
- C Case Study: The self-proposed rubrics effectively guided the model on this specific problem and served as auxiliary reward for the reasoner.