PaperlayerView exact revision on arXiv

Source-linked AI summary

CriPO: Enhancing Rubric-based RL via Self-Distillation

Mingxuan Xia, Yuhang Yang, Chao Ye, Shuai Zhu, Shenzhi Yang, Guangcheng Zhu, Yuhang Zhang, Cheng Peng, Haobo Wang, Siqing Wang

arXiv:2607.18082v3cs.LGcs.AI

TL;DR

Rubric-based RL struggles with unexplored criteria and suppressed criterion-level behaviors, while external rollout guidance creates a train-inference mismatch. CriPO uses on-policy self-distillation to address both failure modes and consistently outperforms existing baselines with roughly 2.0× fewer optimization steps.

  • Problem

    Rubric-based RL provides no effective signal for unexplored criteria and can suppress useful criterion-satisfying behaviors when criterion scores are aggregated into scalar rewards.

  • Method

    CriPO combines GRPO with on-policy self-distillation, using criterion-injection distillation for unexplored criteria and counterfactual token-level advantage flipping for suppressed criteria.

  • Results

    CriPO consistently outperforms GRPO, HeRL, and OPSD across medicine and science benchmarks and model scales, improving Qwen3-1.7B’s average score from 59.2 to 62.4 (+3.2).

  • Takeaways & Limitations

    CriPO addresses both criterion-level failure modes within a unified on-policy training paradigm while preserving stronger final performance and faster optimization.

  • Takeaways & Limitations

    Applying on-policy self-distillation alone is unstable and can degrade performance, motivating CriPO’s use of GRPO as a reward-grounded backbone.

Abstract

from arXiv · show

Rubric-based RL has recently shown promise in improving LLMs on open-ended tasks. A widely recognized limitation of rubric-based RL is limited exploration: criteria that no rollout manages to satisfy (Unexplored Criteria, UC) receive no optimization signal. Recent methods address this by incorporating rubric information as external guidance during rollout, yet they introduce a train-inference mismatch: the policy is optimized on rollouts produced under external guidance while this guidance is absent at inference time, causing error accumulation through autoregressive decoding. Moreover, these exploration-focused approaches overlook a fundamentally different failure mode that we term Suppressed Criteria (SC) -- criteria that are satisfied by some rollouts yet whose learning signals are lost during optimization because scalar reward aggregation assigns them non-positive aggregate advantages. Our analysis reveals that SC are remarkably prevalent: over 57% of samples exhibit this failure mode throughout training, with an average of 1.8 SC per sample. To simultaneously address both UC and SC without introducing training-inference mismatch, we propose Criterion-Distilled Policy Optimization (CriPO), which enhances rubric-based RL via on-policy self-distillation. For UC, CriPO constructs a criterion-injection self-teacher and computes a localized forward-KL loss to inject missing behaviors into the policy. For SC, CriPO employs a counterfactual self-teacher to locate criterion-relevant tokens in negative-advantage rollouts and flips their token-level advantages to positive values, preserving useful patterns that would otherwise be suppressed. Experiments on medicine and science benchmarks demonstrate that CriPO consistently outperforms rubric-based RL, achieving stronger final performance with approximately $2\times$ fewer optimization steps.

1 Introduction

Rubric-based RL improves open-ended tasks but suffers from limited exploration and a distinct suppression failure in which useful criterion signals are lost during optimization. CriPO addresses both failure modes through unified on-policy self-distillation without off-policy mismatch.

  • Limitations: Rubric-based RL has limited exploration because criteria absent from all sampled rollouts receive no effective optimization signal.This limitation arises because GRPO optimizes only behaviors present in the sampled rollout group.
  • Limitations: External rubric guidance during rollout generation addresses unexplored criteria but introduces a training-inference mismatch.The policy is optimized on externally guided rollouts even though that guidance is unavailable at inference.
  • Limitations: Suppressed Criteria are explored criteria whose learning signals are lost when GRPO aggregates criterion scores into one scalar reward broadcast across the response.Criterion-satisfying rollouts can therefore receive negative or negligible aggregate advantages when they underperform on other criteria.
  • CriPO: CriPO enhances rubric-based RL with on-policy self-distillation to address unexplored and suppressed criteria without off-policy mismatch.For unexplored criteria, it constructs a privilege-conditioned self-teacher rather than generating rollouts under privileged external guidance.
  • CriPO: For suppressed criteria, CriPO uses a counterfactual self-teacher to locate criterion-relevant tokens and locally flip their advantages positive.This preserves useful patterns in negative-advantage rollouts, and together with the unexplored-criterion intervention forms a unified on-policy framework.

2 Preliminaries

Rubric-based RL uses criterion-wise judge scores aggregated into scalar rewards, with GRPO computing group-relative advantages for policy optimization. This setup leaves criteria either unexplored when absent from rollouts or suppressed when criterion-satisfying rollouts receive non-positive aggregate advantages, and both failures are prevalent in practice.

  • Rubric-based Reinforcement Learning: A judge scores each rollout against weighted criteria, whose criterion-wise scores are aggregated into a scalar reward before GRPO optimization.GRPO normalizes each rollout’s reward within its sampled group using A_i = (R_i − µ_R)/(σ_R + ϵ).
  • Failure in Rubric-based RL: Unexplored and Suppressed Criteria: Unexplored Criteria are behaviors absent from the current rollout group because no sampled rollout satisfies the corresponding criterion.Under binary criterion scores, a criterion is unexplored when it is satisfied by none of the group’s rollouts.
  • Failure in Rubric-based RL: Unexplored and Suppressed Criteria: Suppressed Criteria are discovered behaviors whose satisfying rollouts receive non-positive aggregate advantages because scalar reward aggregation obscures criterion-level signals.Suppression includes negative-advantage suppression and zero-advantage suppression.
  • Failure in Rubric-based RL: Unexplored and Suppressed Criteria: More than 83% of medicine-task samples contain unexplored criteria, averaging 2.5 per sample, while over 57% contain suppressed criteria, averaging 1.8 per sample.These failures remain prevalent and persistent throughout training Qwen3-4B on medicine tasks.
  • On-Policy Self-Distillation: On-policy self-distillation converts privileged rubric information into dense token-level supervision while avoiding training-inference distribution mismatch.It compares student and self-teacher next-token distributions and reveals how privileged information changes local generation preferences.
  • On-Policy Self-Distillation: Directly applying OPSD with all rubrics as privileged teacher information is unstable, with all tested KL variants causing performance degradation.The instability is reported for training Qwen3-4B on RaR-Medicine.

3 CriPO: Criterion-Distilled Policy Optimization

CriPO preserves GRPO as its reward-grounded optimization backbone while using on-policy self-distillation to correct unexplored and suppressed criteria without training-inference mismatch. It injects missing behaviors through localized forward-KL distillation and preserves useful behaviors in negative rollouts through criterion-localized advantage flipping.

  • Framework Overview: CriPO combines GRPO with OPSD-derived signals to target both unexplored and suppressed criteria while avoiding training-inference mismatch.The framework retains GRPO as the stable reward-grounded optimization backbone and applies criterion-level corrections on-policy.
  • Behavior Injection for Unexplored Criteria: For unexplored criteria, CriPO revises the highest-advantage rollout with a criterion-injection self-teacher and distills its behavior using a localized forward-KL auxiliary loss.Best-rollout selection limits revisions to the response already satisfying the largest proportion of criteria, producing more reliable corrections.
  • Behavior Injection for Unexplored Criteria: 34.6% of tokens contribute 95% of the total KL, motivating contribution-guided filtering that focuses distillation on informative positions and removes noisy supervision.The teacher’s revision is localized, so most token-level KL primarily reflects prompt-modification perturbations rather than criterion-specific signals.
  • Advantage Flipping for Suppressed Criteria: For suppressed criteria, CriPO uses a counterfactual self-teacher to localize criterion-satisfying tokens in negative rollouts, flips their advantages positive, and feeds them back into GRPO.This preserves useful criterion-satisfying parts without promoting the entire negative trajectory, locally reversing updates for the selected tokens.
  • Overall Objective: CriPO’s overall objective preserves GRPO’s global optimization structure while adding an OPSD loss whose strength is controlled by β.Algorithm 1 applies behavior injection for unexplored criteria and advantage flipping for suppressed criteria before computing the combined loss.

4 Experiment

Experiments on medicine and science QA show that CriPO consistently outperforms rubric-based RL baselines across model scales and domains, while reaching strong performance with substantially fewer optimization steps. Ablations and training analyses further support its criterion-targeted self-distillation design.

  • Baselines and Models: CriPO consistently outperforms existing baselines across medicine and science evaluation domains and both Qwen3-1.7B and Qwen3-4B model scales.Compared with GRPO, average scores rise from 59.2 to 62.4 on Qwen3-1.7B (+3.2) and from 68.2 to 69.6 on Qwen3-4B (+1.4).
  • Efficiency and Convergence Analysis: 2.0× fewer optimization steps are required for CriPO to reach GRPO’s best performance, while CriPO ultimately converges to higher final accuracy despite modest per-step overhead.CriPO reaches GRPO’s best performance at around step 175 and surpasses GRPO before GRPO completes training across model scales and domains.
  • Training Dynamics: CriPO and its variants achieve higher reward trajectories and maintain higher entropy than GRPO throughout 200 optimization steps, with full CriPO obtaining the highest reward.CriPO-U shows a notably larger relative entropy gain, reflecting the exploration benefit of behavior injection.
  • Statistics of Unexplored and Suppressed Criteria: CriPO consistently reduces unexplored and suppressed-criteria prevalence relative to GRPO, while each component targets its corresponding failure mode and their combination improves overall performance.The average number of suppressed criteria increases for both GRPO and CriPO versus the base model, plausibly reflecting improved exploration during training.
  • Ablation Study: Removing contribution-guided token filtering reduces CriPO-U’s average score from 66.4 to 65.5, while random suppressed-criterion localization drops CriPO-S’s average score from 68.9 to 64.9.The ablations support selecting high-contribution tokens and accurately locating criterion-relevant tokens rather than applying dense or random interventions.

5 Related Work

Rubric-based RL structures open-ended-task supervision through explicit quality criteria, while on-policy self-distillation turns privileged or feedback-augmented contexts into dense token-level guidance. However, dense self-distillation can be unstable because of noisy gradients, teacher–student inconsistency, and privileged-information leakage.

  • Rubric-based RL: Rubric-based RL decomposes response quality into explicit criteria, enabling more interpretable and controllable supervision than holistic scalar judgments.Examples include factuality, completeness, safety, evidence grounding, and task utility.
  • On-policy Self-distillation: On-policy self-distillation uses the current policy under privileged or feedback-augmented contexts as a self-teacher for dense token-level supervision.This approach converts auxiliary information into token-level guidance while avoiding off-policy teacher mismatch.
  • On-policy Self-distillation: Dense on-policy self-distillation is often unstable due to noisy token-level gradients, teacher–student inconsistency, and privileged-information leakage.

6 Conclusion

The paper identifies unexplored and suppressed criteria as prevalent failure modes of scalar reward optimization in rubric-based reinforcement learning. It proposes Criterion-Distilled Policy Optimization (CriPO) to address both issues.

  • Rubric-based reinforcement learning suffers from unexplored criteria, where no rollout satisfies a criterion and no learning signal is available.
  • It also suffers from suppressed criteria, where criterion-satisfying behaviors are penalized or ignored because aggregate advantages are non-positive.
  • CriPO is proposed as a criterion-distilled policy optimization method addressing both criterion-level failure modes.

A Supplementary Experiments · A.1 OOD Generalization on Instruction Following.

CriPO is evaluated for out-of-domain instruction following on IFEval, IFBench, and MulDimIF after training on rubric-based medicine or science domains. It largely preserves base-model OOD performance, with modest gains on selected benchmarks.

  • A.1 OOD Generalization on Instruction Following.: CriPO-RM and CriPO-RS are trained on RaR-Medicine and RaR-Science, respectively, for supplementary OOD evaluation.The evaluation covers IFEval, IFBench, and MulDimIF.
  • A.1 OOD Generalization on Instruction Following.: The supplementary OOD evaluation reports instruction-following performance on IFEval, IFBench, and MulDimIF.Table 3 identifies these three evaluation benchmarks and distinguishes the two training variants.
  • A.1 OOD Generalization on Instruction Following.: CriPO largely maintains the base models’ OOD instruction-following performance beyond the rubric-based training domain.The evaluation tests whether training preserves general instruction-following ability outside the training domain.
  • A.1 OOD Generalization on Instruction Following.: 0.6987 versus 0.6968: CriPO-RS slightly improves Qwen3-1.7B’s IFEval performance over the base model.The reported values are IFEval scores for Qwen3-1.7B.
  • A.1 OOD Generalization on Instruction Following.: 0.2000 versus 0.1866: CriPO-RS slightly improves Qwen3-1.7B’s IFBench performance over the base model.The reported values are IFBench scores for Qwen3-1.7B.
  • A.1 OOD Generalization on Instruction Following.: CriPO-RS matches the Qwen3-1.7B base model on MulDimIF.This result indicates no reported degradation on that OOD benchmark.
  • A.1 OOD Generalization on Instruction Following.: 0.283 versus 0.2732: CriPO-RS improves Qwen3-4B’s IFBench performance over the base model.The passage reports the CriPO-RS and base-model IFBench values for Qwen3-4B.
  • A.1 OOD Generalization on Instruction Following.: Experimental results across medicine and science benchmarks use Qwen3-32B as judge, with all models evaluated after 200 training steps.Table 4 marks best and second-best results and shows changes over the base model.

A.2 Robustness to Different Judges

Using Qwen3-32B as an alternative judge, CriPO preserves the overall trends and outperforms GRPO and HeRL across both evaluated model scales.

  • A.2 Robustness to Different Judges: Using Qwen3-32B as judge, full CriPO raises average scores over GRPO from 43.7 to 47.1 (+3.4) on Qwen3-1.7B and from 59.4 to 61.5 (+2.1) on Qwen3-4B.CriPO also achieves stronger average performance than HeRL on both model scales.

A.3 Case Study on Medical Reasoning

In a representative medical reasoning case, CriPO reaches the correct abdominal-series diagnosis for suspected NEC and provides more explicit, structured clinical reasoning than GRPO. Its response connects clinical cues to diagnostic rationale and distinguishes first-line abdominal X-ray from endoscopy.

  • Case comparison: CriPO identifies abdominal series as the appropriate initial diagnostic step for suspected necrotizing enterocolitis in a premature infant, whereas GRPO selects fiberoptic endoscopy.GRPO briefly mentions abdominal X-ray but incorrectly frames it as insufficient, while CriPO gives the correct final answer.
  • Clinical reasoning: CriPO links extreme prematurity, bloody stool, and abdominal distention to NEC risk before explaining why abdominal series is fast, non-invasive, and diagnostically informative.It notes that abdominal imaging can reveal pneumatosis intestinalis or portal venous gas.
  • Diagnostic rationale: CriPO contrasts abdominal X-ray with fiberoptic endoscopy, explaining why endoscopy is not typically the first-line test in this presentation.The case comparison emphasizes diagnostic prioritization rather than merely naming the answer.
  • Case comparison: CriPO produces a more complete reasoning process than GRPO by exploring rubric-relevant clinical cues and diagnostic rationales, suggesting deeper reasoning rather than superficial length expansion.This interpretation is presented in the figure caption for the medical reasoning case study.

B Detailed Experimental Settings

The experiments filter easy samples, use standardized training and evaluation configurations, and add stability controls for criterion-teacher construction and localized forward-KL optimization.

  • Dataset Details: 15,658 training and 1,936 testing samples are used for RaR-Medicine, while RaR-Science uses 10,874 training and 1,365 testing samples after filtering easy cases.Samples with Qwen3-4B reward higher than 0.9 are removed; HealthBench and ResearchQA each contribute a randomly selected subset of 500 samples.
  • Implementation Details: Qwen3-32B serves as the rubric judge, with evaluation generation fixed at rollout_n=1, top_k=−1, top_p=0.8, and temperature=0.7 across benchmarks.Training is implemented using the verl framework, and detailed configurations for CriPO and baselines appear in Table 5.
  • Stability Details: K=3 limits teacher prompts to the highest-weight unexplored or suppressed criteria when more than three criteria qualify.This criterion-selection rule is introduced to improve algorithmic stability.
  • Stability Details: 10 is the maximum value used to clamp token-level forward KL before token filtering and gradient updates.CriPO applies this clamping as an additional implementation detail for stability.
  • Stability Details: The first and last 1% of tokens are masked because forward KL is higher near response boundaries, reducing boundary noise during localized OPSD.This behavior is illustrated by the token-wise forward-KL analysis in Figure 10.
Loading 2607.18082v3…