PaperlayerView exact revision on arXiv

Source-linked AI summary

Co-Evolving Policy and Rubric via Capability Awareness for Open-Ended Generation

Chengsheng Zhang, Wenshen Xu, Yuchuan Wu, Zhiyuan Yu, Junfeng Fang, Tianlong Wang, Bingxue Qiu, Yongbin Li, Xiang Wang

arXiv:2610.38566v1cs.AI

TL;DR

Static or evolving rubrics may not track a policy's changing capability frontier, limiting informative supervision and guided exploration. CoCA diagnoses capabilities by contrasting query-only and rubric-conditioned rollouts, then adapts criteria and uses frontier expansion or selective self-distillation. Experiments report consistent performance and generalization gains, alongside internalization of guided capabilities into query-only behavior.

  • Problem

    Static rubric criteria may become saturated or remain unexplored, while existing rubric evolution does not explicitly align criterion difficulty with the policy's changing capability boundary.

    Source: §1 · pp. 1–2

  • Method

    CoCA contrasts query-only and rubric-conditioned rollouts for criterion-level diagnosis, recalibrates difficulty, and uses guided frontier expansion or self-distillation to couple rubric and policy updates.

    Source: §1 · p. 2

  • Results

    CoCA consistently improves performance and out-of-domain generalization across diverse tasks and models, and analyses show more favorable capability-state transitions.

    Source: §5 · p. 10, §7 · p. 11

  • Takeaways & Limitations

    CoCA selectively transfers qualified guidance-revealed gains into query-only behavior while promoting more stable capability-state transitions.

    Source: §5 · p. 10, §7 · p. 11

    Psychological-content outputs may be inaccurate, biased, or harmful advice; the resulting models are intended for research, not diagnosis, treatment, crisis intervention, or counseling.

    Source: Ethics statement · p. 12

Abstract

from arXiv · show

Rubric-based reinforcement learning (RL) offers interpretable, fine-grained supervision for aligning large language models (LLMs) on open-ended generation tasks. However, static rubric criteria may become saturated or remain beyond the policy's exploration, rendering the reward signals insufficiently discriminative. While recent efforts seek to guide sampling exploration or evolve rubrics, they fail to explicitly factor in how rubric difficulty should track the evolving policy's capabilities, resulting in a persistent mismatch between the optimization objective and the policy's current capability frontier. In this paper, we propose \textbf{Co}-Evolving Policy and Rubric via \textbf{C}apability \textbf{A}wareness (\textbf{CoCA}), a closed-loop framework that jointly adapts the rubric and policy through criterion-level diagnosis. Specifically, CoCA contrasts query-only and rubric-guided rollouts to characterize each criterion relative to the current policy, recalibrates criteria with mismatched difficulty, and identifies guidance-revealed capability gaps for frontier-based learning. Based on this insight, CoCA either expands rubric-conditioned frontiers through guided optimization or transfers qualified gains through on-policy self-distillation. As the policy updates, CoCA improves the capability diagnosis and enables more reliable distillation, thereby completing the co-evolution loop. Extensive experiments across diverse benchmarks and LLMs demonstrate that CoCA consistently improves performance over rubric-based RL methods. Moreover, CoCA promotes the internalization of guidance-revealed capabilities into query-only behavior while mitigating capability degradation.

1 Introduction

CoCA addresses the mismatch between evolving policy capabilities and static or externally evolved rubric criteria by using criterion-level diagnosis to jointly adapt rubrics and policy learning. Experiments report consistent gains over rubric-based RL, faster convergence, and internalization of guidance-revealed capabilities into query-only behavior.

1 Introduction

  • Predefined criteria can become saturated or remain unexplored as training progresses, limiting reward discriminativeness and stalling policy improvement.The motivation situates this issue in rubric-based RL for open-ended generation.Source: §1 · p. 1
  • Existing rubric-evolution methods may adopt criteria misaligned with task objectives or policy capabilities, while outcome-only supervision leaves desired behaviors to sample-inefficient trial and error.The introduction also cautions that indiscriminate criteria can promote stylistic drift.Source: §1 · p. 2
  • Query-only failures can hide capabilities elicitable under rubric guidance, while guidance may reveal behaviors already within unaided capability.These distinctions complicate identifying the capability boundary and selecting reliable exploration targets.Source: §1 · p. 2
  • CoCA contrasts query-only and rubric-conditioned rollouts to diagnose criterion-level capabilities, recalibrate mismatched difficulty, and either transfer qualified guided gains or expand the frontier through guided optimization.The resulting improved responses inform subsequent diagnosis and self-distillation, closing the co-evolution loop.Source: §1 · p. 2
  • CoCA consistently improves over standard rubric-based RL, reaches comparable performance in approximately 2× fewer training steps, and shows robust out-of-domain generalization.Further analyses trace behaviors elicited by rubric guidance becoming consolidated into query-only responses.Source: §1 · p. 3
  • CoCA uses dual-group capability diagnosis as a shared decision signal for rubric adaptation and policy learning, coupling them in a closed loop.The paper presents persistent tracking of the capability boundary as a way to maintain informative supervision throughout training.Source: §1 · p. 3

2 Preliminaries

The preliminaries define rubric rewards from criterion-level judge verdicts, describe GRPO's sequence-level optimization signal, and introduce on-policy self-distillation as token-level supervision from privileged context. The CoCA pipeline combines these mechanisms with dual-group diagnosis, rubric calibration, frontier expansion, and selective transfer.

2 Preliminaries

  • Rubric-based RL pairs each query with weighted criteria, uses binary judge verdicts for criterion satisfaction, and aggregates them into a scalar reward normalized by total positive weight.Positive weights reward desirable behaviors and negative weights penalize undesirable ones.Source: §2 · p. 3
  • Reward normalization assumes at least one positive-weight criterion, and a single positive-weight binary correctness criterion recovers the standard verifiable-reward setting.Source: §2 · p. 3
  • GRPO uses the rubric score as an outcome reward and applies the same group-normalized sequence-level advantage to every response token, without criterion-level token attribution.The token-level importance ratio compares current and frozen-policy probabilities; ε controls clipping.Source: §2 · p. 3
  • On-policy self-distillation uses privileged training context for a self-teacher and supplies soft token-level targets on student-generated prefixes for query-only behavior.The cited passage contrasts this dense supervision with GRPO's sequence-level signal.Source: §2 · pp. 3–4
  • CoCA contrasts query-only and rubric-conditioned rollout groups to diagnose criterion-level capabilities, calibrates criterion difficulty, and selects qualified frontiers for transfer.Normal GRPO remains active for every query; Guided GRPO expands the rubric-conditioned frontier when no qualified frontier is available.Source: Figure 2 · p. 4

3 Methodology

CoCA diagnoses criterion-level capabilities by contrasting query-only and rubric-guided rollouts, then adapts rubric difficulty and optimization to those diagnoses. Its joint training alternates guided frontier building and selective transfer to query-only behavior, refreshing the diagnosis as the policy changes.

3.1 Capability Probing via Dual-Group Rollouts

  • CoCA samples query-only and rubric-guided responses from the same pre-update policy, compares polarity-aware target achievement, and classifies each criterion into five policy-relative states.The states distinguish Saturated, Learnable, Too-Hard, Discriminative, and Regressive criteria, routing them to calibration, frontier learning, or group-relative optimization.Source: §3.1 · pp. 4–5

3.2 Capability-Aware Rubric Calibration

  • CoCA strengthens Saturated criteria and weakens Too-Hard criteria while preserving each criterion’s intent, polarity, weight, and metadata; other criteria remain unchanged.Both response groups are then evaluated under the same adapted rubric for comparable rewards.Source: §3.2 · p. 5

3.3 Frontier-Guided On-Policy Self-Distillation

  • A Guided response qualifies as a frontier only if it meets every Learnable criterion and exceeds the highest-reward Normal response by the prescribed margin.The eligible Guided response with the highest full-rubric reward becomes the Guided Frontier; the gate requires strict improvement.Source: §3.3 · p. 6
  • For qualified queries, CoCA conditions a self-teacher on Learnable criteria and the Guided Frontier while the student receives only the query, then minimizes forward KL along the Current Frontier.The full-vocabulary KL is approximated with the teacher’s top-k tokens and one tail-mass bin.Source: §3.3 · p. 6

3.4 Guided Frontier Expansion and Joint Optimization

  • Normal GRPO remains active for every query, while Guided GRPO trains queries without a qualified frontier; subsequent guided rollouts refresh diagnosis, calibration, and distillation frontiers.This feedback closes the policy–rubric co-evolution loop.Source: §3.4 · p. 6

4 Experiments

Experiments span three model backbones and medical, science, and psychology benchmarks, comparing CoCA with rubric-based RL baselines. CoCA leads the reported mean scores, generalizes out of domain, and its ablations support complementary component roles.

4.1 Experimental Setup

  • Experiments cover Qwen3-4B, Qwen3-8B, and Gemma-3-4B-it across medical, science, and psychology, with five baselines spanning rubric-free GRPO and rubric-based interventions.Medical training uses HealthBench examples excluding HealthBench-Hard; OOD instruction-following is assessed on InfoBench, FollowBench, and AdvancedIF.Source: §4.1 · p. 7

4.2 Overall Performance

  • CoCA’s mean scores are 61.87%, 66.41%, and 56.08% for Qwen3-4B, Qwen3-8B, and Gemma-3-4B-it, exceeding RaR by +2.51%, +2.43%, and +3.10%.It also outperforms RuscaRL, Online Rubrics, and RGSD on most individual benchmarks.Source: §4.2 · p. 8
  • With the full ground-truth rubric, CoCA improves over the base model by +17.38% for Qwen3-4B and surpasses RuscaRL by +1.89% ∼+4.80% on HealthBench-Hard.Across OOD benchmarks, CoCA consistently outperforms RuscaRL and generally improves over base models, while RuscaRL drops −0.22% to −4.15%.Source: §4.2 · p. 8, Table 2 · p. 8

4.3 Ablation Study

  • Under three alternative evaluation judges, CoCA improves Qwen3-8B’s mean score over the base model by +11.23%, +12.16%, and +13.72%, and exceeds RaR by +2.71% ∼+4.00%.DeepSeek-V3.2 remains the training judge; evaluation uses other judges.Source: §4.3 · p. 8, Table 3 · p. 8
  • The full CoCA reaches 46.86% on HealthBench-Hard and 52.77% on LLMEval-Med, exceeding RaR by +4.98% and +2.84%, respectively.Ablating Rubric Calibration, LFrontier-OPSD, and LGGRPO changes HealthBench-Hard performance by −3.75%, −2.39%, and −1.85%, and LLMEval-Med by −1.80%, −0.15%, and −1.65%.Source: §4.3 · p. 9, Table 4 · p. 9

5 Analysis and Discussions

The analysis examines rubric capability states, transitions toward internalization, and training-time and inference-time behavior. CoCA shifts state distributions, improves internalization transitions while suppressing degradation, and sustains exploration with efficient learning.

Capability-state distributions

  • CoCA shifts Saturated proportions by +14.9% ∼+20.5% and Learnable by −8.1% ∼−5.9% across backbones, while Too-Hard and Regressive proportions also decrease.Marginal state counts alone cannot distinguish internalization from degradation, motivating transition-level analysis.Source: §5 · p. 10

Capability-state transitions

  • CoCA exceeds RaR on every internalization transition across all three backbones; on Qwen3-8B, Learnable ⇒ Saturated and Discriminative ⇒ Saturated rise by +7.06% and +10.53%.It also reduces Learnable ⇒ Too-Hard transitions by −0.51%, −0.07%, and −1.11% across Qwen3-4B, Qwen3-8B, and Gemma-3-4B-it.Source: §5 · p. 10

Training dynamics and inference-time scaling

  • For Qwen3-8B, CoCA reaches 42.19% at step 250, slightly exceeding RaR’s 41.88% at step 500, with approximately 2.0× fewer training steps.CoCA maintains higher actor entropy than RaR over most of training, and its Best-of-K advantage remains +3.85% to +4.45% for K ∈{1, 2, 4, 8, 16}.Source: §5 · p. 10, Figure 4 · p. 10

6 Related Work

Related work develops rubric-based rewards, rubric evolution, and on-policy distillation for open-ended generation. CoCA’s distinction is to couple capability-aware rubric calibration with transfer of capability-qualified frontiers.

Rubric as Reward in RL

  • Rubric-based RL decomposes open-ended response evaluation into interpretable criteria, while later methods scale rubric generation, jointly optimize rubric generators and judges, or scaffold exploration.DR Tulu and Online Rubrics update criteria from policy-generated responses, but do not track the policy’s evolving ability to fulfill the task objective.Source: §6 · pp. 10–11
  • CoCA adapts criterion difficulty to evolving policy capabilities while preserving the original evaluation intent, unlike rubric updates that risk stylistic drift.The paper reports further verification in Appendix E.3.Source: §6 · p. 11

On-Policy Distillation and Self-Distillation

  • On-policy self-distillation uses privileged information from the same model, but sample-level teacher signals may miss whether criterion-level gains are elicitable and transferable.CoCA instead transfers capability-qualified frontiers while jointly expanding their source through capability-aware rubric calibration.Source: §6 · p. 11

7 Conclusion

CoCA jointly evolves rubrics and policies through capability-aware diagnosis, aligning optimization with the policy’s evolving capability frontier and selectively transferring guided gains into query-only behavior. The paper reports improved performance and out-of-domain generalization across tasks and models, while noting that benchmark gains do not establish real-world reliability or clinical safety.

7 Conclusion

  • CoCA consistently improves performance and out-of-domain generalization across diverse tasks and models, while analyses indicate guided gains are progressively consolidated into stable query-only behavior.The authors also report complementary effects among CoCA’s components and more favorable capability-state transitions.Source: §7 · p. 11

AI use statement

  • LLMs supported psychology-data construction, rubric rewriting, reward computation, capability diagnosis, evaluation, and manuscript editing; the authors take responsibility for the research and final manuscript.The paper points to appendices for data-construction procedures, prompts, and rubric calibration examples.Source: AI use statement · p. 11

Ethics statement

  • The experiments use publicly available research benchmarks and pretrained models, without collecting new human-subject data or attempting to identify individuals.Medical and psychological outputs may be inaccurate, biased, or harmful; the models are intended only for research, not clinical or counseling use.Source: Ethics statement · pp. 11–12
  • Rubric generation and LLM-based judging can inherit source-data and evaluator biases, while adaptive optimization may amplify misspecified criteria or judge artifacts.The authors state that deployment-oriented use would require domain-expert review, bias audits, privacy assessment, and application-specific safety evaluation.Source: Ethics statement · p. 12

Reproducibility statement

  • Dataset construction, preprocessing, model configurations, training hyperparameters, implementation details, and prompts are documented in the cited appendices.The section directs readers to Appendix C.1, C.2, and H, respectively.Source: Reproducibility statement · p. 12

A Extended Related Work and Comparison

  • CoCA’s shared capability diagnosis coordinates criterion recalibration and frontier learning by distinguishing guidance-elicited behaviors from targets unmet in both rollout groups.Renewed diagnosis updates calibration and learning targets as the policy evolves; the method comparison distinguishes adaptive rubrics, token-level signals, and state-based learning mechanisms.Source: A Extended Related Work and Comparison · pp. 18–19, Table 6 · p. 19

B CoCa Algorithm

  • Each training iteration diagnoses capability with dual-group rollouts, recalibrates criteria, and routes queries to frontier transfer or expansion while retaining Normal GRPO.Rewritten criteria reconstruct rewards for the current rollout groups only; each later visit to a query starts from its original rubric.Source: Appendix Contents · p. 18, B CoCa Algorithm · p. 19

C.1 Open-ended Generation Datasets

The experiments span medical, scientific, psychological, and out-of-domain instruction-following tasks, with benchmark-specific datasets and metrics. Implementation and training configuration details are documented alongside the dataset descriptions.

C.1 Open-ended Generation Datasets

  • Medical evaluation uses HealthBench-Hard, RaR-Medicine, and LLMEval-Med, with benchmark-specific rubric scoring or automated Overall Performance scoring.HealthBench training uses 4,000 examples outside its 1,000 hard test examples; RaR-Medicine contributes 2,242 test examples, and LLMEval-Med covers 667 Chinese-language clinical questions.Source: §C.1 · p. 19
  • ResearchQA uses 4,000 training examples and 500 held-out validation examples, while RaR-Science provides 500 held-out questions for cross-dataset evaluation using Rubric Score.Source: §C.1 · p. 20
  • Psychological evaluation uses 2,400 CPsyCoun training examples and 500 test examples, plus HeartBench’s 296 rubric-annotated counseling dialogues.CPsyCoun is evaluated with Rubric Score; HeartBench averages normalized dialogue scores across ability dimensions, with designated veto violations zeroing affected dimension scores.Source: §C.1 · p. 20
  • Out-of-domain instruction following is evaluated on InfoBench, FollowBench, and AdvancedIF using requirement pass rate, Hard Satisfaction Rate, and conversation pass rate, respectively.The benchmarks include 500, 820, and 1,645 evaluated prompts or conversations, respectively.Source: §C.1 · p. 20

C.2 Implementation Details and Reproducibility

  • CoCA and baselines use PyTorch with verl, Hugging Face Transformers, and vLLM, with full-parameter fine-tuning, FSDP, and mixed precision.The section also states that AdamW uses a constant learning rate and method-specific settings override general RL settings where applicable.Source: §C.2 · p. 20

D Additional Experiments

D.1 Rollout Budget Controls and Sampling Robustness

CoCA’s gains persist under matched rollout budgets, extend to Qwen3-14B, and vary with the self-distillation trajectory and use of privileged demonstrations.

D.1 Rollout Budget Controls and Sampling Robustness

  • +1.20% to +2.21% mean-score gains over the strongest same-budget baseline are reported across CoCA configurations; even 8 rollouts exceed every baseline using 12.These budget-control results are reported in Table 10.Source: §D.1 · p. 21, Table 10 · p. 21
  • +1.01% mean score (55.69 vs. 56.70) results from shifting the 12-rollout allocation from (6, 6) to (8, 4), improving all three benchmarks.Source: Table 10 · p. 21, §D.1 · p. 22

D.2 Scaling to Larger Models

  • On Qwen3-14B, CoCA achieves state-of-the-art performance across the reported benchmarks, with medical gains over RaR and improvements on three out-of-domain instruction-following benchmarks.Medical gains are +1.59% to +6.45% over RaR and +1.08% to +2.25% over the strongest competing baseline.Source: §D.2 · p. 22, Table 11 · p. 22

D.3 Ablation of the Distillation Trajectory and Privileged Information.

  • Using the Current Frontier instead of the Guided Frontier improves HealthBench-Hard by +1.80% and LLMEval-Med by +3.44%; privileged-context demonstrations yield further gains of +3.04% and +1.20%, respectively.The authors describe the Current Frontier as aligning token-level supervision with the learner’s trajectory and the Guided Frontier as enriching teacher context rather than serving as a direct imitation target.Source: §D.3 · p. 22

D.4 Ablation of the Guided GRPO Gate

Dynamic gating outperforms always-active Guided GRPO on the reported medical benchmarks, while CoCA also complements alternative GRPO updates. OPSD activation generally declines during training, consistent with changing distillation coverage as query-only capabilities develop.

D.4 Ablation of the Guided GRPO Gate

  • +0.41% to +1.79% on HealthBench-Hard and +1.95% to +3.00% on LLMEval-Med are reported for dynamic gating over always-active Guided GRPO across three backbones.The gate focuses auxiliary learning on internalizing available gains when a qualified distillation target exists; otherwise, Guided GRPO seeks a qualified frontier while Normal GRPO remains active.Source: §D.4 · pp. 22–23, Table 13 · p. 22

D.5 Extension to Other GRPO Variants

  • With GSPO and DAPO updates, CoCA yields reported absolute gains of +4.21% and +1.35%, and +4.28% and +2.54%, respectively.The authors characterize these integrations as combining alternative update rules with capability-aware rewards and fine-grained distillation targets.Source: §D.5 · p. 23, Table 14 · p. 23

E.1 OPSD Activation and Capability Internalization

  • OPSD activation generally decreases across all three datasets during training, particularly on CPsyCoun, while HealthBench and ResearchQA retain higher activation.The authors relate the decline to progressive internalization and the sustained activation to remaining learnable requirements exposed by guided rollouts.Source: §E.1 · p. 23

E.2 Training-Time Overhead

CoCA reduces training time relative to rubric baselines while its calibration preserves rubric intent and restores query-only reward discrimination; a case study illustrates criterion-specific adjustment.

E.2 Training-Time Overhead

  • −10.1% (22.50 vs. 25.03 hours) fewer training time than RaR; CoCA's estimate is −14.0% (51.37 vs. 59.71 hours) below Online Rubrics.Reward computation takes a substantial share of training time, partly due to API latency and concurrency limits; API-bound reward computation remains a bottleneck.Source: §E.2 · pp. 23–24, Figure 6 · p. 24

E.3 Rubric Intent Preservation

  • Across datasets, IPR is 94.13% ∼ 98.37% and DDA is 93.10% ∼98.91%, indicating most rewrites retained the evaluation target and reward/penalty direction while adjusting difficulty as intended.Extensive additional validation and filtering were omitted to limit API calls and latency; additional human inspection or filtering could further improve reliability.Source: §E.3 · pp. 24–25, Table 15 · p. 24

E.4 Rubric Discrimination and Reward Signals

  • Calibration raised the discriminative-criterion fraction by +9.20% ∼+10.04% and mean within-group reward standard deviation by +33.48% ∼+37.75%, while only 13.68% ∼15.76% of rewrites became discriminative.It restored reward variation in 42.06% ∼44.54% of initially zero-variance groups and increased nonzero-group coverage by +9.41% ∼+10.49%.Source: §E.4 · p. 25, Figure 7 · p. 25, Table 16 · p. 25

F Case Study

  • On a HealthBench screening case, CoCA relaxed location-seeking after guidance elicited regional advice, but strengthened saturated screening to require an explicit repeat-screening recommendation.The case also preserves discriminative criteria and treats guided failures on the Regressive criterion as a signal to further strengthen rubric-conditioned policy capability.Source: F Case Study · p. 26, Figure 8 · p. 26

G Limitation

CoCA's diagnosis, calibration, and training carry reliability, overhead, and scope limitations. Appendix H documents the prompts used for rubric rewriting, judging, and psychology-data construction.

G Limitation

  • Diagnosis depends on finite rollout groups and LLM judgments, calibration on rewriter capability, and added per-step overhead leaves API-bound reward computation a bottleneck; evaluation covers text generation only.Extending CoCA to multimodal generation or interactive environments with delayed feedback requires further validation.Source: G Limitation · p. 26

H Prompt Design

  • Appendix H supplies polarity-aware rewrite prompts, rubric-free medical listwise judging and criterion-level binary judging, plus a prompt that builds Chinese psychology queries and HealthBench-style rubrics from CPsyCoun cases.Source: H Prompt Design · p. 27, Figure 9 · p. 27, Figure 10 · p. 28

Related papers

Loading 2610.38566v1…