Reward-Adaptive Co-Evolution of Policy and Rubric

1 Introduction

• Existing co-evolution methods prescribe updates globally and ignore prompt-level reward informativeness, even though some rollout groups can undergo advantage collapse as the policy improves.Collapsed groups have nearly identical rewards, causing normalized advantages to vanish.Source: §1 · p. 1
• RACE routes informative rollout groups to policy optimization, caches rubric failures for generator optimization, and masks diversity-collapsed groups from policy updates.Rubric optimization uses a frozen judge and a scale-normalized pairwise-margin reward on cached failure pairs.Source: §1 · p. 1
• Event-triggered switching enters rubric optimization after persistent reward saturation and returns to policy optimization when rubric-reward improvement plateaus.This replaces fixed alternation with state-dependent transitions.Source: §1 · p. 1
• Online threshold adaptation adjusts routing and phase-switching thresholds as training dynamics evolve.Together, group-level and phase-level routing allocate policy updates to informative rewards and rubric adaptation to cases that need it.Source: §1 · p. 2
• RACE achieves the best average downstream performance among compared methods and improves rubric discrimination over the strongest fixed-length alternation baseline on JudgeBench and RewardBench2 with fewer optimization steps.The evaluation spans mathematics, code generation, reasoning, knowledge, and instruction following.Source: §1 · p. 2

2.1 Rubric-Based Reinforcement Learning

• Rubric-based reinforcement learning decomposes response quality into explicit criteria and uses rubric-conditioned judgments as training rewards for open-ended tasks.Prior applications include science and healthcare, instruction following, and agentic tasks.Source: §2.1 · p. 2
• Rubrics generated by teacher models or learned from external supervision are typically fixed during policy optimization and cannot adapt to changes in the policy distribution.The paper instead co-evolves the policy and rubric generator, allocating updates according to observed reward informativeness.Source: §2.1 · p. 2

2.2 Self-Evolving LLMs

• Self-evolving LLM methods use predefined coupling schemes, while their update allocation remains global rather than responsive to rollout-group reward informativeness.RACE combines per-rollout-group signal routing with event-triggered phase switching, without external labels or task-specific verifiers.Source: §2.2 · p. 2

3 Preliminary

• For a prompt, the rubric generator produces criteria with weights, and implicit aggregation uses rubric-conditioned judgments to score a policy response.This avoids separate judge calls for individual criteria while retaining structured supervision.Source: §3 · p. 2
• For each prompt, the policy samples a response group, obtains a reward vector, and uses group-normalized advantages in policy optimization.The advantage normalization uses within-group reward mean and standard deviation, with epsilon for numerical stability.Source: §3 · pp. 2–3
• GRPO minimizes a negative clipped surrogate objective using an importance-sampling ratio and a token-level KL penalty against a reference policy.The rubric generator is also optimized with GRPO, using the rubric-phase reward in Eq. 16.Source: §3 · p. 3

4 Advantage Collapse under Fixed-Length Policy–Rubric Alternation

• The study trains a shared Qwen3-8B model with fixed-length policy and rubric phases, measuring rubric properties and collapse at five checkpoints and downstream performance at steps 500 and 1,000.At each checkpoint, it evaluates 100 fixed prompts and defines exact advantage collapse as zero reward standard deviation.Source: §4 · p. 3
• Advantage collapse remains at 27–32% through step 800 and rises to 44% at step 1,000, despite persistent semantic redundancy and improved JSON validity.Table 1 reports rubric analysis under fixed-length alternation; the results show that improved rubric quality does not necessarily yield more informative rewards.Source: Table 1 · p. 3, §4 · p. 3
• Average score increases only from 60.8 to 61.0 under fixed-length alternation.Table 2 reports downstream performance.Source: Table 2 · p. 3, §4 · p. 3
• Low reward variation can stem from poor rubric discrimination or homogeneous policy outputs, but fixed-length alternation cannot distinguish these cases when allocating updates..Source: §4 · p. 3

5 Method

• RACE uses a shared model for policy and rubric generation, switching phases when reward saturation persists and returning when rubric-reward improvement plateaus.Rubric candidates are trained on cached response pairs with a scale-normalized pairwise-margin reward.Source: §5 · p. 3, §5.1 · p. 4

5.1 Per-Rollout-Group Training-Signal Routing

• Informative groups update the policy; diversity-collapsed groups are masked, while rubric-failed groups are masked and cached for rubric-generator training.Routing distinguishes low reward variation caused by homogeneous responses from weak rubric discrimination over varied responses.Source: §5.1 · pp. 3–4

5.2 Event-Triggered Phase Switching

• Persistent reward saturation triggers rubric optimization, and a plateau in rubric-reward improvement triggers a return to policy optimization.The saturation trigger is smoothed across batches and requires persistence before switching.Source: §5.2 · p. 5

5.3 Online Threshold Adaptation

• Online proportional controllers adapt routing and phase-switching thresholds as training evolves, using target quarantine rates and target phase durations.Threshold updates are projected onto valid ranges; phase-duration targets are soft operating points, not fixed phase lengths.Source: §5.3 · pp. 5–6

6.1 Setup

• Training uses 271K deduplicated Tulu 3 prompts and a shared Qwen3-8B policy/generator, with frozen Qwen3-1.7B judging and common initialization, data, and 500 policy updates.The prompts are used without reference responses or externally generated rubrics.Source: §6.1 · p. 6

6.2 Main Comparisons

• 73.0 is RACE’s highest unweighted average across eight benchmarks; it improves over EvoLM on all eight, with largest gains on BBH (+4.7), PopQA (+3.3), and GPQA (+1.9).RACE also outperforms GPT-4.1- and Qwen3-8B-prompted rubrics by 2.0 and 2.9 points on average, respectively, and the scalar reward model by 9.5 points.Source: §6.2 · pp. 6–7, Table 4 · p. 7
• 35.0 vs. 32.2 on JudgeBench Avg. and 49.2 vs. 46.0 on RewardBench2 averages show stronger rubric discrimination for RACE than EvoLM under frozen judging.The averages are unweighted; JudgeBench Avg. excludes its separately reported Overall score.Source: Table 5 · p. 7, §6.2 · p. 7
• RACE lowers label-only criteria from 20.9% to 9.9% and raises specificity from 10.1% to 18.0%; at convergence it beats EvoLM on both rates.Average criterion length increases from 56.9 to 62.6 characters.Source: §6.2 · p. 7
• RACE's advantage-collapse rate remains 22.4%–24.6% late in training (24.5% overall), versus EvoLM's 27–32% through step 800 and 44% at step 1,000.The analysis associates advantage collapse predominantly with high-score saturation.Source: §6.2 · p. 8

6.3 Ablation Study

• Removing group filtering lowers Policy Avg. from 73.0 to 70.8, while uniform rubric sampling lowers RB2 from 49.2 to 47.9; filter-only reaches Policy Avg. 70.9.The study concludes that both group filtering and targeted rubric training contribute to the full method’s gains.Source: Table 6 · p. 7, §6.3 · p. 8
• Random/fixed alternation lowers policy average/RB2 to 71.4/45.3 and 71.3/44.3, respectively; static thresholds yield 72.1 and 47.1, respectively.The authors identify reward-dependent phase transitions as critical for maintaining discriminative rubrics; online calibration adds a more modest aggregate improvement.Source: Table 6 · p. 7, §6.3 · p. 8

6.4 Training Dynamics

• Policy-phase mean length falls from 27.0 to 21.4 steps across training halves, while rubric phases remain near target M*=12.The authors interpret the shorter policy phases as earlier reward saturation prompting more frequent rubric recalibration.Source: §6.4 · p. 8, Figure 2 · p. 8

• Some rollout groups experience advantage collapse while others remain informative, making heterogeneous reward-discriminability degradation a key co-evolution bottleneck.Source: §7 · p. 8
• RACE combines group-level signal routing, event-triggered phase switching, and online threshold adaptation.Source: §7 · p. 8
• RACE achieves the highest average performance and outperforms the strongest fixed-alternation baseline, using one frozen judge without task-specific verifiers or preference-labeled online rewards.The reported gains use fewer optimization updates than fixed-alternation training.Source: §7 · p. 8

Limitations

• The study uses single-turn benchmarks covering reasoning, knowledge, and code generation; multilingual, multi-turn, and long-horizon settings remain future work.The authors also state that generalization across model families, scales, and judges remains to be explored.Source: Limitations · p. 9

A Algorithm

• During policy phases, rollout groups are routed, informative groups update the policy, rubric failures enter a training pool, and persistent saturation triggers rubric optimization.Rubric-phase updates use cached pairs; the algorithm returns to policy optimization when rubric-reward improvement plateaus.Source: A Algorithm · p. 11

B.1 Model Configuration

• Qwen3-8B serves as both policy and rubric generator with shared weights, while a frozen Qwen3-1.7B model supplies rubric-conditioned scores.Role-specific chat templates distinguish the shared model’s roles; updates are sequential.Source: §B.1 · p. 11

B.2 Optimization and Sampling

• Both roles use GRPO with a 10^-6 learning rate and β = 10^-3 KL coefficient; each update uses one epoch and one minibatch.The configuration also uses a constant learning rate without warmup; the full hyperparameter specification is in Table 7.Source: §B.2 · p. 11, §B.3 · p. 11
• Each update uses 64 unique prompts and 8 samples per prompt, yielding 512 training samples; RACE results average three independent runs.The prompt, generation, and packed-sequence limits are 2,048, 16,384, and 18,500 tokens, respectively.Source: §B.2 · p. 11

C Full RewardBench2 Evaluation

• RACE raises the RewardBench2 average from 46.0 for EvoLM to 49.2 and improves five of six dimensions.It also outperforms both prompted-rubric baselines and RLCER.Source: C Full RewardBench2 Evaluation · p. 11
• The scalar reward model has the highest RewardBench2 score but the weakest downstream performance, trailing RACE by 9.5 points.Rubric-ARM has the strongest RewardBench2 result among co-evolution methods but underperforms RACE downstream.Source: C Full RewardBench2 Evaluation · p. 11
• The comparison suggests that static-benchmark discrimination does not necessarily translate into effective supervision for an evolving policy.RACE improves both rubric discrimination relative to EvoLM and downstream policy performance.Source: C Full RewardBench2 Evaluation · p. 11

D Rubric Examples

• At matched 500 policy updates, RACE used 200 rubric updates versus EvoLM’s 500, with 700 versus 1,000 total steps.Rubrics were analyzed on 100 fixed prompts across four task types.Source: D Rubric Examples · p. 12

D.1 Same-Prompt Comparison

• For the rectangle prompt, RACE specifies the square as the maximizing shape and the area as 144, separating answer correctness from explanation quality.The comparison describes EvoLM’s criteria as broader procedural stages and RACE’s as problem-specific, verifiable checks.Source: §D.1 · p. 12, Table 9 · p. 13

D.2 Rubric Evolution over Training

• Across checkpoints, the rectangle rubric shifts from an outcome-heavy checklist toward task-specific intermediate equations, then repartitions criteria by solution component.At step 350 it names y = 24 − x and A = 24x − x2, making partial-credit decisions more checkable.Source: §D.2 · p. 12, Table 10 · p. 14

D.3 Aggregate Rubric Statistics

• From step 50 to 700, JSON validity rises from 73% to 90%, while concrete-value criteria increase from 10.1% to 18.0%.Compared with EvoLM at step 1000, RACE at step 700 has 90% versus 82% parse validity and 18.0% versus 14.5% specificity.Source: §D.3 · pp. 12–13, Table 11 · p. 14
• These aggregate measures describe rubric form and lexical content, not reward discrimination directly.They suggest more task-specific criteria with fewer underspecified labels.Source: §D.3 · p. 13

D.4 Scale-normalized Pairwise Reward Diagnostics

• Only 4.56% of retained rubric groups have zero reward spread; 95.44% retain nonzero within-group variation.The diagnostic evaluates candidate rubrics on the same cached response pair, with reward variation supplying the group-level GRPO signal.Source: §D.4 · p. 13, Table 12 · p. 14

D.5 Policy-Phase Reward Saturation

• During policy phases, 24.5% of evaluated groups have zero reward spread, while mean within-group spread contracts from 0.073 to 0.064.Among collapsed groups, 95.5% score every response at least 0.8, indicating saturation rather than broad rejection.Source: §D.5 · pp. 13–14, Table 13 · p. 14
• The authors attribute this saturation to fixed rubrics losing discrimination as the policy improves and produces responses that meet their criteria.The collapse rate does not drift upward across training, even as mean reward spread contracts.Source: §D.5 · pp. 13–14

E Per-Step Compute Accounting

• A policy update generates 512 responses and 64 prompt-level rubrics, while a rubric update scores 512 candidate rubrics on both responses of cached pairs, requiring 1,024 judge evaluations.Cached responses are reused, and routing and pairwise-margin rewards add no model calls beyond these counts.Source: E Per-Step Compute Accounting · p. 15, Table 15 · p. 15, Table 14 · p. 15

F Hyperparameter Sensitivity

• The sensitivity study varies the quarantine target and phase-length targets; shorter phase targets encourage more frequent alternation.These lengths are soft operating points for calibrating thresholds, while actual transitions depend on reward saturation and rubric-reward improvement.Source: F Hyperparameter Sensitivity · p. 15, Table 16 · p. 16

G IFEval Evaluation Protocol

• Under final-answer-only IFEval scoring, RACE scores 78.2 and EvoLM 77.6; these scores are comparable to each other, not to EvoLM’s full-completion paper score.The re-evaluation removes the <think>...</think> block before scoring.Source: G IFEval Evaluation Protocol · p. 16

H The Use of Large Language Models

• The LLM was used to improve linguistic clarity and sentence structure, not for research ideation, experimental design, data analysis, or conclusions.Source: H The Use of Large Language Models · p. 16