EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning

1 Introduction

• Open-ended tasks lack verifiable answers, making nuanced qualities difficult to formalize as scalar rewards.Rubrics offer interpretable, fine-grained criteria, but their construction can require domain experts or expensive frontier models.Source: §1 · pp. 1–2
• Static rubrics can start too stringent, then saturate as policies improve, weakening reward discrimination and opening the door to reward hacking.The desired evaluation adapts to become more precise and challenging as the model improves.Source: §1 · p. 2
• Prior dynamic approaches rely on external frontier models, update rubrics coarsely, weakly couple adaptation to policy improvement, or require verifiable answers.The open question is how to co-evolve rubrics with policies at training-step granularity for open-ended generation.Source: §1 · p. 2
• EvoRubrics jointly optimizes a Policy LLM and Rubric Generator through within-step adversarial interaction and multi-objective rubric rewards.The components share a base model via dual LoRA adapters; co-evolution can provide informative learning signals even without external supervision.Source: §1 · p. 2
• Within-step co-evolution tracks policy capabilities in real time and induces an automatic curriculum of increasingly fine-grained criteria.The evolved Rubric Generator also supports test-time rubric generation for unseen queries, and the fully self-supervised variant yields meaningful gains.Source: §1 · p. 2
• EvoRubrics consistently outperforms static and dynamic rubric baselines on in-domain and out-of-distribution benchmarks.The trained Rubric Generator also generalizes to test-time rubric generation for unseen datasets.Source: §1 · p. 2

2.1 Static Rubric-based Rewards

• Rubric-based reward methods use structured, interpretable criteria for policy optimization, but pre-constructed rubrics remain fixed as the policy evolves.Related work also studies rubric construction for reward modeling and application to policy optimization.Source: §2.1 · p. 2

2.2 Dynamic Rubrics and Adaptive Evaluation

• Dynamic rubric methods use external frontier models or coarse updates, while learned generators may depend on human preferences or verifiable answers.EvoRubrics couples rubric adaptation with policy improvement within each training step for open-ended RL.Source: §2.2 · p. 3

3 Preliminary: Rubrics as Rewards

• Each rubric criterion describes a quality dimension and assigns a weight, with positive weights rewarding desirable properties and negative weights penalizing undesirable ones.A judge evaluates each criterion independently, and the response score sums the weights of satisfied criteria.Source: §3 · p. 3
• The rubric score is normalized by the maximum achievable positive score, and the normalized score can serve as a transparent, fine-grained RL reward.Pre-defined rubrics can lose discriminative power as the policy evolves, motivating co-evolving rubric rewards.Source: §3 · p. 3
• For open-ended generation without ground-truth answers, the goal is to jointly learn a policy that generates high-quality responses and a query-specific rubric generator.The challenge is keeping the rubrics informative as the policy evolves.Source: §3 · p. 3

4 Methodology

• EvoRubrics jointly optimizes a Policy LLM and Rubric Generator with GRPO for open-ended tasks.The roles use a shared backbone with two independent LoRA adapters.Source: §4 · p. 3

4.1 Dual-LoRA Architecture

• Two independent LoRA adapters specialize a shared backbone for policy and rubric generation, reducing memory and compute costs versus two full models.The inactive adapter receives no gradients during each role’s update phase.Source: §4.1 · p. 3

4.2 Policy LLM Optimization

• Each policy answer is rewarded by its average normalized score across rubric sets, and GRPO normalizes rewards within the group of co-generated responses.Averaging independently generated rubric scores mitigates noise from any single rubric.Source: §4.2 · p. 4

4.3 Rubric Generator Optimization

• The Rubric Generator’s multi-objective reward seeks discriminative, diverse, human-aligned, and constructive criteria that challenge policy weaknesses without drifting toward misaligned evaluation.Discrimination uses score spread across candidate answers; diversity uses pairwise cosine distance between criterion-description embeddings.Source: §4.3 · pp. 4–5
• A fully self-supervised variant removes human-authored rubric dependence by retaining discrimination and diversity rewards and assessing constructiveness through answer revision under generated rubrics.The standard reward formulation uses pre-constructed golden rubrics as a human-preference anchor.Source: §4.3 · p. 5

4.4 Co-Evolutionary Training

• Within-step alternation updates both models against each other’s latest behavior, enabling real-time co-adaptation rather than epoch-level updates.As policy capability improves, stronger rubrics provide finer criteria and create an automatic curriculum.Source: §4.4 · p. 6

5.1 Experimental Setup

• Experiments use HealthBench training, HealthBench Hard and RaR-Medicine in-domain tests, and MT-Bench and FollowBench OOD tests, comparing RL methods from a shared SFT checkpoint.Source: §5.1 · p. 6

5.2 Main Results

• EvoRubrics achieves the strongest overall performance among rubric-based RL methods across backbones, improving in-domain results while largely preserving or sometimes improving OOD performance.The gains are especially pronounced on HealthBench Hard.Source: §5.2 · p. 6, Table 1 · p. 7
• The generator improves RubricBench pairwise judging accuracy over Vanilla and SFT, with the 8B backbone surpassing Qwen3-32B, and generated rubrics improve test-time response quality over base models.Source: Table 2 · p. 7, §5.2 · p. 7

5.3 Ablation Studies

• Removing any reward term degrades performance; constructiveness and diversity most affect in-domain results, while removing discriminativeness harms both in-domain and OOD outcomes.Source: Table 3 · p. 7, §5.3 · pp. 7–8

5.4 Fully Self-Supervised Co-Evolution

• Without external rubric supervision, co-evolution beats the base model and GoldenRubrics on HealthBench but trails the reference-anchored full model; its evaluator improves overall RubricBench accuracy and performs particularly well on Safety.Source: Table 4 · p. 8, Table 5 · p. 8, §5.4 · p. 8

• EvoRubrics jointly improves a Policy LLM and Rubric Generator through real-time interaction, adapting criteria to policy capability and inducing an automatic curriculum.Source: §6 · p. 8
• The framework is effective on in-domain and OOD benchmarks, and a fully self-supervised variant still delivers meaningful gains.Source: §6 · p. 8
• Future work includes multi-domain settings, larger backbone models, and rubric-based feedback for complex tasks.Source: §6 · p. 9

Limitations

• Training is primarily in the medical domain and experiments use models up to 8B parameters, leaving broader-domain transfer and larger-model performance less tested.Whether the dual-LoRA design extends effectively to architectures such as mixture-of-experts models also remains open.Source: Limitations · p. 9
• Co-evolving components trained from the same backbone may amplify shared biases or spurious preferences.The authors also warn of evaluator drift toward narrow or domain-specific criteria and policy exploitation of reward idiosyncrasies.Source: Limitations · p. 9

A EvoRubrics Algorithm

• Algorithm 1's supplied excerpt indicates a per-step query-batch sampling stage followed by a rubric-generator update.The excerpt labels the algorithm as co-evolutionary training but does not provide the intervening update details.Source: A EvoRubrics Algorithm · p. 11

B Training Details

• The four rubric-generator sub-rewards are equally weighted at λ = 0.25; OnlineRubrics and RuscaRL use a 2 × 10−5 learning rate, LoRA rank 32, and alpha 64.The table is identified as the GRPO hyperparameter settings for EvoRubrics.Source: B Training Details · p. 11, Table 6 · p. 11
• Each experiment is run once because training and evaluation incur substantial GPU and API costs.Source: B Training Details · p. 11
• Experiments use VERL, PyTorch 2.6, CUDA 12.4, vLLM 0.8.5, PEFT 0.19, and FlashAttention-2, on one node with eight NVIDIA A100-80GB GPUs coordinated by Ray 2.43.The software and hardware details are given across the implementation passages.Source: B Training Details · p. 11

C.1 Policy LLM Evaluation

• Policy evaluation reports Score and Ratio, respectively measuring average rubric points and scores normalized by each response's attainable maximum.The metrics are used on rubric-based benchmarks.Source: §C.1 · p. 11
• The Policy LLM is evaluated on seen HealthBench Hard, unseen-within-domain RaR-Medicine, and out-of-domain MT-Bench and FollowBench.The supplied passages describe HealthBench Hard as 1,000 physician-curated cases; MT-Bench reports round scores, while FollowBench reports hard and soft satisfaction rates.Source: §C.1 · pp. 11–12

C.2 Rubric Generator Transfer Evaluation

• Rubric Generator transfer is tested as a RubricBench preference reward model and as test-time guidance measured by Ratio on HealthBench.RubricBench contains 1,147 comparisons across five domains, and the reward-model Accuracy measures agreement with human preference labels.Source: §C.2 · p. 12

D Baseline Details

• Baselines include fixed expert-authored GoldenRubrics, fixed-source RuscaRL scaffolding, and OnlineRubrics, which adds criteria extracted by an external LLM during training.The excerpt identifies the baseline families and their rubric-update distinctions.Source: D Baseline Details · p. 12

E.1 Per-Step Complexity

• EvoRubrics keeps the rollout count constant by halving batch size, while its M×M cross-evaluation requires 16 judge calls per prompt at M=4 and adds a second GRPO adapter update.The adapter-switch term is described as negligible at under 2 seconds; the practical overhead factor is about 1.8×.Source: §E.1 · pp. 12–13

E.2 Empirical Wall-Clock Time

• 78% overhead is below the roughly 3× expected from doubling roles and quadrupling judge calls, with concurrent API calls, small LoRA adapters, and fast adapter switching cited as reasons.The adapter switch is under 2 seconds per swap.Source: §E.2 · p. 13

E.3 Discussion

• EvoRubrics maintains 6.2–7.9 minutes per step, while Golden RL and RuscaRL reach up to 8.7 minutes at steps 400–499; a local judge could reduce the dominant M² judge-call cost.The analysis characterizes the 1.8× time cost as a moderate price for joint rubric-policy optimization.Source: §E.3 · p. 13

F Prompts

• The prompt appendix collects templates for policy generation, rubric generation, judging, alignment and constructiveness rewards, and rubric-as-guidance generation.Source: F Prompts · p. 13

F.2 Rubric Generator Prompt Template

• The Rubric Generator prompt requests 9–11 question-specific criteria in a strict three-line format, with roughly balanced positive and negative points across five named axes.The instructions specify machine-parsable output and define axes including accuracy, completeness, context awareness, communication quality, and instruction following.Source: §F.2 · pp. 13–14

F.3 Judge Model Prompt Template

• The judge prompt evaluates each rubric criterion independently and returns an explanation plus a Boolean decision about whether it is met.Source: §F.3 · pp. 14–15

F.4 Rubric Generator Reward Prompts

• Generator rewards include similarity to golden rubrics and constructiveness, measured by the quality improvement when the Policy LLM revises an initial answer using generated rubrics.Source: §F.4 · p. 15

F.5 Rubric-as-Guidance Prompt Template (Test-Time)

• At test time, generated rubrics are prepended as optional reference guidance, with instructions to prioritize safe, accurate reasoning when rubrics conflict.Source: §F.5 · p. 16

G Case Study

• The case study presents two rubric sets generated during training with Qwen3-4B as the backbone.Source: G Case Study · p. 16
• After training, EvoRubrics distinguished postpartum blues from depression and added management and emergency guidance absent from the base model’s response.Source: G Case Study · p. 16
• For a hand, foot, and mouth disease query, the trained model recommended professional diagnosis, listed warning signs, and promoted safer symptom management.Source: G Case Study · p. 16

I Ethical Considerations

• Experiments focus primarily on English-language medical data; models are research-only, not intended for clinical deployment, and future healthcare use would require clinical validation and regulatory compliance.Source: I Ethical Considerations · p. 16
• 90 (EvoRubrics score) versus -13 (Base model score) in the postpartum emotional changes case.Source: Table 11 · p. 18
• 49.0 (EvoRubrics score) versus -51 (Base model score) in the hand, foot, and mouth disease medication case.Source: Table 12 · p. 18

J Artifacts, Licensing, and Usage

• Third-party artifacts are used under their licenses, terms, or research-access conditions and are not redistributed unless permitted; produced artifacts are also research-only.Source: J Artifacts, Licensing, and Usage · p. 19

K Use of Large Language Models

• LLMs assisted only with language polishing and code debugging; authors reviewed and revised outputs, while independently developing the core research contributions.Source: K Use of Large Language Models · p. 19