PaperlayerView exact revision on arXiv

Source-linked AI summary

EvoRubric: Self-Evolving Rubric-Driven RL for Open-Ended Generation

Xin Guan, Xiaomeng Hu, Shen Huang, Zhenyi Wang, Bo Zhang, Zijian Li, Pengjun Xie, Bo Liu, Jiuxin Cao

arXiv:2605.29847v2cs.CL

TL;DR

Open-ended generation lacks definitive correctness targets, making rubric construction difficult for rubric-based RL. EvoRubric jointly learns responses and rubrics using validity, discrimination, and peer-agreement feedback. It outperforms matched baselines across five benchmarks, with reported macro-average gains at both model scales.

  • Problem

    Open-ended generation lacks definitive rewards, leaving rubric-based RL with the challenge of learning useful criteria when final-answer correctness is not verifiable.

    Source: §1 · p. 1

  • Method

    EvoRubric uses a shared policy to generate responses and rubrics, combining criterion-validity checks, response discrimination, peer consensus, and persistent rubric memory.

    Source: §1 · p. 2

  • Results

    56.28 at 8B and 61.13 at 14B exceed the strongest matched baselines by 3.09 and 2.06 points, respectively, across five benchmarks.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    Experiments with expert-initialized rubrics show compatibility with human priors, alongside human audits of criterion validity and response quality.

    Source: §1 · p. 2

Abstract

from arXiv · show

Reinforcement Learning (RL) has advanced Large Language Models (LLMs) in verifiable domains, while open-ended generation remains challenging due to the absence of definitive rewards. Rubric-based RL provides explicit evaluation criteria, but learning to construct these criteria remains challenging when final-answer correctness is not verifiable. We propose EvoRubric, a co-evolutionary RL framework that combines criterion-validity feedback, response discrimination, and peer agreement to learn rubrics for open-ended generation. A shared policy acts as both a Reasoner and a Rubric Generator, using its current responses and historical rubrics to discover new evaluation dimensions. To combine adaptive rubric discovery with a stable validity check, a frozen copy of the initial policy serves as the Meta-Verifier, while a frozen Grader scores responses against the retained criteria. Discriminative feedback, Leave-One-Out peer consensus, and a persistent memory pool transform this feedback into complementary rewards that jointly optimize both policy roles, closing the loop between response improvement and rubric discovery. EvoRubric improves over matched static and external evolving-rubric baselines across five benchmarks in Medical, Writing, and Science. Across three training seeds, it achieves five-benchmark averages of 56.28 at 8B and 61.13 at 14B, exceeding the strongest matched baselines by 3.09 and 2.06 points, respectively. Human audits assess criterion validity and response quality, and experiments with expert-initialized rubrics demonstrate compatibility with human priors.

1 INTRODUCTION

Open-ended tasks lack definitive correctness targets, making rubric construction a key challenge for rubric-based RL. EvoRubric jointly learns responses and rubrics, using multiple feedback signals, and improves over matched methods across five benchmarks.

1 INTRODUCTION

  • Open-ended generation lacks definitive, unique answers, while fixed rubrics can miss emerging response properties and correctness-based rubric supervision is unavailable.Rubric-based RL offers explicit criteria, but learning useful criteria without a verifiable final-answer target remains challenging.Source: §1 · pp. 1–2
  • EvoRubric jointly learns response generation and rubric construction, discovering evaluation dimensions from its current responses and carrying selected criteria forward through rubric memory.A frozen initial policy checks criterion validity, while a frozen Grader applies retained criteria; discriminative feedback and peer consensus provide additional rubric-learning signals.Source: §1 · p. 2
  • 56.28 at 8B and 61.13 at 14B exceed the strongest matched baselines by 3.09 and 2.06 points, respectively, across five benchmarks.EvoRubric achieves the highest mean scores among matched methods on all five benchmarks at both scales; human audits and expert-initialized rubric experiments provide complementary evidence.Source: §1 · p. 2
  • The framework constructs rubric-learning feedback without a verifiable final-answer correctness reference.The shared policy performs both response generation and rubric construction, without a separately parameterized rubric generator.Source: §1 · p. 2

2 RELATED WORK

Related work traces the shift from scalar preference rewards toward fine-grained rubrics for open-ended responses. Prior methods differ in how rubrics are created, updated, and supervised.

Open-Ended QA

  • Open-ended QA makes correctness and verifiability harder to assess, and fine-grained rubric benchmarks still rely heavily on human expertise or closed-source models.HealthBench uses expert-curated medical-grade rubrics as checklists.Source: §2 · p. 3

Reinforcement Learning with Rubrics

  • Rubric-based RL uses structured criteria as optimization signals, while rubric construction determines the supervision available to a policy.Earlier RLHF methods used scalar human-preference scores, which the passage says often lack interpretability and suffer from reward hacking and sparse feedback.Source: §2 · p. 3
  • DR-Tulu updates externally generated criteria using current responses and a maintained pool, whereas RLCER jointly optimizes a shared Reasoner and Rubricator with correctness-based supervision.Rubric-ARM instead alternates optimization of a rubric generator and judge against preference labels.Source: §2 · p. 3

3 EVORUBRIC

EvoRubric jointly learns response generation and rubric construction with a shared policy, using frozen verification and grading alongside adaptive feedback and persistent rubric memory. The resulting loop uses responses to discover criteria and criteria to supervise later responses.

3 EVORUBRIC

  • A shared policy performs response and rubric generation; its frozen initial checkpoint verifies criteria, while a frozen Grader produces criterion-level response scores.Rubrics contain positively weighted requirements and negatively weighted constraints, with scores normalized to [−1, 1].Source: §3 · p. 3

3.1 MULTI-ROLE ROLLOUT UNDER A SINGLE POLICY

  • The Reasoner samples responses, then the Generator uses those responses, base rubrics, and retrieved historical criteria to propose dimensions missing from existing criteria.The method separately standardizes Reasoner and Generator advantages before combining rollouts for a shared clipped GRPO update.Source: §3 · p. 3, §3.1 · p. 4

3.2.1 MOTIVATION

  • Rubric learning jointly checks criterion validity, response separation, and agreement across rubric batches because open-ended tasks lack a verifiable correctness target.The Meta-Verifier screens criteria, the Grader scores responses, discriminative feedback rewards response-separating criteria, and Leave-One-Out consensus measures agreement across nonconstant batches.Source: §3.2.1 · p. 4, §3.2.2 · pp. 4–5

3.3 DYNAMIC REWARD SHAPING & CO-EVOLUTION

  • Generator rewards combine validity, discriminative value, and peer agreement; the Memory Pool admits the highest-consensus eligible batch and retires saturated criteria.New entries are checked against the retained pool, with oldest entries evicted when capacity is exceeded.Source: §3.3 · p. 5
  • Reasoner rewards combine base-rubric and evolving-pool scores with a length penalty, while separate role advantages update the shared policy.Memory updates retain selected criteria and retire saturated ones; subsequent rollouts use both the updated policy and revised memory while the base rubrics, Meta-Verifier, and Grader stay fixed.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

Across five benchmarks, EvoRubric outperforms matched methods at both model scales. Experiments examine its training setup, component contributions, initialization, and rubric and response evaluations.

4.1 EXPERIMENTAL SETUP

  • The study trains separate Qwen3-8B and Qwen3-14B policies on 3,000 queries per domain and evaluates them on five corresponding benchmarks.The three training domains are Medical, Writing, and Science; evaluations use benchmark protocols with Claude Sonnet 4.6 as the default judge.Source: §4.1 · p. 6

4.2 OVERALL PERFORMANCE

  • 56.28 at 8B and 61.13 at 14B are the macro-average scores, exceeding the strongest matched baselines by 3.09 and 2.06 points, respectively.EvoRubric has the highest mean among matched methods on all five benchmarks at both scales, with consistent mean improvements across Medical, Writing, and Science under an independent evaluator.Source: §4.2 · p. 7

4.3 COMPONENT AND GENERATOR ANALYSIS

  • Removing the Generator loss lowers HealthBench and LLMEval-Med by 3.55 and 3.82 points, while removing peer consensus or memory lowers HealthBench by 4.33 or 2.87 points, respectively.Every other reported ablation also lowers both medical scores; removing meta-verification reduces LLMEval-Med to 47.81, below the base model’s 49.25.Source: §4.3 · p. 7, Table 2 · p. 7
  • Against the frozen same-scale generator, GPT-4.1 improves HealthBench/LLMEval-Med by 2.72/1.19 points, while EvoRubric gains a further 2.88/0.83 points.The GPT-4.1 variant follows DR-Tulu’s external rubric-generation strategy in the Medical setting.Source: §4.3 · p. 7, Table 3 · p. 7

4.4.1 TRAINING DYNAMICS

  • With fixed base rubrics, EvoRubric reaches higher scores than static and external evolving-rubric baselines, with its advantage growing in later training stages.The curves begin at similar score levels; the fixed rubrics provide a stable evaluation target while the Memory Pool evolves.Source: §4.4.1 · p. 8, Figure 3 · p. 8

4.4.2 EXPERT-INITIALIZED RUBRICS

  • 44.64 on HealthBench with expert-initialized rubrics is 2.81 points above Static Rubric-RL’s expert-initialized score of 41.83.The model-initialized EvoRubric score is 45.13, 0.49 points above expert initialization in this experiment.Source: §4.4.2 · p. 8, Table 4 · p. 8

4.5 RUBRIC AND HUMAN EVALUATION

  • Effective diversity falls from 89.19% to 75.16% at 8B and from 91.62% to 76.26% at 14B between the first and fourth epoch-end snapshots; final pools retain 118 and 106 distinct human-valid dimensions.The audit also finds that meta-verification reduces the overall invalid-criterion rate from 39.06% to 5.10%.Source: §4.5 · pp. 8–9, Table 5 · p. 9
  • Blind response comparisons favor EvoRubric in overall quality against both external-generator baselines, while a fixed-response rubric comparison favors EvoRubric 72% to 28%.Human response preferences are strongest for conciseness and relevance; completeness favors the baselines more often, factual-correctness outcomes are close, and safety comparisons are mostly ties.Source: §4.5 · p. 9, Table 6 · p. 9

5 CONCLUSION

EvoRubric jointly learns response generation and rubric construction without a verifiable final-answer correctness target. Across five benchmarks, it outperforms matched baselines, with audits and initialization experiments providing complementary evidence.

5 CONCLUSION

  • EvoRubric constructs rubric-learning supervision for open-ended generation without a verifiable final-answer correctness target.A shared policy discovers evaluation dimensions from current responses and improves through the supervision those dimensions provide.Source: §5 · p. 10
  • Across five benchmarks, EvoRubric exceeds the strongest matched baselines by 3.09 points at 8B and 2.06 points at 14B across three training seeds.Rubric audits, response comparisons, and expert-initialization experiments provide complementary evidence for the approach.Source: §5 · p. 10

A COMPARISON WITH RELATED METHODS

EvoRubric differs from prior rubric-learning approaches in how criteria are updated and optimized with the response policy. It learns rubric construction within that policy and uses validity, discrimination, and agreement signals.

A COMPARISON WITH RELATED METHODS

  • Unlike static rubric methods, EvoRubric augments persistent base rubrics with criteria discovered from current policy responses, carrying selected criteria across iterations.The shared policy learns to generate both responses and new criteria.Source: A COMPARISON WITH RELATED METHODS · p. 14
  • Unlike DR-Tulu’s external generator, EvoRubric learns criterion construction within the response policy and adds validity and peer-agreement feedback.In the Medical comparison, EvoRubric gains 2.88/0.83 points on HealthBench/LLMEval-Med over the GPT-4.1-generator variant.Source: A COMPARISON WITH RELATED METHODS · p. 14
  • Unlike RLCER, which uses association with final-answer correctness to select rubrics, EvoRubric uses frozen verification, discrimination, and LOO consensus without that reference.Consensus measures agreement across rubric batches but does not itself establish correctness.Source: A COMPARISON WITH RELATED METHODS · p. 14
  • Unlike Rubric-ARM, EvoRubric updates the Generator and Reasoner while keeping the Grader fixed; the approaches also differ in their supervision sources.Rubric-ARM alternately trains the rubric generator and judge using preference supervision.Source: A COMPARISON WITH RELATED METHODS · p. 14

B TRAINING LOOP AND JOINT GRPO OBJECTIVE

The training loop retrieves historical criteria before updating memory, then samples outputs for both policy roles. Role-specific advantages feed one shared clipped-objective update.

B TRAINING LOOP AND JOINT GRPO OBJECTIVE

  • Historical criteria for current-response scoring are retrieved before the memory update, and selected additions are carried forward to the next visit to that query.This ordering keeps current scoring based on previously retained criteria before adding new ones.Source: B TRAINING LOOP AND JOINT GRPO OBJECTIVE · p. 14
  • For each query, the rollout policy samples N Reasoner outputs and M Generator outputs, with advantages standardized separately within each query–role group.The combined batch contains role-specific inputs and sampled outputs.Source: B TRAINING LOOP AND JOINT GRPO OBJECTIVE · p. 14
  • Both roles contribute to one shared policy update using the clipped objective, but their rewards are not standardized together.A numerical stabilizer gives zero advantages for groups with constant rewards.Source: B TRAINING LOOP AND JOINT GRPO OBJECTIVE · p. 14

C REWARD COMPUTATION

EvoRubric computes complementary rewards for rubric generation and response quality from validity, discrimination, consensus, and retained criteria. Its training loop filters candidate rubrics, scores responses, and updates shared policy roles using separate feedback.

C REWARD COMPUTATION

  • The training loop grades responses against base, historical, and verified candidate criteria, then computes role-specific rewards and updates the shared policy.Empty or fully rejected batches add no criteria; eligible batches require a nonconstant response-score vector and LOO correlation above 0.2.Source: C REWARD COMPUTATION · pp. 15–16
  • Reasoner scores combine fixed base-rubric scores with historical and current eligible-batch feedback, minus a length penalty.Current supervision averages all eligible batches, while only the highest-consensus eligible batch contributes new memory entries.Source: C REWARD COMPUTATION · pp. 15–16
  • The Generator’s validity reward is the fraction of valid criteria among proposals, with zero assigned to an empty batch.Only valid criteria contribute to subsequent response scoring.Source: C REWARD COMPUTATION · p. 15
  • Peer consensus uses nonconstant candidate-batch score vectors, while discriminative feedback measures the fraction of verified criteria that split responses.Both positive feedback components are scaled by the validity fraction; consensus is zero when the other batches or their mean vector are constant or unavailable.Source: C REWARD COMPUTATION · p. 15

D MEMORY POOL AND INITIALIZATION DETAILS

The appendix describes bounded per-query rubric memory, fixed base rubrics, training configurations, and evidence that response-conditioned rubrics can extend fixed criteria. A controlled comparison favors EvoRubric’s generated rubric sets.

D MEMORY POOL AND INITIALIZATION DETAILS

  • Only the eligible candidate batch with the highest LOO consensus can add criteria to memory, which retains at most 10 entries and evicts the oldest when full.Saturated criteria are removed, exact text duplicates are checked, and historical criteria that do not distinguish current responses are excluded from the current pool score.Source: D MEMORY POOL AND INITIALIZATION DETAILS · p. 16
  • Fixed base rubrics are generated once with few-shot prompting from the frozen pre-RL Qwen3 model and remain fixed during RL.Per-prompt base-rubric counts range from 2 to 8; six criteria are most common at 38.96%.Source: D MEMORY POOL AND INITIALIZATION DETAILS · p. 16

E HYPERPARAMETERS FOR REINFORCEMENT LEARNING

F QUALITATIVE CASE STUDY: RUBRIC EVOLUTION

  • For an insulin-storage query, response differences motivate added criteria on formulation-specific duration, handling, device instructions, and source verification beyond general fixed rubrics.The example illustrates how response-conditioned rubric generation extends fixed base criteria.Source: F QUALITATIVE CASE STUDY: RUBRIC… · p. 16, Table 8 · p. 17

G CONTROLLED RUBRIC QUALITY COMPARISON

  • In a blind pairwise comparison on identical query-response groups, the evaluator preferred EvoRubric’s rubric set in 72% of cases and the external baseline in 28%.The comparison used 100 randomly sampled HealthBench queries and a same-scale External Evolving-RL generator.Source: G CONTROLLED RUBRIC QUALITY COMPARISON · p. 17

H EFFECT OF EXTERNAL GENERATOR CAPACITY

The GPT-4.1-generator variant’s reported API-use estimate covers rubric generation.

H EFFECT OF EXTERNAL GENERATOR CAPACITY

  • GPT-4.1 rubric generation averaged 81.20 million input tokens, 2.91 million output tokens, and 84.11 million total tokens per run, costing approximately USD 185.70 per run.The estimate covers generator API usage at the experiment’s pricing.Source: H EFFECT OF EXTERNAL GENERATOR CAPACITY · p. 17

I HUMAN EVALUATION AND RUBRIC DYNAMICS

Audits examine rubric validity, retained-dimension diversity, and blind response comparisons, alongside rubric-generation prompt guidance. Results show residual validity errors and substantial nonredundant pool coverage, while response-quality comparisons reveal dimension-specific trade-offs.

I.1 LONGITUDINAL RUBRIC AUDIT

  • At epoch 4, verification yielded 8/157 invalid criteria versus 75/192 without verification at 8B, an 86.96% relative reduction.Residual errors increased over training even with verification; final valid-criterion rates were 94.90% (8B), 95.68% (14B), and 60.94% (8B without verification).Source: §I.1 · p. 18
  • Factual conflicts remained the largest residual error category with verification; unmeasurable, trivial, and length-hallucination errors were absent from the audited final verified pools.Without verification, all six audited error categories occurred.Source: §I.1 · p. 19

I.2 EFFECTIVE DIVERSITY OF THE MEMORY POOL

  • Distinct human-valid dimensions beyond base rubrics made up 75.16% of the 8B pool and 76.26% of the 14B pool at epoch 4.Effective diversity declined across snapshots, with 14B slightly higher at each epoch; semantically equivalent eligible entries count once per prompt in the numerator, while all pool entries remain in the denominator.Source: §I.2 · p. 19

I.3 BLIND EVALUATION OF FINAL RESPONSES

  • Overall-quality wins exceeded losses against both baselines, with non-tie win rates of 60.0% and 55.8%.The largest advantages were in conciseness and relevance; completeness favored baselines more often, factual-correctness outcomes were close, and safety comparisons were mostly ties.Source: §I.3 · p. 19

2. Radical Non-Redundancy & Sparsity (CRITICAL)

  • The rubric-generation prompt requires checking whether existing rubrics already cover a quality or error and forbids duplicating criteria in meaning or scope.It instructs the Generator to return an empty list when existing rubrics fully cover key response aspects.Source: §2 · p. 20

3. Style Alignment & Formatting

  • Generated criteria must match existing rubrics’ style and be condition-based, with positive rubrics describing excellence and negative rubrics describing active failures.Negative rubrics should describe the active flaw rather than say “Fails to do X.”Source: §3 · p. 20

4. Actionable & Objective

  • Criteria must describe observable, verifiable response qualities or errors and provide clear instructions for human raters and future reward-model training.The workflow checks whether a significant strength or fatal flaw escapes the existing criteria.Source: §4 · p. 20

Related papers

Loading 2605.29847v2…