PaperlayerView exact revision on arXiv

Source-linked AI summary

EvoLM: Self-Evolving Language Models through Co-Evolved Discriminative Rubrics

Shuyue Stella Li, Rui Xin, Teng Xiao, Yike Wang, Rulin Shao, Zoey Hao, Melanie Sclar, Sewoong Oh, Faeze Brahman, Pang Wei Koh, Yulia Tsvetkov

arXiv:2605.03871v1cs.AI

TL;DR

Post-training reward signals rely on external supervision that can be limited by human capability, proprietary dependencies, or domain-specific verification. EVOLM instead co-evolves a policy with discriminative natural-language rubrics derived from the model’s own outputs, achieving the strongest reported downstream policy while retaining important scope limitations.

  • Problem

    Current post-training methods depend on external rewards, while the evaluative knowledge already encoded in language models remains insufficiently used as a self-improvement signal.

  • Method

    EVOLM alternates policy updates using rubric-conditioned judge scores with rubric-generator updates that optimize preference discrimination, using the model’s outputs and a frozen judge.

  • Results

    69.3% average on the OLMo3-Adapt suite makes EVOLM the strongest downstream policy, ahead of GPT-4.1-prompted rubrics at 66.7%.

  • Takeaways & Limitations

    Co-evolving explicit rubrics with the policy structures pretrained evaluative knowledge into a reward signal without human annotations, proprietary APIs, or task-specific verifiers.

  • Takeaways & Limitations

    EVOLM is validated on general-purpose post-training data; specialized domains such as medicine or law remain open questions, and frozen judges limit rubric complexity.

Abstract

from arXiv · show

Language models encode substantial evaluative knowledge from pretraining, yet current post-training methods rely on external supervision (human annotations, proprietary models, or scalar reward models) to produce reward signals. Each imposes a ceiling. Human judgment cannot supervise capabilities beyond its own, proprietary APIs create dependencies, and verifiable rewards cover only domains with ground-truth answers. Self-improvement from a model's own evaluative capacity is a reward source that scales with the model itself, yet remains largely untapped by current methods. We introduce EVOLM, a post-training method that structures this capacity into explicit discriminative rubrics and uses them as training signal. EVOLM trains two capabilities within a single language model in alternation: (1) a rubric generator producing instance-specific evaluation criteria optimized for discriminative utility, which maximizes a small frozen judge's ability to distinguish preferred from dispreferred responses; and (2) a policy trained using those rubric-conditioned scores as reward. All preference signals are constructed from the policy's own outputs via temporal contrast with earlier checkpoints, requiring no human annotation or external supervision. EVOLM trains a Qwen3-8B model to generate rubrics that outperform GPT-4.1 on RewardBench-2 by 25.7%. The co-trained policy achieves 69.3% average on the OLMo3-Adapt suite, outperforming policies trained with GPT-4.1 prompted rubrics by 3.9% and with the state-of-the-art 8B reward model SkyWork-RM by 16%. Overall, EVOLM demonstrates that structuring a model's evaluative capacity into co-evolving discriminative rubrics enables self-improvement without external supervision.

1 Introduction

EVOLM addresses the limits of externally supervised rewards by co-evolving explicit rubrics and a policy, using the model’s own evaluative capacity and outputs as training signal. The method alternates policy updates with rubric updates that sharpen the judge’s discrimination.

  • Motivation: External human, proprietary-model, scalar, and verifiable rewards constrain post-training supervision in different ways.The passage identifies human-judgment limits, proprietary dependencies, reward hacking, and ground-truth restrictions.
  • Motivation: Rubrics make evaluation criteria explicit, interpretable, and modular while helping a small judge score responses against concrete requirements.Rubric generation specifies what to measure, while judging scores responses against those criteria.
  • EVOLM: EVOLM trains rubrics for discriminative utility, rewarding criteria that widen the judge’s score gap between preferred and dispreferred responses.Preference pairs come from temporal contrast between current policy outputs and earlier-checkpoint outputs, without human-written rubrics or external labels.
  • EVOLM: EVOLM alternates rubric generation and policy optimization so improving policies require sharper rubrics, while sharper rubrics provide more informative reward.The judge remains frozen while rubric and policy capabilities co-evolve.
  • Results: 69.3% average on OLMo3-Adapt is achieved by EVOLM, versus 66.7% for GPT-4.1-prompted rubrics across twelve benchmarks.The reported suite spans math, code, reasoning, knowledge, instruction following, and open-ended generation.

2 EVOLM

EVOLM decomposes reward into natural-language rubric generation and judge scoring, then trains the rubric generator and policy in alternating phases. Its rubric objective rewards criteria that reconstruct preference orderings, using preference pairs derived from policy outputs.

  • Reward decomposition: A rubric generator produces question-specific natural-language criteria, and a judge scores responses against those criteria.This replaces implicit scalar reward criteria with an explicit natural-language interface.
  • Alternating training: The joint model alternates policy updates using rubric-based scores with rubric updates using preference-ranking rewards while keeping the judge fixed.The alternating procedure updates each learner while holding the other component fixed during its phase.
  • Rubrics as latent variables: EVOLM treats the rubric as a latent variable whose likelihood explains observed preference orderings.The latent-rubric formulation generalizes scalar-reward preference modeling by conditioning preferences on natural-language criteria.
  • Rubrics as latent variables: The rubric generator is optimized with variational inference and policy gradient so rubrics receive reward for correctly reconstructing preferences under the fixed judge.Maximizing the ELBO brings the variational rubric distribution toward the posterior over rubrics.
  • Optimization: Practical training combines a margin-based discriminative reward with a format reward, and both policy and rubric generator use GRPO.The format reward checks whether rubrics conform to a judge-parseable structured JSON schema.
  • Preference construction: Preference pairs are generated from policy outputs through temporal contrast, inferred questions, and rubric-conditioned sampling.These sources respectively vary checkpoint age, test question relevance, and test whether rubric-conditioned responses outperform question-only responses.

3 Experiments

EVOLM is evaluated against prompted rubrics, scalar reward models, rubric-based RL methods, and alternative training regimes across downstream tasks, out-of-distribution domains, model architectures, and judges. The results show that co-evolved rubrics provide strong and transferable reward signals, while downstream quality depends more on adaptive rubric utility than static rubric accuracy.

  • Main Comparisons: 69.3% average downstream performance makes EVOLM the strongest policy, exceeding GPT-4.1-prompted rubrics at 66.7% and rubric-based RL methods at 66.7–67.6%.EVOLM achieves the highest downstream average without proprietary APIs, external labels, or task-specific verifiers.
  • Main Comparisons: 69.3% versus 68.3% shows co-evolving training outperforming sequential training, despite lower static RewardBench-2 accuracy of 46.0% versus 47.2%.The comparison indicates that adapting rubrics to the evolving policy distribution matters for downstream policy quality.
  • Main Comparisons: 86.4% RewardBench-2 and 80.8% JudgeBench for the scalar reward model still yielded only 59.7% downstream performance, 9.6 points below EVOLM.The results contrast strong static preference performance with weaker downstream policy quality.
  • Rubrics Evolve from Abstract Labels to Verifiable Checks: 80% of mathematical rubric weight can concentrate on an expected answer, while label-only criteria fall from 21.9% to 0.3% and constraint criteria rise from 7.7% to 20.3%.Trained rubrics shift evaluation toward specific, verifiable checks that reduce the judge’s interpretive burden.
  • Alignment to Expert Deep Research Rubrics: 58.4% on HealthBench and 59.3% on ResearchQA are EVOLM’s highest pairwise-ranking accuracies, exceeding GPT-4.1’s 52.5% and 51.0%.These out-of-distribution results use expert-written rubrics on health and research tasks unseen during rubric-generator training.
  • Cross-Model Transfer: 65.2% versus 64.4% on Qwen3-4B and 46.9% versus 45.7% on Llama-3.1-8B show transfer to policies from different architectures.The frozen rubric generator provides effective reward signals to models not seen during rubric training.
  • Multi-Judge Training: 61.9% RewardBench-2 and 66.1% JudgeBench make MAF stronger than the single-judge generator on JudgeBench for Qwen3-8B, while BA raises OLMo-3-7B-Instruct JudgeBench from 40.2% to 46.2%.Multi-judge training improves transfer to inference-time judges outside the training mix.
  • Ablations: 23% rubric validity without the format reward, versus more than 85% with it, identifies structural validity as the critical ablation requirement.Other choices, including alternation frequency, preference signal, and temporal step gap, keep downstream policies within 1–2 points of one another.

4 Related Work

Existing post-training methods obtain rewards from scalar reward models or prompted language-model judges, which encode evaluation criteria implicitly or through fixed prompts. EVOLM instead learns explicit criteria from preference signals.

  • Scalar reward models and prompted language-model judges are the main existing sources of post-training rewards.These approaches generate scalar scores or ask language models to judge responses.
  • Existing approaches encode evaluation criteria implicitly in model weights or through fixed prompts.
  • EVOLM produces explicit criteria learned from preference signals rather than relying on implicit criteria or fixed prompts.

5 Conclusion

EVOLM co-evolves explicit discriminative rubrics with the policy they score, using the model’s evaluative knowledge to structure reward signals without external supervision. The method reports strong rubric and downstream policy performance and supports transfer across several settings.

  • EVOLM co-evolves explicit discriminative rubrics with the policy they score.
  • The method extracts evaluative knowledge from the model into criteria that a small frozen judge can apply reliably without human annotations, proprietary APIs, or task-specific verifiers.
  • EVOLM outperforms frontier-model rubric baselines on rubric quality and downstream policy performance.
  • The learned evaluation structure transfers across unseen policies, judges, and out-of-distribution domains without retraining.

Limitations

The evaluation covers general-purpose post-training data, while specialized domains and purely subjective criteria remain less characterized. The frozen judge also limits the complexity of criteria the rubric generator can learn to produce.

  • EVOLM has been validated on general-purpose post-training data, leaving domains such as medicine and law open for evaluation.
  • The rubric enrichment mechanism is most clearly observed for tasks with verifiable intermediate steps.
  • The effect of rubric enrichment on purely subjective evaluation criteria is less characterized.
  • Using a frozen judge isolates the rubric generator as the source of training-signal improvement but bounds the complexity of learned criteria.

A.2 Training Hyperparameters

The appendix describes batch, generation, optimization, infrastructure, baseline, and multi-judge configurations used for training and comparison. These settings include structured rubric validation, asynchronous generation, active sampling, and multiple baseline implementations.

  • Training Hyperparameters: Batch and generation settings are documented in Tables 8 and 10, while optimization settings are documented in Table 9.
  • Training Hyperparameters: Generated rubrics must be valid JSON with at least two weighted criteria whose weights approximately sum to 1.0.
  • Training Hyperparameters: Asynchronous training keeps four batches in flight, allowing the trainer to process batch t while generation produces batches t + 1 through t + 4.
  • Training Hyperparameters: Active sampling filters zero-variance prompt groups and continues sampling until 64 prompts have non-zero reward variance.
  • Training Hyperparameters: The infrastructure uses ZeRO Stage 3 with gradient checkpointing, 56 vLLM engines, and one GPU per engine under tensor parallelism.
  • Training Hyperparameters: Baselines include prompted, sequential, scalar, RAR, RRD, RLCER, and Rubric-ARM configurations with distinct reward-source and training procedures.
  • Training Hyperparameters: Multi-judge ablations replace the single Qwen3-1.7B judge with five judge models and compare three aggregation rules.

A.7 Per-Step Compute Accounting

The compute accounting tracks online reward costs by separating rollout, rubric-generation, and reward-side evaluation work. Experiments use fixed sampling and checkpoint-retention settings, while offline costs are excluded.

  • Online compute is reported using P prompt groups, N rollouts per prompt, R rubric items, and J judges in an ensemble.All experiments use P=64 and N=8.
  • Reward accounting separately counts policy generations, rubric generations, and reward-side evaluations.Offline costs, such as pretraining a scalar reward model, are excluded.
  • Table 11 summarizes per-prompt reward computation cost for each baseline method, with N rollout responses and R rubric items as key cost variables.
  • Each experience buffer contains 2,048 experiences sampled uniformly from policy ages 20–100 steps in the past.The age gap avoids both near-duplicate responses and excessively stale samples.
  • Checkpoints are saved every 25 steps, all checkpoints are retained, and weights use a HuggingFace-compatible format.

B Rubric Examples

Rubric evolution shifts evaluation from generic or label-only checks toward fewer, longer, and more specific criteria. Across examples and domains, co-evolving rubrics increasingly encode expected answers, concrete constraints, and verifiable details for a small judge.

  • B Rubric Examples: The trained generator produces 3.0 criteria averaging 112 characters, compared with 5.0 criteria averaging 57 characters for prompted Qwen3-8B.It also embeds specific expected values in 19.3% of criteria versus 6.9% for Qwen3-8B and 7.5% for GPT-4.1.
  • Example 1: Math Problem: In the rectangle problem, 80% of the trained rubric weight targets the expected answer 144 and its perimeter-based derivation, simplifying verification for a 1.7B judge.
  • Example 1: Math Problem: The trained math rubric uses 3 criteria averaging 71 characters, rather than 5 criteria averaging 65 characters, concentrating information into fewer checks.
  • B Rubric Examples: For constrained writing, trained criteria state exact paragraph, keyword, and forbidden-word checks instead of abstract labels requiring judge inference.
  • B.2 Rubric Evolution Over Training: By step 1000, the math rubric links the expected value 144 to its perimeter-of-48 derivation, with the top criterion retaining weight 0.80.
  • B.2 Rubric Evolution Over Training: Emotional-support rubrics evolve from four generic criteria to three information-dense criteria, including examples such as grounding techniques and self-compassion.
  • B.2 Rubric Evolution Over Training: Science rubrics grow from label-only checks into 150-plus-character criteria specifying platelet roles, clotting factors, cascade mechanisms, and medical conditions.
  • B.3 Aggregate Rubric Statistics: Across four domains, V3 mainly enriches roughly three criteria rather than decomposing into many shorter ones, reducing the frozen judge’s interpretive burden.Aggregate statistics show label-only criteria falling from 21.9% for Qwen3-8B to 0.3% in main-prompt co-evolve, while alternate-prompt co-evolve increases criterion count from 4.9 to 11.4.

C Detailed Ablation Results

Ablations show that downstream policy quality depends on the interaction between rubric design, temporal coupling, model configuration, preference construction, judge size, and sampling gaps. Held-out rubric accuracy and downstream training effectiveness frequently diverge.

  • Rubric reward design varies prompt structure, binary versus continuous margin rewards, and format rewards within co-evolving training with temporal contrast and K=50.Format% measures valid rubrics at training step 1000.
  • Without explicit format reward, valid-rubric rates fall from above 85% to 23% for the main prompt and 67% for main prompt plus margin by step 1000.Margin reward slows but does not prevent format degradation.
  • K=50 achieves the best downstream average at 69.3%, while K=2 reaches 67.9% and K=100 reaches 68.7%.Small K limits phase convergence, whereas large K trains on an increasingly stale policy distribution.
  • At K=100, RB2 reaches 48.5% despite lower downstream performance, illustrating a disconnect between held-out discriminative accuracy and training-signal effectiveness.
  • Two-model training improves RB2 by 2.4 points, from 46.0% to 48.4%, while downstream policy quality remains identical at 69.3%.Single-model mode is preferred because it halves memory requirements without downstream cost.
  • Temporal contrast yields the highest downstream average among single signals at 69.3%, whereas combining all three signals raises RB2 to 49.3% but lowers downstream performance to 67.8%.Its downstream advantage is especially visible on HumanEval+ and AlpacaEval.
  • The 1.7B judge produces the best downstream policy at 69.3%, although 14B achieves higher RB2 at 67.6% versus 46.0% for 1.7B.Larger judges improve rubric accuracy but introduce diminishing returns and higher evaluation cost.
  • The moderate temporal gap [20, 100] achieves the best downstream average within a competitive 68.6–69.3% range.It balances contrast strength with distributional similarity between paired responses.

C.7 Cross-Architecture Generalization

Cross-architecture tests show that co-evolving training transfers unevenly across base models. It matches the prompted baseline on OLMo-3-7B and improves aggregate performance on Llama-3.1-8B, with task-specific differences and possible interference concerns.

  • C.7 Cross-Architecture Generalization: On OLMo-3-7B, co-evolving training reaches 45.3% RB2 and 64.0% downstream quality, matching the prompted baseline’s 63.9%.
  • C.7 Cross-Architecture Generalization: On Llama-3.1-8B, co-evolving training improves aggregate performance over the prompted baseline, 43.8% versus 42.2%.The largest gains occur on GSM8K (+9.5) and IFEval (+25.1).
  • C.7 Cross-Architecture Generalization: The Llama comparison is less informative for open-ended generation because the prompted baseline shows mode collapse on AlpacaEval at 0.2%.
  • C.7 Cross-Architecture Generalization: The contrasting architecture results suggest that effectiveness depends on whether a base model can stably serve as both rubric generator and policy.The passage identifies possible architecture-specific tuning or two-model training for interference concerns.

C.8 Multi-Judge Aggregation Strategies

The experiments compare single-judge and multi-judge reward aggregation for downstream policy quality and rubric evaluation. The single-judge baseline performs best overall, while among multi-judge methods, Binary Agreement is strongest downstream.

  • Results: 69.3% downstream policy quality is achieved by the single-judge baseline, outperforming all multi-judge strategies at 66.2–67.7%.Adding judges may introduce calibration mismatches that dilute the discriminative training signal.
  • Aggregation effects: 37.9% RewardBench 2 performance is observed with score averaging, illustrating the weakness of averaging rewards from differently calibrated judges.The passage attributes the degradation to destroyed rank ordering across judges.
  • Experimental setup: Table 22 evaluates rubric quality and downstream policy quality across aggregation strategies using Qwen3-1.7B as the evaluation judge.The compared strategies include score averaging, majority voting, Binary Agreement, Margin+Format, and Margin+Agreement+Format.
  • Multi-judge results: 50.4% JudgeBench accuracy and 67.7% downstream quality are achieved by Binary Agreement, the strongest multi-judge variant.Binary Agreement penalizes judge disagreement through Fleiss’s kappa.
  • Multi-judge results: 46.2% RewardBench 2 performance is achieved by Margin+Format, exceeding the other multi-judge variants on that metric.Margin+Format uses weights (0.7, 0.3, 0).

D Prompts

The appendix specifies the prompts used for rubric generation, judging, policy responses, and preference-pair construction. It also documents prompt variants for reward-shaping ablations and the formatting constraints imposed on generated rubrics and scores.

  • D.1 Main Training Prompts: The main training configuration combines the main rubric-generation prompt with standard policy and judge prompts.This setup is used for the main result and main ablation reruns.
  • D.1 Main Training Prompts: The rubric generator identifies the question’s most important aspect, explicit requirements, and implicit requirements before producing evaluation criteria.The prompts distinguish directly stated constraints from qualities inferred from context.
  • D.1 Main Training Prompts: Rubrics contain 2–5 atomic, objective criteria whose weights sum to exactly 1.0, with a possible dealbreaker criterion for essential requirements.Each criterion includes scoring levels, and the output follows a specified JSON structure.
  • D.1 Main Training Prompts: The judge evaluates each rubric criterion from 0.0 to 1.0, multiplies by its weight, and sums the results into a score between 0.0 and 1.0.The judge output is constrained to valid JSON containing reasoning and a scalar score.
  • D.2 Preference Pair Construction Prompts: Temporal contrast constructs preference pairs by comparing current policy responses with earlier responses retrieved from a stored rollout buffer.Current and earlier responses use the standard policy system prompt.
  • D.2 Preference Pair Construction Prompts: The inferred-question variant reconstructs the intended question from a response, then generates the dispreferred response for that inferred question.The inferred-question prompt requests only a specific reconstructed question.
  • D.2 Preference Pair Construction Prompts: The rubric-conditioned variant generates a preferred response by prepending the generated rubric to the policy prompt, while reward-shaping ablations vary rubric-generation prompts.The ablation keeps the policy and judge prompts fixed while comparing the 5–10-criteria and main prompt variants.
  • D.3 Rubric Prompt Variants Used in Reward-Shaping Ablations: The alternative rubric prompt generates 5–10 atomic, objective criteria with weights summing to 1.0 and scoring levels from 0.0 to 1.0.Its output is constrained to a JSON object containing criteria, weights, and scoring-level descriptions.
Loading 2605.03871v1…