PaperlayerView exact revision on arXiv

Source-linked AI summary

EvoRS: On-Policy Self-Evolution of Reward Systems for Open-Ended Reinforcement Learning

Weiyuan Li, Aili Chen, Xintao Wang, Yikai Zhang, Qingqing Dong, Jinghan Xu, Hongru Hou, Wenxuan Zhao, Chengkun Lang, Jun Gao, Yuanli Guo, Hongcheng Guo, Yanghua Xiao, Deqing Yang

arXiv:2609.12459v1cs.AI

TL;DR

Open-ended RL relies on rubric-based rewards, but policy adaptation can make the complete reward system unreliable, while dynamic-rubric methods mainly adapt criteria. EVORS evolves an executable Reward-DAG using on-policy experience and comparative feedback; on writing and roleplay, it achieves the best final benchmark quality under all three judges while improving reward reliability.

  • Problem

    Rubric-based reward systems for open-ended tasks may become less effective during training, and adapting criteria alone leaves scoring mechanisms and signal composition unchanged.

    Source: §1 · pp. 1–2

  • Method

    EVORS represents the complete reward system as an executable Reward-DAG and uses on-policy rollouts and reward traces to generate candidate states selected through comparative feedback.

    Source: §1 · p. 2

  • Results

    EVORS achieves the best final benchmark quality under all three judges on writing and roleplay, reduces reward-hacking and coverage-failure rates, and preserves reward informativeness.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A comprehensive reward system specified before training underperforms one that evolves with the policy, supporting reward-system evolution throughout training.

    Source: §7 · p. 9

    The evaluation focuses on writing and roleplay; extension to tool-using environments still requires validation.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Open-ended reinforcement learning often relies on rubric-based rewards for tasks without directly verifiable answers. Yet the policy and reward system form a dynamic feedback loop: as the policy optimizes the current reward, an initially useful reward system may become unreliable due to reward hacking or reduced response discriminability. The reward system should therefore evolve rather than remain fixed during training. Existing dynamic-rubric methods adapt evaluation criteria, but reward failures can also arise from scoring mechanisms or signal composition. We introduce EvoRS, a self-evolving RL framework that evolves the reward system from on-policy experience, representing it as an executable Reward-DAG. Specifically, an agentic designer updates this system from on-policy rollouts and reward traces to maintain train-time reliability. Across writing and roleplay, EvoRS achieves the best quality under all three judges, outperforming the policy by \(2.107\) and \(4.767\) points, respectively, while reducing reward hacking and coverage failures and preserving reward informativeness. Ablations confirm that a comprehensive fixed reward system cannot remain reliable in open-ended tasks and must evolve throughout training.

1 Introduction

In open-ended RL, reward reliability can deteriorate as the policy adapts, through reward hacking, insufficient coverage, or diminishing informativeness. EVORS evolves the complete reward system from on-policy evidence, and improves final quality and reliability on writing and roleplay.

1 Introduction

  • Reward reliability can deteriorate as the policy adapts: validity, coverage, and informativeness may fail during training, so the reward system cannot be treated as fixed.Failures include reward hacking, emerging on-policy responses missing from the system, and rewards becoming less useful or separable.Source: §1 · p. 2
  • Dynamic-rubric methods adapt criteria, but scoring mechanisms and signal composition may remain fixed, so reliability depends on evolving the complete reward system.For example, adding a grounding criterion may still leave a hacked response highly scored under continuous scoring.Source: §1 · p. 2
  • EVORS achieves the best final benchmark quality under all three judges on writing and roleplay, while reducing reward-hacking and coverage-failure rates and preserving reward informativeness.It also improves all four roleplay dimensions; component-control ablations find gains depend on evolving the reward system alongside the policy throughout training.Source: §1 · p. 2
  • EVORS represents the complete reward system as an executable Reward-DAG and generates bounded candidate states from on-policy evidence, with comparative feedback selecting the next state.Candidates may revise criteria, scoring mechanisms, or signal composition.Source: §1 · p. 2

2 Related Work

Prior work adapts rubric criteria and studies reward reliability under policy optimization, while self-evolving systems modify other parts of the learning loop. EVORS applies experience-guided evolution to the complete reward system.

2.1 Rubric-Based Rewards and Dynamic Rubrics

  • Dynamic-rubric methods adapt criteria, but generally leave scoring and composition procedures unchanged, placing broader reward-signal construction outside their scope.The literature includes continuous scores, checklist verification, penalties, hybrid designs, and pairwise comparisons.Source: §2.1 · pp. 2–3

2.2 Reward Reliability under Policy Optimization

  • Under policy optimization, evaluators can be exploited, fixed criteria can miss emerging responses, and accuracy on fixed distributions does not ensure informative distinctions among on-policy responses.Reward reliability must therefore be evaluated on policy-generated responses, especially within the same query.Source: §2.2 · p. 3

2.3 Self-Evolving Training and Agent Systems

  • Other self-evolving systems adapt training data, critics, skills, harnesses, or environments; EVORS applies experience-guided targeted changes to the complete reward system supplying the RL signal.These approaches use experience to identify deficiencies and guide validated changes.Source: §2.3 · p. 3

3 Motivating Analysis

The analysis argues that fixed reward systems can lose discriminability as policy responses change, and that criteria, scoring, and signal composition all shape the reward signal. On-policy diagnosis and comparative validation guide targeted updates.

3 Motivating Analysis

  • A reward system that discriminates useful responses earlier may later lose reward variance and sample-level separability as the policy changes.Figure 2 illustrates later responses concentrating in a narrower score region.Source: §3 · p. 3, Figure 2 · p. 4
  • Rubrics do not determine rewards by themselves: scoring mechanisms and signal composition shape how evaluation content becomes the policy’s training signal.A checklist can ensure requirements are met while continuous scoring distinguishes quality among responses that pass those checks.Source: §3 · p. 3, Table 1 · p. 4
  • Low within-query variance can reflect generic criteria or scorer saturation, which call for different repairs; ambiguous evidence leaves the current system unchanged.Diagnosis should use on-policy rollouts and reward traces rather than map one failure indicator directly to an edit.Source: §3 · p. 4
  • Candidate updates are compared with the current system on the same rollout cases to test whether they repair the target failure while preserving useful reward behavior.Comparative feedback selects the state carried into subsequent training and informs later updates.Source: §3 · p. 4

4 Method

EVORS couples policy learning to reward-system evolution: on-policy responses and reward traces inform diagnosis, candidate generation, and matched-replay selection. Its Reward-DAG makes criteria, scoring, and signal composition part of one evolvable state.

4 Method

  • EVORS couples policy learning with periodic reward-system checks, using current-policy responses and reward traces to decide whether to retain or evolve the active state.When an update is needed, it diagnoses evidence, proposes alternatives, and compares them through matched replay.Source: §4 · p. 4

4.1 The Reward System as an Evolving State

  • The executable Reward-DAG represents reward nodes, their dependencies and execution order, criteria and scoring, and rules for combining outputs in one state.Its composition operators aggregate, normalize, gate, or route existing signals rather than reevaluate responses.Source: §4.1 · pp. 4–5

4.2 From On-Policy Experience to Candidate States

  • An agentic designer diagnoses failures from on-policy responses and reward traces, then generates bounded, targeted candidate states using accumulated run-local memory and dynamic skills.If evidence does not support an update, the active state remains unchanged.Source: §4.2 · pp. 5–6

4.3 Self-Validation and Evolutionary State Transition

  • Matched replay applies the current and candidate states to the same recent rollout cases, selecting a candidate that repairs failures while preserving clean-case rewards and within-query distinctions.If no improvement is supported, the current state is retained; selected, rejected, and no-op trajectories inform future search.Source: §4.3 · p. 6

5 Experiments and Results

EVORS is evaluated on writing and roleplay against fixed and adaptive reward-system baselines, using multiple judges and measures of final quality and reward reliability. It achieves the largest WritingBench gain, improves CoSER Overall and all four roleplay dimensions, and is the only method to reduce writing HR below the base level.

5.1 Experimental Setup

  • The evaluation uses 944 filtered synthetic writing instructions and 5,000 MiniMax Role-Play Bench queries, with multiple strong judges on both tasks.All methods share the GRPO pipeline, training data, rollout budget, evaluation protocol, and Qwen3.5-27B reward model; EVORS evolves every five RL steps.Source: §5.1 · p. 6

5.2 Main Results

  • EVORS gains 2.107 WritingBench points and 4.767 CoSER Overall points over the base policy, ranking first across all four roleplay dimensions.Table 2 averages scores from GPT-5.6-Terra, DeepSeek-V4-Pro, and GLM-5.2; CoSER Overall aggregates four roleplay dimensions.Source: Table 2 · p. 7, §5.2 · p. 7

5.1 Experimental Setup

  • Table 2 reports final policy quality alongside HR, CFR, and R-Var measures of reward reliability.HR and CFR assess validity and coverage; R-Var measures within-query reward separation.Source: Table 2 · p. 7

5.2 Main Results

  • On WritingBench, EVORS is the only method to reduce HR below the base level; fixed or partially adapted methods show higher HR and no CoSER Overall improvement over base.RaR improves SC and CF but reduces AN and SQ, illustrating uneven roleplay-dimension gains.Source: §5.2 · p. 7, Table 2 · p. 7
  • Figure 5 tracks training-time reward dynamics, showing reduced writing HR and roleplay CFR after selected EVORS transitions, alongside preserved distinctions and stronger preference agreement.The preference comparison uses blind rankings.Source: Figure 5 · p. 7

6 Analysis

EVORS maintains lower reward-hacking and coverage-failure rates as the policy changes, while preserving meaningful within-query reward distinctions. Its advantage also holds across tested policy and reward models, and ablations favor evolving the full reward system over freezing it or restricting updates to criteria.

6.1 Reward-System Evolution under Policy Shift

  • EVORS remains lower on writing reward-hacking patterns and roleplay CFR, with further reductions after reward-system transitions; audits were hidden from the designer.The trajectories therefore reflect failures discovered from on-policy experience rather than direct optimization of the reported audit metrics.Source: §6.1 · pp. 7–8
  • Across 480 audit groups, EVORS maintains a low Zero-distinction rate and stronger agreement with blind same-query preference rankings across most checkpoints.The distinctions remain aligned with independent response preferences rather than merely producing more dispersed scores.Source: §6.1 · p. 8

6.2 Generality across Policy and Reward Models

  • With Qwen3-8B and either Qwen3.5-27B or RewardAnything-8B as GRM, EVORS outperforms RaR and improves over the base policy.Table 3 reports CoSER Overall averaged across the same three judges as Table 2.Source: §6.2 · p. 8, Table 3 · p. 8

6.3 Ablation Studies

  • Both criteria-only adaptation and a fully discovered Reward-DAG frozen from the start underperform full reward-system evolution, especially on CoSER.Adapting criteria alone cannot address failures in scoring mechanisms or signal composition; a comprehensive fixed system does not replace the evolution path.Source: §6.3 · p. 8
  • Removing candidate selection, update rejection, or evolution memory reduces final quality, with larger effects in multi-turn roleplay.Some variants retain high SC or CF while sharply reducing AN and SQ, indicating less balanced optimization.Source: §6.3 · p. 8

7 Conclusion

EVORS evolves an executable Reward-DAG using on-policy reward traces to diagnose failures and search bounded candidate states. Matched-rollout comparative feedback selects the next active state, and the experiments find that a comprehensive reward system specified before training underperforms one that evolves with the policy.

7 Conclusion

  • EVORS represents the active reward system as an executable Reward-DAG and diagnoses failures from on-policy reward traces to search bounded candidate states.This makes reward-system evolution part of open-ended RL training.Source: §7 · p. 8
  • Comparative feedback on matched rollout cases determines the next active reward-system state, closing the loop between policy learning and reward evolution.Source: §7 · p. 8
  • A comprehensive reward system specified before training underperforms one that evolves with the policy; dynamic-rubric methods leave scoring and signal composition unchanged.The paper frames EVORS as a broader self-evolution approach that searches across the full reward-system state.Source: §7 · pp. 8–9

Limitations

The evaluation establishes reward-system self-evolution on writing and roleplay, with limits in domain coverage, update scheduling, and training cost.

Limitations

  • Evaluation covers writing and roleplay; extending EVORS to tool-using environments remains unvalidated.The Reward-DAG could in principle incorporate tool state, interaction history, and delayed outcomes.Source: Limitations · p. 9
  • Evolution runs every N RL steps to simplify design and control computational cost; failure-conditioned scheduling is left for future study.Source: Limitations · p. 9
  • Periodic diagnosis and candidate evaluation add training cost, which adaptive scheduling or batched evaluation may reduce.Source: Limitations · p. 9

A Appendix Roadmap and Robustness Checks

The appendix roadmap points readers to robustness and implementation materials, while the reward-mechanism taxonomy distinguishes scoring protocols and output signals from criterion-source adaptation.

A Appendix Roadmap and Robustness Checks

B.1 Reward Mechanism Taxonomy

  • Reward mechanisms specify how judgments over criteria become optimization signals, distinct from criterion sources and composition operators.The taxonomy groups mechanisms by scoring protocol and output signal, not by whether a paper calls them a rubric or reward.Source: §B.1 · p. 13, Table 6 · p. 15
  • Continuous rubric scores provide dense, scalarizable signals but may saturate; checklist scoring exposes interpretable item-level satisfaction and pass rates.Source: §B.1 · p. 13
  • Deduction scoring subtracts penalties for detected flaws, producing a downward correction targeted at sparse failures.Violation severity and its corresponding weight determine the penalty.Source: §B.1 · p. 13
  • Pairwise judgments compare two responses, while listwise mechanisms rank candidate sets; they offer relative discrimination but carry order, cost, or set-composition risks.Source: §B.1 · p. 14
  • Claim-level factuality scoring extracts response claims or probes and judges their support against source documents, dialogue, retrieved evidence, or citations.Source: §B.1 · p. 14

B.2 Additional Motivating Results

Supplementary analyses show that criterion specialization and scoring mechanisms reshape local reward distinctions, while different mechanisms shift differently as policy responses change.

B.2 Additional Motivating Results

  • Query-level criteria lowered global mean score from 0.687 to 0.678 while raising same-query range from 0.064 to 0.093 and same-query standard deviation from 0.026 to 0.037.Paired score correlations remained high: Pearson 0.825 and Spearman 0.823.Source: §B.2 · p. 16
  • Flaw-penalty scores fell to 0.423 on fabricated-authority cases and 0.513 on scaffold leakage, while continuous scores remained 0.709 on fabricated-authority cases.Checklist scores dropped to 0.717 on explicit incompleteness but remained 0.958 and 0.942 on fabricated-authority and scaffold-leakage cases.Source: §B.2 · p. 16
  • Checklist same-query standard deviation fell from 0.100 to 0.050, while its zero-variance group rate rose from 0.188 to 0.531 between steps 0 and 120.Under RaR, continuous scoring’s same-query standard deviation rose from 0.024 to 0.038 and its zero-variance group rate fell from 0.500 to 0.344.Source: §B.2 · p. 16

C.1 Implementation Details for Reward Evolution

This section documents EvoRS’s reward-system scoring and evolution procedures, including how candidate Reward-DAG edits are constrained, evaluated, and selected during training. It also describes the models, controls, data, rubrics, baselines, runtime-cost reporting, and research-artifact practices used in the implementation.

C.1 Implementation Details for Reward Evolution

  • Main GRPO runs use Qwen3-4B (Yang et al., 2025), Qwen3.5-27B for local GRM/rubric updating, GPT-5.4 for design, and GPT-5.6-Terra, DeepSeek-V4-Pro, and GLM-5.2 for evaluation.Methods share the same policy initialization, data, rollout budget, batching, optimizer settings, and checkpoint schedule, with evolution thresholds and update cadences fixed before comparison.Source: §C.1 · pp. 16, 18
  • The local Qwen3.5-27B scores each query-response against the active rubric or Reward-DAG node, normalizing continuous item scores, checklist pass rates, and flaw penalties into rewards in [0, 1].Query-level criteria are instantiated under a fixed generation policy and do not change the active reward-system state.Source: §C.1 · p. 18
  • EVORS runs the designer every 5 GRPO steps at 242.8s average evolution-step time; training-only step time subtracts designer elapsed time on update steps.Online reward scoring is included in RL GPU-hour accounting, but offline benchmark scoring, audits, blind ranking annotations, and multi-judge evaluation are excluded.Source: §C.1 · p. 18, Table 8 · p. 19
  • The Codex SDK designer uses GPT-5.4 to inspect rollout summaries, reward traces, diagnostics, the current Reward-DAG, and memory, and can edit rewards only through a typed interface.Codex is not used as a response-level judge; fixed and dynamic skills guide candidate search.Source: §C.1 · p. 19
  • The cited text names On-policy Reward Validity and Sustainable Reward Informativeness, says the current DAG remains pointwise, and states that each rubric node sees one sample, not other real responses.Source: §C.1 · p. 20
  • Every 5 RL steps, the designer diagnoses recent trajectories and reward traces, evaluates bounded candidate Reward-DAG states on matched rollouts, and keeps the current state if none improves reliably.Candidate directions address validity and coverage failures separately from reward informativeness.Source: §C.1 · pp. 21–22

C.2 Reward-DAG Design Details

  • In pointwise RL, the query-response input is processed by nodes sequentially, which write variables, node-level scores, and trace details to context; training reward is the terminal Rubric Node or Composition Operator final_score.Typed edits commit only after candidate validation succeeds; pointwise legality checks require mode=pointwise and a valid terminal final node.Source: §C.2 · pp. 22, 24
  • On 48 recent cases from 12 query groups, the selected writing update achieved execution success rate 1.0 and increased score std from 0.0673 to 0.0730 and score range from 0.2666 to 0.2916.The update tightened rubric criteria without changing graph topology; it addressed weak same-query separation and template-like artifacts.Source: §C.2 · p. 25

C.3 EVORS Training Algorithm

  • At each training step, EVORS scores grouped responses with the current Reward-DAG, updates the policy using GRPO-style group-relative advantages, and periodically diagnoses failures to select supported typed updates.The algorithm stores selected, rejected, and no-op trajectories and distills their comparative outcomes into dynamic skills.Source: §C.3 · p. 25

C.4 Agentic Designer Tool Details

  • The designer uses restricted diagnostic tools to inspect rollouts, reward traces, and candidate behavior, while typed editing tools define the admissible Reward-DAG changes.Diagnostics connect aggregate patterns to concrete cases and organize evidence around validity, coverage, and informativeness.Source: §C.4 · p. 25, Table 10 · p. 26
  • Reward-DAG edits are applied transactionally and rejected if they violate graph validity checks; candidates are evaluated on recent matched rollout cases, with no transition absent a supported advantage.Checks include acyclicity, reachability, valid aggregation inputs, pointwise reward semantics, and scenario/domain consistency. Comparative review also checks preservation of clean cases and useful reward distinctions.Source: §C.4 · p. 26

C.5 Reward-Evolution Guidance Skill Details

  • The designer diagnoses reward validity, coverage, and informativeness, routes failures to bounded edit families, compares candidates on matched cases, and keeps the current graph when none improves.Fixed guidance protects objective continuity and stable reward components; run-local dynamic skills record scenario-specific failures, candidate outcomes, and side effects for later updates. The harness defines admissible actions and checks, while the language-model agent supplies diagnosis and candidate judgments.Source: §C.5 · pp. 27–28

D.1 Artifact Use, Intended Use, and Data Safety

  • The work uses existing models and benchmarks for reward-system research, and its created artifacts are intended for research reproduction and analysis rather than general-purpose safety filtering or moderation.Writing prompts are synthetic and filtered; roleplay prompts require explicit persona, relationship, context, and dialogue state, while outputs are used for aggregate evaluation and audit statistics.Source: §D.1 · pp. 29–30

D.2 Use of AI Assistants in Research and Writing

  • AI assistants supported language polishing and code development; the authors reviewed and remain responsible for the research design, experiments, analyses, claims, and manuscript.Source: §D.2 · p. 30

D.3 Training Data and Initial Rubric Construction

  • 944 writing queries and 5,000 roleplay queries form the training sets; initial rubrics are generated, audited, and revised to yield stable, nonredundant, task-relevant distinctions.Task-level rubrics initialize EVORS's single-node continuous Reward-DAG and serve the RLAIF-style baseline, while query-level rubrics serve RaR-style reward systems.Source: §D.3 · p. 30

D.4 Baseline Reward-System Details

  • Static baselines keep task-level or query-level rubrics fixed, RLER augments task rubrics after rollouts, and OpenRS uses pairwise comparisons without experience-driven candidate evolution.RLER follows the evolving-rubric design of DR-Tulu (Shao et al., 2025); OpenRS converts pairwise comparison margins into scalar rewards in [0, 1].Source: §D.4 · pp. 30–31

D.5 Evaluation Diagnostics and HR/CFR Audits

The audits measure final-output reward hacking and hard coverage failures, while R-Var and Zero diagnose whether rewards retain useful within-query distinctions. Frozen, task-specific taxonomies and independent validation support interpretation of these diagnostics.

HR and CFR metrics

  • HR audits predefined reward-attractive shortcut patterns, while CFR measures outputs with at least one predefined hard coverage failure.HR is normalized per assistant turn for roleplay; CFR is an output-level rate, and an output may receive neither, either, or both labels.Source: §D.5 · p. 31

Audit protocol

  • The HR/CFR taxonomies are frozen before method comparisons, withheld from the designer, and applied by an auditor that cannot invent new failure types.Taxonomies are built from a base-model-only calibration pool; training rollouts and reward traces are not used to define the reported categories.Source: §D.5 · p. 31

Task-specific taxonomies

  • Writing HR patterns were fixed from base-model outputs; roleplay HR excludes ordinary shallow characterization, weak scene progression, and generic prose.Writing HR focuses on outputs that appear attractive to judges without improving the requested deliverable; roleplay HR targets observable shortcuts without in-character consequence.Source: §D.5 · p. 31
  • Roleplay CFR counts predefined validity and completion failures, not broad continuous quality dimensions such as character depth or scene progress.Writing CFR likewise excludes ordinary shortness, weak prose, and missing minor subpoints, focusing instead on discrete non-final or wrong-deliverable failures.Source: §D.5 · pp. 31–32

Independent validation of the HR audit

  • All three auditors found the same HR ordering, EVORS < Base < RaR, at both model scales; human/model agreement was 85.0%.Pairwise turn-level auditor agreement ranged from 86.48% to 93.72%; human agreement with Qwen3.5-27B and DeepSeek-V4-Flash was 90.0% and 88.3%, respectively.Source: §D.5 · p. 32

R-Var and Zero

  • R-Var is average within-query reward variance, while Zero is the share of same-query groups whose reward range falls within a small tolerance.Larger R-Var and lower Zero usually indicate more useful group-relative signal, but neither alone establishes reward quality.Source: §D.5 · p. 33

D.6 Multi-Judge Benchmark Evaluation

The evaluation aggregates three judges on common-success benchmark subsets and compares final-policy quality, paired outcomes, and roleplay behavior. EVORS ranks highest under every judge on both benchmarks, with paired comparisons and dialogue measures providing additional context.

D.6 Multi-Judge Benchmark Evaluation

  • WritingBench scores average five query-specific criteria for 1,000 test instructions, while CoSER evaluates 200 multi-turn simulations across SC, AN, CF, and SQ.Common-success intersections contain 994 writing instructions and 194 CoSER simulations; exclusions are uniform across methods.Source: §D.6 · p. 33
  • EVORS obtains the highest WritingBench and CoSER Overall score under every judge, with the ranking consistent across both benchmarks.Table 16 reports each judge’s scores before aggregation.Source: §D.6 · p. 33, Table 16 · p. 34
  • EVORS leads RaR by 0.454 points on WritingBench, and its CoSER improvements over Base and RaR remain positive throughout paired bootstrap intervals.EVORS outperforms RaR on 145 of 194 matched roleplay simulations; intervals use 20,000 paired resamples.Source: Table 17 · p. 34, §D.6 · p. 34
  • Relative to Base, RaR raises roleplay SC by 1.607 points and CF by 1.991, but lowers AN by 2.420 and SQ by 4.812.The reported shift is uneven across dimensions rather than a uniform loss of capability.Source: §D.6 · p. 34
  • EVORS stays close to the base policy in response length while reducing repetition, unlike RLAIF, RaR, RLER, and OpenRS, whose dialogues show more repetition or truncation.The table’s repetition and truncation statistics are computed directly from dialogues, independently of quality judges.Source: §D.6 · p. 34, Table 18 · p. 34

E.1 Detailed Generality and Ablation Results

  • Table 19 gives judge-specific Qwen3-8B CoSER Overall scores and three-judge dimension averages; Table 20 details downstream ablation quality and reliability audits.These captions specify the contents of the detailed generality and ablation results, but do not provide the cell values.Source: §E.1 · p. 34

E.2 Candidate-State Trace Analysis

The supplementary analyses trace candidate reward-system updates, audit whether reward rankings align with independent preferences, and illustrate selected writing and roleplay repairs. They distinguish local reward-signal diagnostics from downstream policy outcomes.

E.2 Candidate-State Trace Analysis

  • Selected writing candidate updates are grouped by update family, and local two-step R-Var changes are process diagnostics rather than standalone downstream quality gains.The table reports positive/zero/negative counts of 2/0/0, 4/0/0, and 3/1/0 for its three rows.Source: §E.2 · pp. 34–35, Table 21 · p. 35

E.4 Reward-Signal Ranking Audit

  • Table 24 summarizes reliability checks for the blind list-wise ranking audit, including mean inter-judge agreement and a human-vs-LLM check on a 20-group subset.The passage identifies the checks and subset but supplies no numerical outcomes.Source: Table 24 · p. 37
  • The blind ranking audit compares reward-induced rankings with independent preferences over the same four-response groups, using six pairwise preferences per group.A three-model panel is aggregated by Borda voting; the audit also reports judge-vs-reward agreement, top-1 match, reward-top mean rank, and inter-judge agreement.Source: §E.4 · pp. 35, 37

E.5 Case Studies

  • In the writing case, a guard-style integrity node was added to penalize artifacts, and later bounded updates calibrated it against targeted failures and clean examples.A flaw-penalty probe was not selected because empty flaw extraction produced an always-pass score; the case illustrates structural repair, not only rubric rewriting.Source: §E.5 · pp. 37–38
  • A roleplay query-rubric update targeted latest-turn consequence and grounded progression after a sampled same-query group had a score range of about 0.01.Comparative feedback changed mean score from 0.6313 to 0.6390, within-group standard deviation from 0.0765 to 0.0844, and range from 0.1969 to 0.2177.Source: §E.5 · p. 38
Loading 2609.12459v1…