PaperlayerView exact revision on arXiv

Source-linked AI summary

HarnessRL: Co-Evolving Policies with a Just-in-Time Training Harness for Open-Ended RL

Xiaoqiang Wang, Ting-En Lin, Yuchuan Wu, Pinyi Zhang, Caixin Kang, Ling Li, Bingxue Qiu, Jieping Ye, Yongbin Li, Bang Liu

arXiv:2610.26426v1cs.CL

TL;DR

Rubric rewards for open-ended generation can become ineffective or misleading as training progresses. HarnessRL co-evolves task sampling, rollout guidance, and rubric criteria with the policy, using memory and a just-in-time critic to assess updates. On Rubric-Mix-10k, it outperforms the strongest baseline across both model scales and evaluation splits.

  • Problem

    Rubric rewards can become ineffective when scores collapse to uniformly low or high values, or spurious when higher scores do not correspond to better responses.

    Source: §1 · p. 2

  • Method

    HarnessRL co-evolves task sampling, rollout guidance, and rubric criteria with the policy, using memory to attribute failures and a JIT critic to measure candidate updates on the frozen policy.

    Source: §1 · p. 2

  • Results

    HarnessRL beats the strongest baseline by 3.0 points in-domain and 2.8 points OOD at Qwen3.5-4B, and by 2.9 and 3.1 points, respectively, at Qwen3.5-9B.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The reported failure classes call for different interface updates: rollout guidance for blind criteria, sampling for saturated tasks, and rubric changes for spurious criteria.

    Source: §1 · p. 2

    Released references describe the corpus only and are not used in training or the reward.

    Source: §3 · p. 7

Abstract

from arXiv · show

Rubric-based reinforcement learning (RL) trains policies for open-ended generation with natural-language criteria as rewards, but these rewards can become ineffective as training progresses. We identify two failure modes. Under non-discriminative rewards, scores collapse to uniformly low or high values due to insufficient exploration or a mismatch between task difficulty and policy capability. Under spurious rewards, higher scores do not correspond to better responses. Both reflect a training configuration whose task sampling, exploration, and evaluation stay fixed as the policy evolves. We introduce HarnessRL, a training-time harness that co-evolves with the policy through three interfaces for task sampling, rollout guidance, and rubric criteria. HarnessRL maintains a training-time memory from which a harness evolver retrieves within-task history and cross-task evidence. Using this evidence, the evolver attributes each reward failure to the interface whose update can address it. It then runs a proposer and just-in-time critic loop in which the critic accepts a candidate update only after measuring its effect on the current frozen policy, returning rejected candidates for revision. We also curate Rubric-Mix-10k, a five-domain training and evaluation suite for open-ended RL across medical, science, writing, role-play, and instruction-following tasks. On Rubric-Mix-10k, HarnessRL outperforms the strongest baselines by 2.8 to 3.1 points across both 4B and 9B model scales and nearly halves the training steps to its best checkpoint. Code and data are available at https://github.com/Tongyi-ConvAI/Qwen-Character/tree/main/HarnessRL.

1 Introduction

Rubric rewards can become ineffective or misleading as policies evolve, but adapting only the rubric or rollout guidance can shift failures elsewhere. HarnessRL co-evolves task sampling, rollout guidance, and rubric criteria with the policy, and outperforms the strongest baselines on Rubric-Mix-10k at both model scales.

1 Introduction

  • Rubric rewards can become unreliable as training progresses, and adapting only guidance or rubrics can shift failures elsewhere in training.Guidance can saturate fixed criteria, while tightening a rubric without guidance can put criteria beyond the policy’s reach.Source: §1 · p. 2
  • In a rubric-based GRPO run, 61.6% of response pairs have ineffective reward signals: 27.6% lack usable contrast and 34.0% provide contrast that does not reliably track quality.The latter includes 19.8% of pairs ranked in the wrong direction by the reward.Source: §1 · p. 2
  • HarnessRL matches three reward-failure classes to updates in rollout guidance for blind criteria, task sampling for saturated tasks, and rubrics for spurious criteria.Its harness co-evolves these interfaces with the policy, reallocating sampling toward useful learning signals and adjusting guidance as training progresses.Source: §1 · p. 2
  • A training-time memory supplies within-task history and cross-task evidence for attributing failures, while a JIT critic measures candidate updates on the current frozen policy before accepting them.Rejected updates return to the proposer with their measurements; when no update is supported, it proposes noop.Source: §1 · p. 2
  • HarnessRL exceeds the strongest baseline by 3.0 points in-domain and 2.8 points OOD at Qwen3.5-4B, and by 2.9 and 3.1 points, respectively, at Qwen3.5-9B.Reward-oracle agreement is 48.9% under HarnessRL versus 38.4% under GRPO; HarnessRL also reaches its best OOD checkpoint before every adaptive-training baseline.Source: §1 · p. 3

2 HarnessRL

HarnessRL adapts task sampling, rollout guidance, and rubric criteria alongside the policy. Its memory-backed evolver attributes reward failures and accepts candidate updates only after measuring their effects on the frozen policy.

2 HarnessRL

  • HarnessRL’s training-time harness co-evolves with the policy through task-sampling, rollout-guidance, and rubric-criteria interfaces.The harness memory supports failure attribution, while the attributor, proposer, and just-in-time critic separate diagnosis, proposal, and acceptance.Source: §2 · p. 3, §2.3 · p. 5

2.1 Harness-Controlled RL Training

  • The RL loop samples tasks, generates rollouts with optional privileged guidance, scores them under weighted rubric criteria, and weights task losses.Sampling reallocates probability toward tasks with useful learning signal; importance correction preserves each task’s expected contribution under the original uniform distribution.Source: §2 · p. 3, §2.1 · p. 4

2.2 Harness Memory

  • Harness memory stores task rollouts, scores, harness actions, and training dynamics, then retrieves task history and related cross-task criteria.History helps distinguish persistent failures from single-step noise; related responses can supply guidance when a task lacks useful examples.Source: §2.2 · p. 5

2.3 Harness Evolver

  • The attributor routes persistent unmet criteria to rollout guidance, mastered tasks with weak discrimination to sampling, and quality-misaligned criteria to rubric updates.It uses task history and related criteria, and supports labels with quoted rollouts and a wording check for the phrase permitting superficial compliance.Source: §2.3 · p. 5
  • The proposer can adjust task weights, mount or remove guidance fields, and create, rewrite, delete, reweight, merge, or split rubric criteria.Guidance fields operate under a per-task cap that declines over training; anchor criteria may be reweighted but not rewritten.Source: §2.3 · pp. 5–6
  • The JIT critic tests candidate updates on the current frozen policy and accepts them only when interface-specific conditions and reward-change bounds are met.Rejected candidates return to the proposer for revision or replacement with noop; proposer-critic iteration ends when updates are accepted or the round budget is exhausted.Source: §2.3 · pp. 5–6

3 Experiments

Experiments evaluate HarnessRL across five open-ended domains, two policy scales, and in-domain and OOD settings. HarnessRL leads the baselines, while ablations and training analyses examine interface contributions, failure patterns, update stability, and compute efficiency.

3 Experiments

  • Rubric-Mix-10k contains 8,600 training tasks across five domains and 250 held-out tasks per domain, with benchmark-specific OOD evaluation.The evaluation uses unchanged seed rubrics in-domain and four OOD benchmarks that test whether gains persist beyond the harness-updated reward.Source: §3 · pp. 6–7, Table 1 · p. 7

3.1 Main Results

  • HarnessRL exceeds AMARIS++ by 3.0 and 2.8 pp in-domain and OOD at Qwen3.5-4B, and by 2.9 and 3.1 pp at Qwen3.5-9B.It leads every benchmark at both scales and exceeds AMARIS++ on every seed.Source: Table 2 · p. 8, §3.1 · p. 8
  • At Qwen3.5-4B, the in-domain margin over the best baseline is +3.7 pp on writing and role-play but +0.9 pp on instruction following.The paper attributes the wider margins to rubric saturation on open-ended tasks and the narrower margin to near-verifiable criteria, where GRPO reaches 68.1.Source: §3.1 · p. 8

3.2 In-Depth Analysis

  • Removing the critic lowers scores to 59.5 in-domain and 45.9 OOD; removing memory too yields 58.1 and 44.1, near GRPO.Single-interface HarnessRLg and HarnessRLc score 60.6/46.3 and 61.2/47.2, respectively, versus HarnessRL’s 64.5/49.8.Source: §3.2 · p. 9, Table 3 · p. 9
  • Guidance-side and rubric-side arms trade failure classes; HarnessRL records 10.5/13.4/27.2 blind/saturated/spurious versus GRPO’s 12.0/15.6/34.0, while RuscaRL’s blind proportion is 10.0.HarnessRL’s reward-oracle agreement is 48.9, compared with GRPO’s 38.4.Source: §3.2 · p. 10
  • By step 350, 86% of tasks carry no guidance fields, and accepted rubric updates fall from 1,240 in the first window to 140 in the last.By step 500, 52% of tasks are below full sampling weight and 8% are at the lowest level.Source: §3.2 · p. 10, Figure 4 · p. 11
  • Reversed rubric updates end at 1.6% for HarnessRL, versus 18.2% for AMARIS++ and 22.3% without critic and memory.The critic accepts 71% of updates grounded in retrieved evidence, compared with 44% when retrieval finds no evidence above the similarity floor.Source: §3.2 · p. 10, Figure 4 · p. 11
  • At 1.6× GRPO’s per-step cost, HarnessRL reaches its best OOD checkpoint in 17.5 cluster hours, versus 17.9 for AMARIS and 18.3 for AMARIS++.Within the same 17.5 hours, HarnessRL scores 49.8 OOD, compared with AMARIS++ at 46.9; at step 500 it leads AMARIS++ by 3.0 pp on both evaluation sets.Source: §3.2 · p. 11, Figure 5 · p. 12

4 Related work

Prior work adapts rubric criteria, rollout guidance, or task distributions, while HarnessRL combines task sampling, rollout guidance, and rubric criteria in one training-time harness that routes attributed failures to the relevant interface.

Evolving rubric-based RL

  • HarnessRL combines adaptive task sampling, rollout guidance, and rubric criteria, routing each attributed failure to the interface whose update addresses it.Earlier work separately evolves rubrics or task distributions; HarnessRL brings those with rollout guidance into one harness.Source: §4 · p. 12

Scaffolded and privileged rollouts

  • Scaffolded rollouts alter sampled groups or add privileged context, while HarnessRL mounts guidance after measuring a reward increase and can withdraw it early.RuscaRL schedules scaffold fading; HarnessRL withdraws guidance once the target criterion leaves blind and reward remains within tolerance without it.Source: §4 · p. 13

Training-time harness

  • Among training-time harness approaches, HarnessRL's JIT critic measures each primary candidate update on the current frozen policy before accepting it.Its harness memory supplies rollout and update history as evidence.Source: §4 · p. 13

5 Conclusion

HarnessRL co-evolves a training-time harness with the policy and introduces Rubric-Mix-10k for five-domain training and evaluation. It outperforms the strongest baseline across both evaluation settings and model scales, while improving reward-oracle agreement and reaching its best OOD checkpoint in fewer steps than GRPO.

Conclusion

  • HarnessRL outperforms the strongest baseline on in-domain and OOD evaluations at both model scales.Its training dynamics also show lower failure-class proportions and higher reward-oracle agreement than GRPO.Source: §5 · p. 14
  • Rubric-Mix-10k provides training tasks and held-out in-domain and OOD evaluation across five domains.The domains are medical, science, writing, role-play, and instruction-following.Source: §5 · p. 14
  • HarnessRL reaches its best OOD checkpoint in about half the steps GRPO needs, with fewer cluster hours than any baseline.Reversed rubric updates remain rare.Source: §5 · p. 14

Related papers

Loading 2610.26426v1…