PaperlayerView exact revision on arXiv

Source-linked AI summary

CARE: Contrastive Anchor-based Rubric Evolution for Large Language Model Post-Training

Siyuan Li, Xinxin Song, Chen Ruinian, Jingjing Fan, Tingxiong Xiao, Yangen Hu, Ke Zeng, Jinli Suo

arXiv:2609.00892v1cs.AI

TL;DR

Open-ended post-training needs rewards that remain reliable as policies evolve, because static rubrics are hacked and existing dynamic methods lack dependable anchors. CARE contrasts each top rollout with a frontier-generated anchor, using Adaptive repair and Chase refinement to maintain high-reward discrimination. On WildChecklist-9K, it achieves state-of-the-art results across major benchmarks and uniquely sustains improvement against GPT-4.1 anchors over 300 steps.

  • Problem

    Static rubrics are hacked as policies evolve, while dynamic rubric methods lack reliable anchors and suffer from undirected extraction, blind hack detection, and rubric proliferation.

  • Method

    CARE contrasts each highest-scoring rollout with a frontier-model anchor conditioned on the prompt and rubrics, then adaptively repairs hacks or chases substantive quality gaps.

  • Results

    CARE achieves state-of-the-art performance on Arena-Hard-2.0, FollowBench, and InfoBench, and is the only method with sustained win-rate improvement against GPT-4.1 anchors throughout 300 training steps.

  • Takeaways & Limitations

    Online rubric-accuracy maintenance improves robustness to reward hacking and final response quality in open-ended post-training.

  • Takeaways & Limitations

    CARE currently evolves rubrics at every training step, and a principled framework for evaluating generated rubric quality remains missing.

Abstract

from arXiv · show

Rubric-based reinforcement learning decomposes open-ended instructions into prompt-specific, flexible rubrics, making it better suited than reinforcement learning with verifiable rewards for post-training LLMs on open-ended tasks. However, static rubrics are inevitably hacked as the policy evolves, and existing dynamic approaches introduce new problems: undirected rubric extraction, unreliable hack detection, and unbounded rubric proliferation. We propose $\textbf{CARE}$ ($\textbf{C}$ontrastive $\textbf{A}$nchor-based $\textbf{R}$ubric $\textbf{E}$volution), which grounds every rubric evolution step in a high-quality anchor response generated by a frontier model conditioned on the prompt and its rubrics. At each training step, CARE contrasts the highest-scoring rollout against the anchor, enabling two complementary mechanisms: an Adaptive branch that reactively repairs reward misspecification; and a Chase branch that proactively converts frontier-level quality gaps into sharper rubrics. Together, the two branches $\textbf{maintain discriminative accuracy in the high-reward region}$---the precise region where reward over-optimization mostly originates. Experiments on WildChecklist-9K with Qwen2.5-7B-Base and Qwen2.5-7B-Instruct show that CARE achieves state-of-the-art performance on Arena-Hard-2.0, InfoBench, and FollowBench, and is the $\textbf{only}$ method whose win rate against GPT-4.1 anchor responses shows sustained improvement throughout 300 training steps; additional results on Llama-3.1-8B-Instruct and Qwen3-8B further indicate that CARE generalizes across model families.

1 Introduction

Open-ended post-training lacks reliable reward signals, while static and existing dynamic rubric methods remain vulnerable to hacking and poor rubric evolution. CARE addresses these issues with frontier-model anchors and achieves strong benchmark performance with sustained training stability.

  • Open-ended instruction following and long-form generation still lack reliable reward signals, limiting direct use of verifiable rewards.
  • Static rubrics are inevitably hacked as policies and response distributions shift, and they cannot adapt to emergent behaviors.
  • Existing dynamic methods suffer from undirected extraction, blind hack detection, and unbounded rubric proliferation without a reliable quality anchor.
  • CARE grounds rubric evolution in frontier-model anchor responses conditioned on each prompt and its rubrics.
  • CARE combines Adaptive repair of exploited rubrics with Chase extraction of sharper rubrics from quality gaps, maintaining discrimination in the high-reward region.
  • CARE achieves state-of-the-art performance on Arena-Hard-2.0, FollowBench, and InfoBench, while uniquely improving win rate against GPT-4.1 anchors throughout 300 training steps.

2 Related Work

Prior reward and rubric-based approaches provide useful supervision but remain vulnerable to proxy misspecification and static-rubric exploitation. Recent online evolution methods address adaptivity, yet the related-work landscape still motivates CARE’s anchor-guided design.

  • Scalar reward models compress multidimensional quality into one proxy and are structurally susceptible to reward hacking and systematic out-of-distribution errors.
  • Rubric-based RL decomposes instructions into atomic, flexible rubrics that provide finer-grained and harder-to-game rewards than scalar models.
  • Existing rubric-based methods rely on static rubrics fixed before training, leaving them vulnerable to emergent hacking patterns.

3 Method

CARE evolves prompt-specific rubrics online by comparing the highest-scoring rollout with a frontier-generated anchor. Its Adaptive and Chase branches respectively repair hacking and sharpen high-reward discrimination.

  • Rubric RL setup: The rubric RL setup uses prompt-specific rubric sets, GRPO optimization, policy rollouts, and an LLM verifier that scores rubric satisfaction.
  • Reward hacking in rubric RL: Reward hacking occurs when a response receives a high rubric score despite substantially lower true quality than a similarly scored anchor.
  • Rubric types and veto reward: [Resist] rubrics are enforced through a veto reward whose product becomes zero when any resist constraint is violated.
  • Anchor response generation: CARE uses a per-instance GPT-4.1 anchor generated from the prompt and rubric set, treating it as a reference rather than a quality upper bound.
  • High-reward-region objective: At each training step, CARE selects the highest-scoring rollout and contrasts it with the anchor to maintain discriminative accuracy in the moving high-reward region.
  • Unified CARE objective: Both branches refine rubric accuracy online: Adaptive repairs misspecification reactively, while Chase sharpens discrimination before exploitation emerges.
  • Adaptive branch: The Adaptive branch detects hacking among satisfied rubrics, then rewrites the exploited rubric or adds a [Resist] constraint; hacking skips Chase.
  • Chase branch: The Chase branch runs when no hacking is detected and converts the most significant substantive gap into an existing or new [llm] rubric.

4 Experiments

CARE is evaluated across multiple benchmarks, model variants, ablations, and training-trajectory analyses. It consistently outperforms baselines, with its anchor-guided Adaptive and Chase branches contributing complementary benefits and sustained high-reward-region improvement.

  • Evaluation setup: CARE is evaluated on Arena-Hard-2.0, InfoBench, and FollowBench using Qwen2.5-7B-Base and Qwen2.5-7B-Instruct, with additional model-family experiments reported in Appendix D.The evaluation also includes Win-rate against GPT-4.1 anchor responses and ablations on Qwen2.5-7B-Base.
  • Main results: CARE achieves state-of-the-art performance across all three benchmarks under both Qwen model variants and consistently outperforms static rubric-based methods and Online Rubrics.SFT on Anchor performs inconsistently across variants and benchmarks, whereas CARE maintains broad gains.
  • Ablation study: Adaptive Rubric RL outperforms its no-anchor ablation on Arena-Hard-2.0 (5.3 vs. 4.4 Vanilla), InfoBench (82.9 vs. 77.4 Overall), and FollowBench (2.91 vs. 2.57 CSL).With anchors, frontier-judge agreement is 95.2% versus 65.3% without them, while human agreement is 99.1% versus 67.1%.
  • Ablation study: The veto reward improves Adaptive Rubric RL across all benchmarks by preventing [Resist] violations from being offset through weighted averaging.Because veto enforcement amplifies incorrectly specified constraints, its reliability depends on accurate Adaptive decisions.
  • Ablation study: Chase improves early high-reward-region discrimination, whereas Adaptive provides reactive robustness; CARE combines both and achieves the highest, sustained win-rate trajectory.Chase reaches the second-highest win rate through step 150 before declining, while Adaptive variants remain more stable but plateau lower.
  • Training dynamics: CARE is the only method whose win rate against GPT-4.1 anchor responses improves sustainably across all 300 training steps.Contemporaneous reward fluctuations partly reflect revised rubrics taking effect when samples are next encountered; independent evaluations support reproducible late-stage improvement.
  • Constraint analysis: CARE nearly doubles FollowBench Style from 48.0 to 94.7 and improves Situation from 55.5 to 73.6 on Qwen2.5-7B-Base.These are the two most implicit constraint types, where static rubrics largely fail to capture the relevant quality gaps.

5 Conclusion

CARE provides anchor-guided online rubric evolution for rubric-based reinforcement learning, comparing the highest-scoring rollout with a high-quality anchor to repair misspecification and sharpen frontier discrimination. On WildChecklist-9K, it improves performance across model variants and supports robustness to reward hacking and final response quality.

  • CARE is an anchor-guided framework for online rubric evolution in rubric-based reinforcement learning.
  • At each training step, CARE compares the highest-scoring rollout with a high-quality anchor to repair exploited misspecification and sharpen frontier discrimination.
  • CARE achieves substantial performance gains across model variants on WildChecklist-9K.
  • Maintaining rubric accuracy online improves robustness to reward hacking and final response quality in open-ended post-training.

Limitations

The study identifies two limitations: rubric evolution currently runs at every training step, and rubric quality lacks a principled direct evaluation framework.

  • CARE currently activates rubric evolution at every training step rather than selectively targeting informative failures.The authors suggest periodic or hacking-likelihood-based gating to improve update precision and reduce training cost.
  • The study lacks a principled framework for directly evaluating generated rubric quality.Current quality improvement is indirect, relying on maintained discriminative accuracy in the high-reward region.

Ethical Considerations

CARE may be misused for harmful objectives, and its frontier-model anchors and LLM judges may propagate biases or unsafe preferences; the authors therefore do not present it as a deployment-ready safety guarantee.

  • CARE could optimize models toward undesirable objectives when rubrics encode harmful, manipulative, or deceptive goals.
  • Frontier-model-generated anchors and LLM judges may inherit biases, cultural assumptions, or unsafe preferences from their underlying models.Treating these signals as universally correct may propagate bias and over-reward stylistic conformity or particular response preferences.
  • The authors characterize CARE as a research method rather than a deployment-ready safety guarantee and recommend human review, task restrictions, and explicit risk auditing for high-stakes settings.

A Why Online Maintenance of High-Reward-Region Accuracy Matters

The appendix formalizes why high-reward-region accuracy must be maintained throughout optimization: cumulative true-quality advantages compete with cumulative rubric misspecification, which can eventually reverse frontier rankings.

  • Setup: The analysis extends a static high-reward-region argument from a fixed proxy reward to CARE’s dynamic setting, where rubric rewards change during training.
  • Setup: The idealized update separates latent gold reward from stepwise rubric misspecification across the evolving high-reward region.Here, the stepwise distortion is defined as Δ_t(x, y) = r_t(x, y) − r⋆(x, y).
  • Proposition: Final frontier rankings are governed by competition between the cumulative true-quality gap and cumulative misspecification gap.
  • Proposition: Persistent misspecification on repeatedly high-reward responses can overturn the true ordering, causing a lower-quality response to receive higher final policy probability.
  • Corollary: For frontier pairs with correct latent ordering, the corollary gives a sufficient condition for preserving that ordering through step T.
  • Corollary: If cumulative static frontier distortion eventually exceeds the initial log-odds and cumulative true-quality advantage, wrong frontier ordering becomes inevitable.
  • Connection to CARE: The formal result motivates maintaining high-reward-region accuracy throughout training, while not proving that any particular rubric-update rule reduces frontier distortion or enlarges separation.

B LLM-Based Quantitative Analysis of CARE-Extracted Rubrics

Appendix B validates CARE’s anchor construction and Adaptive decisions quantitatively, then organizes the evolved rubrics into separate Adaptive and Chase taxonomies.

  • B.3 Adaptive Branch Taxonomy: The appendix presents a fine-grained taxonomy for Adaptive-branch [Resist] rubrics.Table 8 lists its major classes and subtypes.
  • B.4 Chase Branch Taxonomy: The appendix presents a corresponding fine-grained taxonomy for Chase-branch [llm] rubrics.Table 9 lists its major classes and subtypes.
  • B.1 Anchor Quality Validation: 93.8% of 10K candidate anchors were accepted by both automatic judges, while 0.2% were rejected by both.The final WildChecklist-9K set uses only unanimously accepted candidates.
  • B.1 Anchor Quality Validation: 99.5% of 200 retained anchors passed a subsequent stratified human audit.The audit found 199 passing anchors.
  • B.2 Adaptive-Decision Reliability: Table 4 compares automatic and human audits of Adaptive decisions with and without anchor responses.The comparison evaluates whether anchors improve hack detection and constraint selection.
  • B.2 Adaptive-Decision Reliability: Anchor responses increased frontier-judge agreement from 65.3% to 95.2%.The audit directly tests whether Adaptive detects hacking and introduces an appropriate constraint.
  • B.2 Adaptive-Decision Reliability: Anchor responses increased human judgment-level agreement from 67.1% to 99.1%.The same audit procedure was repeated after removing the anchor from the detection context.
  • C.1 Hyperparameter Settings: CARE is implemented on Verl v0.4.0 and trained with GRPO on 16 NVIDIA A100 GPUs.The key GRPO hyperparameters are summarized in Table 10.

C.2 Training Time Comparison

CARE adds offline anchor generation and repeated online comparisons, increasing measured per-step training time while limiting comparisons to one top rollout per prompt.

  • C.2 Training Time Comparison: 448.9s versus 318.2s: CARE increases Qwen2.5-7B-Base per-step training time relative to Rubric RL.This corresponds to 1.411× total time, or a 41.1% increase.
  • C.2 Training Time Comparison: CARE’s offline anchor-generation cost is paid once, while its online comparison cost repeats during training.The costs scale with dataset size N and training epochs E as described by the paper’s cost model.
  • C.2 Training Time Comparison: CARE performs B comparisons per step rather than BG by selecting only the highest-scoring rollout for each prompt.Increasing the number of rollouts per prompt therefore does not increase CARE-specific comparison count.
  • C.2 Training Time Comparison: The relative overhead becomes smaller when rollout generation dominates comparison time.Anchor-generator cost affects only CA, whereas judge cost affects repeated CC.
  • C.2 Training Time Comparison: More selective CARE activation is identified as a promising direction for reducing online overhead.This is stated as a limitation-related future direction.
  • C.2 Training Time Comparison: CARE is additionally evaluated on Llama-3.1-8B-Instruct and Qwen3-8B across Arena-Hard-2.0, InfoBench, and FollowBench.The evaluation follows the same protocol as Table 1.
  • C.2 Training Time Comparison: Independent evaluations of Qwen2.5-7B-Base checkpoints improve throughout training against Qwen3-8B-Thinking and on external InfoBench.These measures are separate from evaluation against CARE’s GPT-4.1 training anchors.

D.3 Training-Seed Robustness

Across three independent training seeds, CARE shows consistent improvement against GPT-4.1 anchor responses with limited cross-seed variation.

  • D.3 Training-Seed Robustness: Every retraining run improves by 21 percentage points from step 100 to step 300.The runs use the same hyperparameters and evaluate fixed 300 held-out prompts.
  • D.3 Training-Seed Robustness: 45.0% final win rate: every independent retraining run reaches this value against GPT-4.1 anchors.These retraining results are distinct from the 47% final win rate reported for the main single-run experiment.
  • D.3 Training-Seed Robustness: Cross-seed variation is limited to 0.6 points at step 200.The results are summarized across three independent CARE training seeds.

D.4 Paired-Bootstrap Analysis

CARE’s paired-bootstrap analysis supports late-stage improvement and final advantage over Adaptive, while its rubric-evolution prompt operationalizes anchor comparison through hacking checks and quality-gap extraction.

  • D.4 Paired-Bootstrap Analysis: All paired-bootstrap 95% confidence intervals in Table 7 exclude zero.The analysis uses 10,000 resamples of the same 300 prompts and supports CARE’s late-stage improvement and final advantage over Adaptive.
  • E.2 Prompt for CARE Rubric Evolution: CARE compares the Agent with the Golden response first for reward-hacking vulnerabilities, then for substantive quality gaps when no hack is found.The shared prompt is used by both Adaptive and Chase branches.
  • C.2 Training Time Comparison: CARE’s measured per-step time is 448.9s versus 318.2s for Rubric RL on Qwen2.5-7B-Base.The comparison corresponds to 1.411× total time, or a 41.1% increase.
  • Additional Model Results: CARE consistently outperforms Rubric RL on Llama-3.1-8B-Instruct and Qwen3-8B under the reported evaluation format.Table 12 covers Arena-Hard-2.0, InfoBench, and FollowBench.
  • E.2 Prompt for CARE Rubric Evolution: The rubric guidelines require atomic, binary, specific criteria and prohibit vague terms that are not objectively judgeable.Specific unwanted patterns should be named concretely.
  • E.2 Prompt for CARE Rubric Evolution: For hack detection, the Golden demonstrates substantive fulfillment of satisfied rubrics; for divergence extraction, it is a reference rather than a quality ceiling.The Agent may be genuinely superior or inferior on a dimension.
  • E.2 Prompt for CARE Rubric Evolution: Reward hacking is defined as satisfying a rubric’s letter while violating its intent.The prompt analyzes only satisfied rubrics because unsatisfied rubrics cannot exhibit hacking under this procedure.
  • E.2 Prompt for CARE Rubric Evolution: The prompt distinguishes hacking from verbosity, style or structure differences, and honest lower-quality responses.Capability gaps that fail rubrics honestly do not trigger patching.
Loading 2609.00892v1…