PaperlayerView exact revision on arXiv

Source-linked AI summary

RubricArmor: Adversarial Evolution Improves LLM-Based Rubric Generation

Haocheng Yang, Yuchao Zhang, Licheng Pan, Jiajun Fan, Maolin Wang, Kangning Zhang, Shuai Shao, Shijian Wang, Yuan Lu, Chunyuan Zheng, Hao Wang

arXiv:2610.05308v1cs.CL

TL;DR

Directly generated rubrics can omit or underspecify criteria, allowing low-quality responses to receive high rewards, while existing generation methods do not proactively test for this vulnerability. RubricArmor alternates adversarial responses and rubric repairs; it outperforms the strongest baseline on preference evaluation and achieves the best downstream RL performance in most tested settings.

  • Problem

    Omitted or underspecified rubric criteria can let low-quality responses obtain high rewards, and existing generation methods do not proactively guard against this reward hacking.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    RubricArmor alternates attacks that construct task-failing responses satisfying the current rubric with repairs that detect exposed defects while preserving valid criteria.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    RubricArmor exceeds the strongest baseline by 3.12 points in macro accuracy across nine preference evaluation sets and achieves the best downstream RL performance in 13 of 15 settings across three policy models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The evolved rubric is reported to resist potential reward hacking and guide policies toward high-quality responses.

    Source: §1 · p. 2

    RubricArmor requires multiple rounds of LLM calls per rubric, and errors by LLM quality judges can propagate into revised rubrics.

    Source: §5 · p. 9

Abstract

from arXiv · show

Rubric-based reinforcement learning (RL) provides interpretable rewards for aligning large language models (LLMs) by evaluating responses against query-specific evaluation criteria. To construct rubrics at scale, a straightforward approach to LLM-based rubric generation is to prompt an LLM to generate a rubric directly from the query. However, rubrics directly generated by LLMs are vulnerable to reward hacking, since omitted or underspecified criteria allow the policy to obtain high rubric rewards with low-quality responses. Existing LLM-based rubric generation methods improve the granularity and coverage of the generated criteria but do not proactively guard against reward hacking. To address this limitation, we propose RubricArmor, an adversarial framework that exposes and mitigates potential reward hacking at the rubric generation stage before it occurs in subsequent RL. Specifically, RubricArmor performs adversarial evolution, in which an attack step and a repair step alternate over multiple rounds. The attack step simulates the reward hacking of the policy by constructing adversarial responses that satisfy the current rubric but fail to properly complete the task. The repair step then revises the rubric to detect the response defects exposed by the attack step while preserving other valid criteria. Extensive experiments demonstrate that RubricArmor outperforms competitive rubric generation baselines and translates into more effective downstream rubric-based RL.

1 Introduction

Directly generated rubrics can leave essential requirements unchecked or underspecified, allowing low-quality responses to earn high rewards. RubricArmor uses adversarial attack–repair rounds to expose these gaps, and experiments report improvements in rubric accuracy and downstream RL.

1 Introduction

  • Omitted or underspecified criteria can let a response violate quality requirements yet retain reward, enabling RL to reinforce high-scoring, low-quality responses.Rubric-based RL rewards responses according to which rubric criteria they satisfy.Source: §1 · p. 1
  • Prior methods use finer-grained checklists or refine criteria from reference responses and preferences, but do not test responses that actively exploit the rubric.The cited approaches are described as improving granularity and coverage without proactively addressing reward hacking.Source: §1 · pp. 1–2
  • RubricArmor alternates attacks that satisfy the current rubric but fail the task with repairs that detect the exposed defects while preserving valid criteria.Repeated rounds are intended to cover more boundary cases and implicit requirements.Source: §1 · p. 2
  • 3.12 points higher macro accuracy than the strongest baseline across nine preference evaluation sets, with the best downstream RL performance in 13 of 15 settings across three policy models.These are the reported headline experimental results.Source: §1 · p. 2

2 Preliminaries

Rubric-based RL evaluates responses against query-specific criteria and aggregates the judgments into rewards for policy optimization. LLM-based rubric generation scales rubric construction, using direct checklists or progressive refinement to express query requirements as verifiable criteria.

2.1 Rubric-Based Reinforcement Learning

  • Rubric-based RL optimizes a policy against rewards formed by aggregating verifier judgments over explicit, query-specific criteria.The paper uses equal weights, so the reward is the fraction of criteria satisfied; common optimization algorithms include PPO, GRPO, and GSPO.Source: §2.1 · p. 2

2.2 LLM-Based Rubric Generation

  • LLM-based rubric generation prompts language models to express ideal-response requirements as verifiable criteria, avoiding the cost and scaling difficulty of expert annotation or training rubric generators.Prior approaches translate requirements directly into checklists (Cook et al., 2024; Wei et al., 2025) or progressively refine criteria (Zhang et al., 2026; Shen et al., 2026; Li et al., 2026a).Source: §2.2 · p. 2
  • The goal is a query-tailored rubric that captures task requirements accurately and comprehensively so its reward reflects response quality.The examples illustrate omissions that leave factual errors unchecked and underspecified criteria that let incorrect responses pass.Source: §2.2 · p. 3, Figure 1 · p. 3

3 Methodology

RubricArmor addresses reward hacking by evolving rubrics through attacks that expose undetected defects and repairs that revise criteria. The workflow validates attacks and revisions, retains accepted changes, and returns the current rubric when evolution stops.

3.1 Motivation

  • Rubrics that omit essential requirements or underspecify criteria can award full reward to defective responses, as illustrated by an unchecked factual error and an incorrect definition of convergence.The section frames three questions: how to expose and mitigate reward hacking, whether mitigation improves rubric quality, and whether that improvement translates to downstream RL.Source: §3.1 · p. 3

3.2 Adversarial Rubric Generation

  • RubricArmor iteratively attacks the current rubric with task-failing responses that satisfy it, then repairs criteria to detect exposed defects while preserving valid criteria.The process is presented as adversarial evolution that expands coverage of boundary cases and implicit requirements; Figure 2 depicts an evolution round.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.2.1 Attack Discovery

  • An attack succeeds only when the response receives full rubric reward and an independent quality judge confirms a substantial defect; that verified response grounds rubric repair.The verifier checks rubric satisfaction, while the quality judge checks whether the defect is present and substantially reduces response quality.Source: §3.2.1 · p. 4

3.2.2 Verified Rubric Repair

  • Verified Rubric Repair revises criteria to catch the confirmed defect and accepts a revision only if the response reward falls and review approves the criteria.Criteria are intended to be atomic, self-contained, and non-overlapping, while preserving other valid criteria.Source: §3.2.2 · p. 5

3.3 The workflow of RubricArmor.

  • The workflow alternates attack discovery and repair, stores adversarial responses, outcomes, and accepted revisions in memory, and returns the current rubric when no revision is accepted.Evolution stops after at most T rounds or P consecutive rounds without an accepted revision; the last accepted rubric is used for downstream policy optimization.Source: §3.3 · p. 5

4 Experiments

The experiments compare RubricArmor with rubric-generation baselines, examine its design and generalization, test sensitivity to evolution rounds, and evaluate downstream rubric-based RL.

4.2 Main Results (RQ1)

  • RubricArmor achieved the highest macro accuracy, beating Direct on all nine evaluation sets and the strongest baseline, ChasingTail, by 3.12 points.Table 1 reports pairwise preference accuracy across nine evaluation sets from three benchmark suites.Source: Table 1 · p. 6, §4.2 · p. 7

4.1 Experimental Setup

  • The evaluation covers nine preference-accuracy sets, compares checklist, refinement, trained-generator, and direct-prompt baselines, and uses GPT-4.1-mini as verifier across methods.Unless otherwise specified, Gemini-3.1-Flash-Lite is the generator backbone; the maximum evolution rounds are T = 7 and patience is P = 3.Source: §4.1 · p. 6

4.3 Ablation Study (RQ2)

  • Ablations scored 71.69% without concrete response construction and 72.80% without the adversarial objective; the two designs are complementary.The gain over Direct exceeds the sum of the gains of the two variants: the objective selects defects to seek, and response construction makes each defect verifiable.Source: Table 2 · p. 7, §4.3 · pp. 7–8

4.5 Hyperparameter Sensitivity (RQ4)

  • Accuracy rises rapidly and then plateaus across three suites for both backbones, while criteria counts remain nearly stable, from 4.36 to 4.66 on JudgeBench with Gemini-3.1-Flash-Lite.The authors attribute the saturation to fewer successful attacks as accepted revisions accumulate.Source: Figure 3 · p. 7, §4.5 · p. 8

4.4 Generalization across Rubric Generator Backbones (RQ3)

  • Across five rubric-generator backbones, RubricArmor improved over Direct on all three benchmark suites under every backbone.The evaluated backbones span different families and capabilities, including Gemini-3.1-Flash-Lite and Qwen3.5-397B-A17B.Source: Table 3 · p. 8, §4.4 · p. 8

4.6 Downstream Rubric-Based RL (RQ5)

  • RubricArmor led on 13 of 15 downstream policy–metric combinations and raised Qwen2.5-3B-Instruct’s AlpacaEval 2.0 LC win rate by 6.57% over ChasingTail.All methods improved BiGGen-Bench scores and AlpacaEval 2.0 win rates over the original policies after GRPO.Source: §4.6 · p. 8, Table 4 · p. 9
  • RubricArmor led on all length- and style-controlled win rates; on Qwen2.5-7B-Instruct, ChasingTail’s vanilla-win-rate lead disappeared under those controls.The authors interpret this as indirect evidence that RubricArmor mitigates reward hacking in rubric-based RL.Source: §4.6 · p. 9

5 Conclusion

RubricArmor repairs rubric defects through adversarial evolution before RL, and the paper reports stronger rubric evaluation and downstream RL results. The related work distinguishes this pre-RL intervention from static and dynamic rubric construction, rubric-generation refinement, and training-time reward-hacking mitigation.

5 Conclusion

  • RubricArmor exceeds the strongest baseline by 3.12 points in macro accuracy across nine preference evaluation sets and achieves the best downstream RL performance in 13 of 15 settings.These results span three policy models.Source: §5 · p. 9
  • RubricArmor is framed as a pre-RL attack–repair framework that simulates policy reward hacking and revises rubrics to detect exposed response defects.The conclusion reports that it improves macro accuracy and downstream RL performance.Source: §5 · p. 9

A.1 Rubric-Based Reinforcement Learning

  • Rubric-based RL uses explicit query-specific criteria as rewards, while static rubrics remain fixed and dynamic rubrics update during training.RubricArmor repairs defects before RL and reuses the resulting rubric during policy optimization.Source: §A.1 · p. 13

A.2 LLM-Based Rubric Generation

  • Checklist generation and progressive refinement improve rubric precision and coverage, but do not test criteria against adversarial responses that exploit the rubric.RubricArmor instead attacks each generated rubric and repairs confirmed defects.Source: §A.2 · p. 13

A.3 Reward Hacking

  • Reward hacking occurs when optimizing an imperfect proxy worsens the true objective; existing mitigations act during policy optimization, whereas RubricArmor repairs rubric defects before RL.The paper describes this earlier intervention as complementary to training-time mitigation.Source: §A.3 · p. 13

B More Experimental Results

The supplementary experiments cover verifier-backbone generalization, computational cost across evolution rounds, and out-of-domain generalization of RL-trained policies.

B More Experimental Results

  • The supplementary results extend the paper’s generalization study to verifier backbones, measure computational cost across evolution rounds, and test RL-trained policies out of domain.The out-of-domain evaluation uses HealthBench-Hard.Source: B More Experimental Results · p. 14
  • Verifier-backbone generalization is examined in Appendix B.1.Source: B More Experimental Results · p. 14
  • Computational cost across evolution rounds is reported in Appendix B.2, while out-of-domain policy generalization on HealthBench-Hard is evaluated in Appendix B.3.Source: B More Experimental Results · p. 14

B.1 Generalization across Verifier Backbones

RubricArmor improves over Direct across verifier families and capabilities, with the largest relative gains under the weakest verifier on two benchmark suites.

B.1 Generalization across Verifier Backbones

  • RubricArmor improves over Direct on all three benchmark suites under every tested verifier backbone.The authors interpret this as evidence that it improves rubrics rather than fitting a particular verifier family.Source: §B.1 · p. 14
  • RubricArmor improves over Direct with both weaker and stronger verifiers, with the largest relative gains under GPT-4o-mini on JudgeBench and RM-Bench.Source: §B.1 · p. 14
  • The authors suggest that self-contained criteria reduce the knowledge a verifier must supply.Source: §B.1 · p. 14

B.2 Computational Cost

RubricArmor’s token and time costs rise quickly in the first three evolution rounds and then flatten as early stopping ends many queries.

B.2 Computational Cost

  • RubricArmor’s measured computational cost is total tokens and elapsed time per query, and early stopping bounds the cost of adversarial evolution.Source: §B.2 · p. 14
  • Cost grows quickly during the first three rounds and flattens afterward because more than half of queries stop after three consecutive rounds without an accepted revision.Source: §B.2 · p. 14
  • Increasing the maximum evolution rounds T adds cost only for the few queries whose rubrics are still being revised in later rounds.Source: §B.2 · p. 14

B.3 Out-of-Domain Policy Generalization

RubricArmor’s WildChat-trained policies are evaluated on HealthBench-Hard, a healthcare benchmark with physician-written criteria. The reported results show gains beyond the training domain across policy models and evaluation axes.

B.3 Out-of-Domain Policy Generalization

  • RubricArmor improves over the original policies on all five axes for all three policy models, and has the highest average score for each model.It also scores best on 9 of 15 policy–axis combinations, including Communication Quality for every policy model.Source: §B.3 · pp. 14–15, Table 6 · p. 15
  • Physician-written criteria are not used in rubric generation or policy optimization, making the transfer results indirect evidence that RubricArmor mitigates reward hacking.The evaluation uses 1,000 healthcare conversations and GPT-4.1 as judge; sample–axis scores are clipped to [0, 1] before averaging.Source: §B.3 · pp. 14–15

C.1 Dataset Descriptions

The experiments use preference benchmarks to assess rubric quality and separate queries and benchmarks for downstream rubric-based RL. The datasets cover general-purpose evaluation as well as healthcare.

Preference Benchmarks for Rubric Evaluation

  • Nine evaluation sets across JudgeBench, RM-Bench, and RewardBench form the preference benchmarks, with each instance containing a query, chosen response, and rejected response.The sets cover objective correctness, sensitivity to subtle content differences and response style, and detection of instruction deviations.Source: §C.1 · p. 15

JudgeBench

  • JudgeBench contributes Knowledge, Reasoning, Math, and Coding sets, where preferences are determined by objective correctness.The benchmark contains challenging response pairs.Source: §C.1 · p. 15

RM-Bench

  • RM-Bench contributes Chat, Math, Code, and Safety sets, with three response styles yielding nine comparisons per query.It evaluates sensitivity to subtle content differences and robustness to response style.Source: §C.1 · p. 15

RewardBench

  • RewardBench contributes Chat-Hard, which tests whether a reward model identifies subtle deviations from an instruction.The benchmark covers chat, reasoning, and safety.Source: §C.1 · p. 15

Datasets for Downstream Rubric-Based RL

  • Downstream RL uses 2K WildChat queries for training and evaluates trained policies on three general-purpose benchmarks and one healthcare benchmark.The listed evaluation datasets include BiGGen-Bench, AlpacaEval 2.0, Arena-Hard, and HealthBench-Hard.Source: §C.1 · p. 16

C.2 Implementation Details

The implementation alternates rubric initialization, adversarial attack, and verified repair, with review and verification checks. The paper also specifies default evolution and downstream RL settings.

Rubric Generation Settings

  • RubricArmor initializes each query’s rubric from the query, then alternates attack discovery and verified repair for at most 7 rounds, stopping after 3 consecutive rounds without an accepted revision.The default shared rubric-generator backbone is Gemini-3.1-Flash-Lite with low reasoning effort; backbone and sensitivity studies vary the setup.Source: §C.2 · p. 16

Downstream RL Settings

  • Downstream RL uses GRPO with Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, and Qwen3-4B policies, with GPT-4.1-mini as verifier and GPT-4.1 as benchmark judge.Training uses 128-query batches, 8 rollouts per query, learning rate 1 × 10^-6, and 3 epochs on 8 NVIDIA H20 GPUs.Source: §C.2 · p. 16

Initializer

  • The initializer receives the user task and produces a complete set of specific, directly verifiable criteria that cover requirements without unsupported obligations.Criteria should be atomic, self-contained, non-overlapping, and allow alternative correct solutions.Source: §C.2 · p. 17

Attacker

  • The attacker uses the task, current criteria, and past attempts to construct a plausible response that passes every criterion but substantially fails the task.A separate verifier checks criterion passes, while the attacker describes the actual defect and its material consequence; prior feedback guides revisions.Source: §C.2 · p. 17

Quality Judge

  • An independent quality judge checks whether the proposed failure occurs in the answer and substantially harms task completion, without seeing the criteria or scores.The judge grounds both decisions in concrete evidence and does not propose scoring criteria.Source: §C.2 · p. 18

Repair Agent

  • After a judge confirms a task failure, the repair agent makes the smallest task-grounded change that detects it while preserving unaffected criteria and valid alternatives.Repairs may correct requirements, add missing checks, or clarify underspecified criteria; criteria retain atomic, self-contained, and non-overlapping properties.Source: §C.2 · p. 18

Review Agent

  • The review agent checks grading evidence for an actual FAIL that catches the confirmed defect and rejects unsupported restrictions or changes that exclude valid answers.It also checks that the revision preserves valid coverage and the rubric properties.Source: §C.2 · p. 19

Verifier

  • The verifier evaluates one criterion against the task and answer, returning PASS or FAIL with a concrete reason while avoiding unstated quality requirements.It judges only the supplied criterion’s scope and does not let unrelated failures affect its verdict.Source: §C.2 · p. 20
Loading 2610.05308v1…