PaperlayerView exact revision on arXiv

Source-linked AI summary

Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skill

Tao Chen, Gangwei Jiang, Pengyu Cheng, Siyuan Huang, Yihao Liu, Jingwei Ni, Jiaqi Guo, Mengyu Zhou, Kai Tang, Junling Liu, Qinliang Su, Xiaoxi Jiang, Guanjun Jiang

arXiv:2606.03980v1cs.LGcs.CL

TL;DR

Reward evaluation must integrate heterogeneous, resource-dependent evidence, but existing scalar and judge-based systems lack a coherent mechanism for doing so. Skill-RM executes a reusable evaluation skill that orchestrates resources and evidence, raising the matched-baseline average from 83.9 to 86.2 across three benchmarks.

  • Problem

    Existing reward models and LLM judges lack a coherent framework for integrating heterogeneous, resource-dependent evaluation evidence.

  • Method

    Skill-RM reformulates reward modeling as executing a reusable skill that selects resources, collects structured evidence, and aggregates criterion-level judgments.

  • Results

    86.2 versus 83.9: Skill-RM raises the matched Qwen3.5-27B LLM-as-a-Judge average and improves all three benchmarks.

  • Takeaways & Limitations

    Skill-RM provides a unified interface for reward evaluation across heterogeneous resources and supports both offline judging and downstream reward applications.

  • Takeaways & Limitations

    Evaluation is scoped to text-based instruction-following and standard reward benchmarks, while the framework relies on manually curated Reward-Evaluation Skills.

Abstract

from arXiv · show

Reward models (RMs) provide critical feedback signals for LLM post-training, notably in reinforced fine-tuning (RFT) and reinforcement learning (RL) pipelines. However, current reward evaluation relies on heterogeneous criteria such as rule-based verifiers, ground-truth references, procedural checklists, and complex rubrics, where a unified mechanism to integrate all types of evidence remains unexplored. To this end, we propose Skill Reward Model (Skill-RM), a unified framework that reformulates reward modeling as the execution of a reusable Reward-Evaluation Skill. By treating reward computation as a structured agentic task, Skill-RM provides a consistent interface to orchestrate heterogeneous resources, dynamically selecting and aggregating evidence tailored to the specific requirements of each input. This approach enables the reward model to move beyond static evaluation, ensuring consistency and transparency across diverse tasks. Extensive experiments on reward benchmarks and downstream applications, including best-of-N selection and reinforcement learning, demonstrate that Skill-RM consistently outperforms traditional judge baselines. Our findings suggest that Skill-RM not only provides a unified solution for reward modeling but also achieves superior performance through the strategic and dynamic orchestration of evidence. The code is at https://github.com/Qwen-Applications/Skill-RM.

1. Introduction

Reward modeling must handle increasingly complex, resource-intensive verification criteria, but existing scalar and judge-based systems struggle to integrate such evidence coherently. Skill-RM addresses this gap by reformulating reward modeling as execution of a reusable Reward-Evaluation Skill with structured resource and evidence interfaces.

  • Motivation: LLM advances in reasoning, coding, and tool-use have shifted reward evaluation from scalar preferences toward complex, resource-intensive verification processes.Factual accuracy, for example, increasingly requires external references or other supporting resources.
  • Limitations: Existing scalar reward models compress resource-grounded evidence into opaque scores, while LLM judges typically organize rubrics, examples, and tools through unstructured flat prompting.These designs are described as uninterpretable and inflexible or unable to coherently integrate diversified resource-dependent judgments.
  • Agent Skills: Agent skills provide a self-contained, filesystem-based abstraction that lets an LLM agent discover, load, and execute procedural logic and supporting resources on demand.A standard skill includes a main SKILL.md document plus auxiliary scripts and assets.
  • Skill-RM: Skill-RM reformulates reward modeling as executing a reusable Reward-Evaluation Skill that defines invocation, heterogeneous-resource access, and structured evidence collection.The framework is designed to provide a unified interface for reward evaluation across diverse evidence types.

2. Preliminary

The preliminary section distinguishes pointwise, pairwise, and rubric-conditioned reward models, then introduces agent skills as reusable, progressively disclosed procedural artifacts with auxiliary resources.

  • Pointwise Reward Modeling: Pointwise reward models assign a scalar score to one response and support policy optimization in RLHF and scalable-feedback approaches such as GRPO and GSPO.DPO instead implicitly recovers the reward function for LLM post-training.
  • Pairwise Reward Modeling: Pairwise reward models compare two responses and output a binary preference indicating whether one response is preferred over the other.The binary output can be interpreted as a sample from a Bernoulli preference distribution.
  • Rubric-conditioned Reward Modeling: Rubric-conditioned reward models make evaluation criteria explicit, produce criterion-level judgments, and aggregate them into a final reward.Aggregation may use rubric-based judging, checklist scoring, criterion-wise rubric aggregation, or calibrated rubric-question outputs.
  • Agent Skills: An agent skill packages reusable procedural knowledge with discovery metadata, on-demand instructions, and auxiliary resources such as references, tools, checklists, and verifiers.Separating the skill specification from its resource bank enables progressive disclosure: irrelevant resources remain out of context while relevant ones can be inspected or executed.

3. Skill Reward Model

Skill-RM reformulates reward modeling as execution of a reusable Reward-Evaluation Skill that orchestrates heterogeneous resources, synthesizes criterion-level evidence, and deterministically produces task-specific rewards. Its modular process externalizes evaluation logic and supports auditable judgments across pointwise, pairwise, and selection settings.

  • Framework: Skill-RM replaces monolithic reward scoring with a reusable skill governing resource orchestration, criterion-level evidence synthesis, and final reward aggregation.This makes reward computation instance-adaptive, evidence-grounded, and modular.
  • Framework: The framework combines a Reward-Evaluation Skill, an agentic evaluation process that retrieves task-relevant resources, and a deterministic readout function.The agent derives structured criterion-level evidence before mapping the resulting trace to the required reward output.
  • Skill and resources: The skill specification MRM defines evaluation criteria, resource invocation, evidence-collection schemas, and the structured-judgment output contract, while URM stores task-specific materials and executable interfaces.Resources can provide textual evidence, deterministic verification procedures, or calibration and aggregation rules.
  • Skill-mediated execution: During evaluation, an agentic judge follows MRM to identify targets, activate criteria, retrieve resources on demand, and populate mandatory criterion-level evidence fields.The execution trace records judge actions, returned observations, and the final structured judgment.
  • Reward readout: A deterministic readout projects the evidence-bearing judgment into pointwise rewards, multi-candidate selections, or pairwise preferences.Pairwise comparison is the specialized K=2 case, while pointwise scoring applies when K=1 and selection when K≥2.

4. Related Work

Related work spans scalar preference-based reward modeling, resource-aware LLM-as-a-Judge evaluation, and reusable agent skills for packaging procedural knowledge and task resources.

  • Reward Modeling & Reasoning: Traditional reward modeling frames language-model alignment as scalar preference prediction, with InstructGPT establishing RLHF from pairwise human preferences and DPO avoiding an explicit online reward model.Reward-model quality has also become a shared evaluation target.
  • Resource-aware Evaluation: LLM-as-a-Judge systems scale open-ended evaluation by prompting models to compare, critique, and explain outputs, while later work studies evaluator biases and fine-grained evaluation dimensions.MT-Bench and Chatbot Arena popularized scalable assistant evaluation.
  • Agent Skills & Skill-mediated Methods: Agent skills package procedural knowledge and task resources into reusable artifacts, including folders of instructions, scripts, and resources for LLM agents.Anthropic presents Agent Skills in Claude as portable procedural knowledge for real-world agents.

5. Experiments

Experiments evaluate Skill-RM across reward benchmarks, sample-specific resource use, resource-organization ablations, fixed-pool Best-of-N selection, instruction-following reward ranking, and downstream RL. Skill-RM improves matched-backbone reward-modeling performance, benefits from skill-mediated evidence aggregation, and achieves the highest reported average in instruction-following RL.

  • Experimental scope: The experiments cover RewardBench2, RM-Bench, JudgeBench, JETTS Best-of-N selection, IF-RewardBench, and instruction-following RL.These tracks assess multidimensional reward capability, subtlety and style-bias robustness, correctness-focused evaluation, fixed-pool selection, instruction-following ranking, and policy optimization.
  • Reward-modeling benchmarks: 2.3 average points: Skill-RM raises the matched Qwen3.5-27B average from 83.9 to 86.2 and improves all three reported benchmarks.Among complete rows, Skill-RM with Qwen3.5-27B has the strongest average, while Qwen3.5-122B-A10B attains the highest JudgeBench score under a different MoE backbone.
  • Resource-use ablations: Same-backbone ablations show that directly appended resources reduce gains, supporting skill-mediated resource organization rather than resource availability or tool access alone.The ablation compares appended resources, sample-specific resources, and Python-tool access against Skill-RM using the same resource types.
  • Best-of-N selection: Best-of-N evaluation uses fixed JETTS reranking pools and the same sequential pairwise knockout protocol for Baseline and Skill-RM, isolating selection quality from generation.Random@10 and Oracle@10 provide random and pool-upper-bound references, while Skywork-Reward-V2-Qwen3-8B independently scores candidates.
  • Instruction-following reward use: 45.9 average: Skill-RM surpasses VerIF at 44.7 and Tulu 3 at 45.1 in downstream instruction-following RL.Skill-RM’s clearest advantage over VerIF occurs on IFEval and AdvancedIF, while IFBench is tied at 27.6.

6. Conclusion

Skill-RM reframes reward modeling as execution of a reusable Reward-Evaluation Skill that orchestrates heterogeneous evaluation resources. Its current evaluation and manually curated design leave multimodal, long-horizon, open-ended, and subjective settings for future work.

  • Conclusion: Skill-RM reformulates reward modeling as a skill-mediated execution procedure rather than monolithic scalar prediction or unstructured prompting.Evaluation logic is externalized into a reusable Reward-Evaluation Skill.
  • Conclusion: The framework systematically orchestrates heterogeneous resources and invocation protocols into an explicit computation flow.This orchestration is implemented through the reusable Reward-Evaluation Skill.
  • Limitations and Future Work: Current evaluation is scoped to text-based instruction-following and standard reward benchmarks, while multimodal, long-horizon, open-ended, and subjective settings remain future directions.The paper identifies these extensions as challenging but promising.
  • Limitations and Future Work: The current design relies on manually curated Reward-Evaluation Skills, which the paper identifies as a limitation for future investigation.This limitation is stated alongside the restricted evaluation scope.

Appendix · A. Evaluation Details

The appendix defines evaluation protocols for RewardBench2, RM-Bench, Best-of-N selection, downstream IFBench evaluation, baseline re-evaluation, and sample-specific resource comparisons. It also clarifies source-reported versus reproduced results and standardizes outcome handling across methods.

  • A. Evaluation Details: Table 2 unmarked external rows are source-reported, while superscript * identifies results reproduced or re-evaluated under this paper’s protocol.A superscript † marks TIR-Judge-Zero’s RewardBench2 value on five non-Ties subsets and its inherited Avg. value.
  • A. Evaluation Details: RewardBench2 contains 1,865 examples and uses official best-of-four evaluation, including separate Ties scoring and the official leaderboard average.RM-Bench contains 11,943 comparisons from 3 × 3 chosen–rejected grids.
  • A. Evaluation Details: For LLM-judge and Skill-RM runs, RM-Bench comparisons are parsed as win, loss, tie, or error, with ties and errors counted as non-wins.Source-reported rows retain the scores reported by their corresponding papers.
  • A. Evaluation Details: Best-of-N evaluation uses official JETTS reranking pools from Qwen2.5-72B-Instruct, with ten candidate responses per prompt and main results at N=10.The four task families are GSM8K, IFEval, HumanEval+, and BigCodeBench, containing 1,319, 541, 270, and 378 public examples, respectively.
  • A. Evaluation Details: IFBench downstream instruction-following evaluation uses the public test set with 300 examples.This protocol is distinct from the JETTS response-selection evaluation.
  • A. Evaluation Details: Locally marked baseline results use each method’s native judgment format, converting outputs to the benchmark metrics used in this paper.RM-R1-DeepSeek-Distill-Qwen-32B is reproduced on RewardBench2 and JudgeBench using separate protocols for non-Ties, Ties, and bidirectional pair accuracy.
  • A. Evaluation Details: OpenRubrics Rubric-RM-8B-voting@5 follows its released two-stage procedure with N=5 generated rubrics, pairwise judgments, and majority-vote aggregation.Because it lacks native absolute RewardBench2-Ties scores, a pairwise-to-Ties adapter uses bidirectional judgments and tournament points.
  • A. Evaluation Details: OpenRS uses public sample-specific resources, released code with Qwen3.5-27B, and full-set accuracy-style definitions counting same, tie, and invalid outputs as non-wins.This replaces released summaries’ exclusion of unresolved same or tie outcomes from the denominator.

B. Skill and Prompt Details

Skill-RM constructs reusable Reward-Evaluation Skills through curated, manually inspected resources and exposes them through prompts, tools, and controlled loading. The reward judge preserves benchmark contracts while selectively using generic or benchmark-specific evidence and visible-text checks.

  • B.1. Construction of Reward-Evaluation Skills: LLM-assisted curation organizes permitted evaluation materials into reusable skills, rather than automatically learning skills.Sources include reward-modeling papers, judging protocols, benchmark documentation, rubrics, references, constraints, checklists, and verifier practices.
  • B.1. Construction of Reward-Evaluation Skills: Each skill normalizes criteria and allowed evidence into resource-bank entries, while SKILL.md defines loading, selection, evidence recording, and output mapping.Entries record resource type, content, applicability scope, and access action.
  • B.1. Construction of Reward-Evaluation Skills: Manual inspection checks grounding, schema consistency, applicability, duplicates, and leakage, after which skills and the resource bank are frozen before evaluation.Excluded materials include hidden labels, target preference labels, model identities, answer-order bookkeeping, solution traces, and evaluation feedback.
  • B.2. Prompt Templates: Skill-RM leaves benchmark prompts and candidate responses unchanged, adding an optional Reward-Evaluation Skill that the judge may load through use_skill.Prompt placeholders are filled by the benchmark formatter and selected skill.
  • B.3. System Prompt of the Reward Judge: The reward judge first evaluates visible text, then may call use_skill for constraints, code or math checks, factuality, safety boundaries, or close bias-prone tradeoffs.The interface permits at most one skill load, a bounded number of viewed resources, and bounded assistant turns before final_answer or the benchmark label.
  • B.3. System Prompt of the Reward Judge: Benchmark contracts preserve required outputs: RewardBench2 uses A/B/C/D, JudgeBench uses forced-choice (a)/(b), and RM-Bench uses scaled pairwise labels.The RewardBench2 prompt requests an impartial choice based on instruction following, helpfulness, relevance, accuracy, depth, creativity, and detail while avoiding position, length, and assistant-name biases.
  • B.5. Tool Interface: Before skill loading, the judge has use_skill and final_answer; afterward, it can access view_resource, python_sandbox, and enabled verifier resources.The sandbox receives only the visible prompt and candidate responses.
  • B.6. Loaded Reward-Evaluation Skill: The core reward_judge_fair skill provides generic judging resources, while reward_judge_operational retains the same judging style and adds benchmark-level rubrics and sample-specific resources.The skill applies shared criteria, bias controls, aggregation, and deterministic visible-text checks; Table 7 indexes the supporting resources and sample-specific extension.

C. Additional Backbone Results

Additional-backbone experiments show that Skill-RM’s gains are not tied to a single judge model. Across four backbones, resource-free Skill-RM improves average performance over direct LLM-as-a-Judge evaluation, while model size does not determine the trend monotonically.

  • Evaluation setup: Table 8 compares direct LLM-as-a-Judge with Skill-RM under the same Qwen3.5 backbone, including optional protocol-exposed sample-specific resources.The sample-specific row supplies such resources to the skill when they are available.
  • Backbone robustness: Across all four backbones, Skill-RM without sample-specific resources improves the average over the corresponding LLM-as-a-Judge baseline.This indicates that the skill formulation is not tied to a single judge model.
  • Backbone robustness: The dense 27B backbone is stronger than the 122B-A10B LLM-as-a-Judge baseline on this evaluation suite.The reported comparison shows that performance does not increase monotonically with model size.
  • Backbone robustness: The 122B-A10B model benefits more from skill-mediated reward evaluation than from direct prompting.This comparison further demonstrates that backbone size alone does not determine the benefit of Skill-RM.

D. Instruction-Following RL Details

The instruction-following RL setup uses pointwise Skill-RM rewards that evaluate visible instructions through constraints, checkers, and verification evidence. An anchored-pairwise ablation shows Skill-RM-style rewards outperform direct or tool-only judging, with verification presentation affecting results.

  • Main pointwise-reward setup: The main run trains Llama-3.1-Tulu-3-8B-SFT on VerInstruct-derived prompts with GRPO in Verl3, using deterministic Qwen3.5-27B judgments mapped to scores in [0, 1].Each rollout response is judged against the visible instruction.
  • Main pointwise-reward setup: The Reward-Evaluation Skill decomposes instructions into atomic constraints, checks hard requirements before style, and returns satisfied_count, total_count, and score.Checklist ratios provide base rewards, while available checker code is executed over the prompt, response, system prompt, and dialogue history.
  • Anchored-pairwise ablation: The anchored-pairwise ablation compares each rollout with a fixed reference from the initial Tulu-3 SFT policy, randomizing presentation to reduce position bias.Wins receive reward 1, losses 0, and failed or unresolved judgments 0.5.
  • Anchored-pairwise ablation: 40.7 to 43.1: adding only a Python tool improves the LLM-as-a-Judge average, while Skill-RM rewards achieve higher averages across all variants.Prompted verifier fields perform best on average, driven by IFEval and AdvancedIF.
Loading 2606.03980v1…