RewardHarness: Self-Evolving Agentic Post-Training

Yuxuan Zhang; Penghui Du; Bo Li; Cong Wei; Junwen Miao; Huaisong Zhang; Songcheng Cai; Yubo Wang; Dongfu Jiang; Yuyu Zhang; Ping Nie; Wenhu Chen; Changqian Yu; Kelsey R. Allen

Source-linked AI summary

RewardHarness: Self-Evolving Agentic Post-Training

Yuxuan Zhang, Penghui Du, Bo Li, Cong Wei, Junwen Miao, Huaisong Zhang, Songcheng Cai, Yubo Wang, Dongfu Jiang, Yuyu Zhang, Ping Nie, Wenhu Chen, Changqian Yu, Kelsey R. Allen

arXiv:2605.08703v1May 9, 2026cs.AIcs.CLcs.CVcs.LG

TL;DR

Reliable evaluation of instruction-guided image edits needs human-aligned rewards, but existing models generally rely on large-scale preference comparisons. RewardHarness instead evolves a library of evaluation skills and tools from about 100 demonstrations, achieving 47.4% average accuracy on editing benchmarks and surpassing GPT-5 by 5.3 points.

Problem
Image-editing evaluation needs human-aligned reward signals, while current models typically require hundreds of thousands of labeled comparisons to acquire similar preferences.
Method
RewardHarness keeps model weights fixed while an Orchestrator iteratively evolves and routes an external library of Skills and Tools for preference judgments.
Results
47.4% average accuracy on EditReward-Bench and GenAI-Bench surpasses GPT-5 by 5.3 points; GRPO-tuned models using it as reward achieve 3.52 on ImgEdit-Bench.
Takeaways & Limitations
The results support context evolution as a viable, data-efficient alternative to large-scale preference annotation for image-editing reward modeling.
Takeaways & Limitations
Its effectiveness beyond instruction-guided image-editing evaluation, including text-to-image, video, and 3D tasks, remains unexplored.

Abstract

from arXiv · show

Evaluating instruction-guided image edits requires rewards that reflect subtle human preferences, yet current reward models typically depend on large-scale preference annotation and additional model training. This creates a data-efficiency gap: humans can often infer the target evaluation criteria from only a few examples, while models are usually trained on hundreds of thousands of comparisons. We present RewardHarness, a self-evolving agentic reward framework that reframes reward modeling as context evolution rather than weight optimization. Instead of learning from large-scale annotations, RewardHarness aligns with human preferences by iteratively evolving a library of tools and skills from as few as 100 preference demonstrations. Given a source image, candidate edited images, and an editing instruction, an Orchestrator selects the most relevant subset of tools and skills from the maintained library, and a frozen Sub-Agent uses them to construct a reasoning chain that produces a preference judgment. By comparing predicted judgments with ground-truth preferences and analyzing successes and failures in the reasoning process, the Orchestrator automatically refines its library of tools and skills without additional human annotation. Using only 0.05% of the EditReward preference data, RewardHarness achieves 47.4% average accuracy on image-editing evaluation benchmarks, surpassing GPT-5 by 5.3 points. When used as a reward signal for GRPO fine-tuning, RL-tuned models achieve 3.52 on ImgEdit-Bench. Project page: https://rewardharness.com.

1 Introduction

RewardHarness reframes image-editing reward modeling as context evolution: it keeps model weights fixed while iteratively building an explicit library of evaluation skills and tools from few preference demonstrations. This addresses the cost, opacity, inflexibility, and data inefficiency of annotation-heavy reward-model training while achieving strong benchmark performance.

Reliable evaluation remains a bottleneck for image editing and reinforcement learning because reward signals must faithfully reflect human preferences.
Existing methods collect large-scale human preference annotations and train dedicated reward models, creating substantial cost, training, opacity, and deployment limitations.
Humans can internalize evaluation criteria from small calibration sets, whereas current models typically require hundreds of thousands of labeled comparisons for similar preference behavior.
RewardHarness evolves external Skills and Tools while keeping model weights fixed, using demonstrations to build an explicit, reusable library of evaluation knowledge.
About 100 preference demonstrations enable RewardHarness to compare predictions with human labels, analyze successes and failures, and refine its library without additional human supervision.
47.4% average accuracy is achieved on EditReward-Bench and GenAI-Bench using only 0.05% of preference data, surpassing the Qwen-based EditReward variant trained with supervised fine-tuning on 200K preference pairs.The system uses a Claude-based Orchestrator and a frozen Qwen2.5-VL-7B Sub-Agent, without gradient-based reward-model training.

2 Method

REWARDHARNESS frames reward modeling as context evolution: an Orchestrator retrieves evolving Skills and Tools for a frozen Sub-Agent, whose structured judgments are compared with human preferences to refine the Library. The final selected Library contains 3 Skills and 4 Tools and reaches 62.5% validation accuracy from a 100-example calibration set.

System overview: REWARDHARNESS uses an Orchestrator and a shared Library of interpretable Skills and Tools to construct evaluation context without updating evaluator parameters.The Library starts empty and grows through self-evolution, while the Orchestrator selects relevant artifacts for the frozen Sub-Agent.
Problem formulation: Given a source image, instruction, and K candidates, the system produces 1–5 ordinal scores and a ranking, using only relative score order for ranking accuracy.Equal scores are treated as ties; downstream GRPO uses the generated edit as the sole candidate and batch-normalizes its resulting score.
Skills and Tools Library: Skills provide declarative rubrics and examples, whereas Tools specify targeted visual-analysis procedures, invocation conditions, inputs, outputs, and execution steps.Tools let a general-purpose VLM temporarily act as a specialized expert, while Skills define assessable evaluation criteria.
Orchestrator: The Orchestrator routes multimodal inputs to relevant Library entries, progressively discloses Tool details, and analyzes reasoning-chain errors to propose updates.It first considers Tool names and descriptions, then loads full schemas when invocation conditions are met.
Sub-Agent: The frozen, pluggable Sub-Agent applies Skills to every candidate and optionally executes applicable Tools before producing a structured reasoning chain and preference judgment.The default configuration is Qwen2.5-VL-7B-Instruct, with Gemini evaluated as a drop-in replacement.
Self-evolution loop: 62.5% validation accuracy was achieved by the final 3-Skill, 4-Tool Library, a 47% relative improvement over the 42.5% empty-library baseline.The self-evolution loop uses N = 100 human demonstrations, split into 60 training and 40 held-out validation examples, and selects the best validation Library without further updates.

3 Experiments

RewardHarness is evaluated on established image-editing preference benchmarks and as a reward signal for downstream GRPO fine-tuning, using frozen Sub-Agents and libraries evolved from only 100 preference examples. It achieves strong benchmark accuracy, improves FLUX.2-klein-base-4B editing performance, and benefits from pluggable Sub-Agents and iterative library pruning.

Evaluation setup: RewardHarness is evaluated on EditReward-Bench at K=2, 3, and 4 and on GenAI-Bench for instruction-guided image-editing preference judgment.The default Sub-Agent is a frozen Qwen2.5-VL-7B-Instruct model served via vLLM, with no evaluator or Sub-Agent parameter updates during evolution.
Main results: 45.7 average accuracy with a frozen Qwen2.5-VL-7B Sub-Agent surpasses all listed baselines, including EditReward (MiMo) at 44.1.The result uses 100 preference examples sampled from the EditReward training set without updating the underlying VLM parameters.
Main results: +15.4 points over the frozen Qwen2.5-VL-7B baseline raises performance from 30.3, while the Qwen-based system reaches 67.5 accuracy on GenAI-Bench.Each library is evolved from 100 EditReward training-split examples, yet the resulting Skills and Tools generalize beyond the evolution data.
Pluggable Sub-Agent: 47.4 best overall average accuracy is achieved with Gemini-2.0-Flash as a pluggable Sub-Agent, with best EditReward-Bench performance at K=2 and tied-best performance at K=4.This demonstrates that gains are not tied to a single VLM backbone and that the framework can use stronger VLMs.
Reward-driven editing improvement: 3.52 overall ImgEdit-Bench score results from GRPO fine-tuning with RewardHarness, improving FLUX.2-klein-base-4B from 3.32 and matching Flux.1 Kontext [dev].Under the same GRPO setup, EditReward reaches 3.45; category trade-offs differ between the two reward signals.
Self-evolution dynamics: 62.5% validation accuracy is reached at iteration 69 after pruning, using a final library of 7 entries comprising 3 Skills and 4 Tools.Across 77 iterations, accuracy first plateaus at 52.5% with 13 entries before improving after pruning begins around iteration 50.

4 Related Work

RewardHarness addresses the data-efficiency gap in visual reward modeling by learning from approximately 100 demonstrations instead of tens of thousands of human preference comparisons. It extends self-evolving and tool-augmented agent paradigms by evolving a reusable Skills-and-Tools Library while keeping the base VLM frozen.

Reward models for visual generation: Existing visual reward models rely on supervised fine-tuning from tens of thousands of human preference comparisons, whereas RewardHarness learns from only ∼100 demonstrations through explicit library evolution.The comparison includes ImageReward, PickScore, VisionReward, EditReward, VideoScore2, and ImagenWorld.
Self-evolving agents: Prior self-evolving agents keep model weights fixed while evolving prompts, memories, or reusable skills for individual agent tasks.Referenced methods include Reflexion, ExpeL, Voyager, SkillRL, and EvolveCoder.
Self-evolving agents: RewardHarness specializes self-evolution for multimodal reward modeling by maintaining a composable Skills-and-Tools Library as reusable evaluator context.This shifts evolution from reasoning for a single agent task to a reusable evaluation library.
Tool-augmented LLMs: Unlike tool-augmented LLM work that learns when to invoke a fixed tool set, RewardHarness iteratively creates and refines the Skills and Tools for the target evaluation domain.The base VLM remains frozen throughout this process.

5 Limitation

RewardHarness remains dependent on a proprietary Orchestrator and has only been evaluated for instruction-guided image-editing preferences. Its small held-out validation set may limit library robustness, motivating diversity-aware sampling, uncertainty estimates, and lightweight human audits.

Proprietary Orchestrator: The Orchestrator relies on proprietary Claude for routing, chain analysis, and library evolution, limiting reproducibility and creating API-availability and cost dependencies.The Sub-Agent is pluggable, with Qwen2.5-VL-7B and Gemini demonstrated as drop-in choices, but open-source Orchestrator alternatives remain unvalidated.
Unexplored Domains: The domain-agnostic Skills and Tools framework remains unexplored on text-to-image quality, video editing, and 3D scene manipulation.Its effectiveness outside instruction-guided image-editing evaluation has not yet been established.
Validation and Deployment: The evolution loop optimizes validation accuracy on a small held-out set, potentially limiting the learned Library’s robustness.Suggested safeguards include diversity-aware sampling, uncertainty estimates, and lightweight human audits, particularly for contested or safety-sensitive preferences.

6 Conclusion … A.3 Tool-augmented Large Language Models

REWARDHARNESS reframes reward modeling as context evolution, refining Skills and Tools while keeping the underlying VLM frozen. The paper situates this approach against large-scale preference-trained reward models, self-evolving agents, and tool-augmented LLMs with fixed tool sets.

6 Conclusion: 47.4% average accuracy on editing reward benchmarks is achieved by REWARDHARNESS using only 100 labeled samples.The framework iteratively refines a compact library of Skills and Tools while keeping the underlying VLM frozen.
6 Conclusion: REWARDHARNESS reframes reward modeling as context evolution rather than weight optimization.Its self-evolving framework refines textual Skills and Tools instead of updating the underlying model weights.
6 Conclusion: When used as a reward signal in GRPO fine-tuning, REWARDHARNESS outperforms the baselines described in the conclusion passage.The supplied passage reports this result but does not include the corresponding numerical value or comparison target.
A.1 Reward Models for Visual Generation: Visual-generation reward models include ImageReward, PickScore, VisionReward, UnifiedReward, OneReward, VideoScore, EditReward, and EditScore.These methods cover image generation, video generation, and image editing.
A.1 Reward Models for Visual Generation: Representative visual reward models rely on supervised learning from large-scale human preference data, often ranging from tens of thousands to hundreds ofThe supplied passage truncates the upper end of this range.
A.2 Self-evolving Agents: Self-evolving agents improve either by updating model parameters or by refining textual artifacts such as instructions and context.Weight-based examples include STaR, SPIN, and self-rewarding language models, while instruction- and context-based approaches keep model weights fixed.
A.3 Tool-augmented Large Language Models: Tool-augmented LLMs typically assume a fixed tool set and focus learning on the invocation policy.Representative systems include ReAct, Gorilla, ToolLLM, ReTool, ToolkenGPT, and CoALA.
A.3 Tool-augmented Large Language Models: REWARDHARNESS inverts the standard tool-augmented emphasis by keeping base models frozen while iteratively creating and refining Skills and Tools.This contrasts with approaches that primarily learn when and how to invoke externally provided tools.

B Additional Experiments and Analyses · B.1 Data Efficiency Comparison · B.2 Additional Qualitative Examples

The additional analyses contrast RewardHarness with conventional reward-model training in data efficiency and report qualitative comparisons across five editing categories. RewardHarness uses roughly 100 preference demonstrations while avoiding conventional data collection and fine-tuning stages.

B Additional Experiments and Analyses: RewardHarness reframes the comparison around data-efficient reward construction rather than conventional large-scale reward-model training.The subsection explicitly contrasts the two paradigms side by side.
B.1 Data Efficiency Comparison: Conventional reward modeling requires a large-scale human preference dataset before reinforcement-learning alignment can begin.The dataset is used to train a reward model via supervised fine-tuning.
B.1 Data Efficiency Comparison: The conventional pipeline trains its reward model through supervised fine-tuning before any RL alignment takes place.This ordering is described as part of the standard approach.
B.1 Data Efficiency Comparison: The conventional approach is characterized as expensive, slow, and infeasible for black-box API models.These limitations are attributed to its data-collection and fine-tuning requirements.
B.1 Data Efficiency Comparison: ∼100 preference demonstrations are sufficient for RewardHarness to operate without the conventional data-collection and fine-tuning stages.The supplied passage states that RewardHarness eliminates both stages.
B.2 Additional Qualitative Examples: Figure 7 presents additional qualitative comparisons across five editing categories: Add, Adjust, Extract, Remove, and Replace.Each row corresponds to a different editing category.
B.2 Additional Qualitative Examples: The qualitative comparison includes the input image, a FLUX.2-klein-base-4B output, and RL-fine-tuned RewardHarness and EditReward variants.These outputs are shown for each editing category in Figure 7.

B.3 Evolution Trajectory

Figure 6 tracks the co-evolution of validation accuracy and library size across 77 self-evolution iterations using the Gemini-2.0-Flash Sub-Agent configuration with the best average accuracy in Table 1.

Evolution Trajectory: Across 77 iterations, Figure 6 plots validation accuracy and library size as the self-evolution loop progresses on Gemini-2.0-Flash.The configuration achieves the best average accuracy in Table 1.

B.4 Library Case Study

The library grows, prunes, and shifts from heuristic skills toward grounded visual-verification tools, while Orchestrator analysis drives targeted fixes for recurring Sub-Agent failures without additional human annotation. The final iteration-69 library contains 3 skills and 4 tools and achieves val acc = 0.625.

Library composition: At iteration 69, the exported library contains 3 skills and 4 tools, making it leaner than the iteration-49 peak of 13 entries.The library peaks at 8 skills and 5 tools at iteration 49, then self-prunes to 7 total entries.
Library composition: At iteration 69, the final configuration reaches val acc = 0.625 despite being smaller than the mid-point peak.Tools outnumber skills in the final library, reflecting a shift toward grounded visual verification.
Final library contents: Final skills encode declarative evaluation heuristics, while tools provide procedural grounding through targeted secondary VLM calls.Examples include describing before judging and allowing surrealism when prompted.
Evolution narrative: The Sub-Agent repeatedly hallucinates visual content, struggles with cultural and style knowledge, and over-penalizes explicitly requested surreal edits.Failures include misreading text, fabricating object presence, and claiming images are completely black.
Evolution narrative: Each evolution step targets observed failures by adding OCR, VQA, anti-hallucination, stronger tool-invocation, and tie-handling guidance.The loop compares model reasoning traces with human labels and requires no additional human annotation.

B.5 Skill and Tool Case Studies

The Library evolves toward targeted guidance through refined skills and structured Tool invocation. Case studies show prompt-aware realism penalties, mandatory verification for hallucination-prone queries, and typed-schema grounding for spatial analysis.

Library Evolution: No zoom-in, crop, or close-up skills emerged; spatial and detail-level queries were instead delegated entirely to the spatial-and-object-analyzer Tool.The Tool routes precise counting and layout questions to a structured VLM call.
Case 1: Skill Refinement: The realism-and-artifact-penalties skill evolved from broadly penalizing unrealistic outputs to allowing prompt-requested surreal content while still penalizing genuine artifacts.The refinement prevents false penalties for prompts such as “polar bears in a grassy savannah.”
Case 2: Tool-Invocation Guidance: The anti-hallucination-and-verification skill mandates Tool use for black-image detection, text reading, and object-attribute verification before judgment.This creates a Skills→Tools handoff: the Skill specifies when to call a Tool, while the Tool specifies how.
Case 3: Structured Tool Schema: The spatial-and-object-analyzer Tool handles spatial, counting, and orientation queries through a secondary VLM call with a typed JSON schema.It grounds queries such as “the 4th surfboard from the left” and “is the cat’s foot under the bag?”