Source-linked AI summary
Trust Your Critic: Robust Reward Modeling and Reinforcement Learning for Faithful Image Editing and Generation
Xiangyu Zhao, Peiyuan Zhang, Junming Lin, Tianhao Liang, Yuchen Duan, Shengyuan Ding, Changyao Tian, Yuhang Zang, Junchi Yan, Xue Yang
TL;DR
Existing RL pipelines rely on critics that can hallucinate and assign noisy scores, limiting reliable optimization for image editing and T2I generation. FIRM builds task-specific reward models, human-annotated evaluation, and Base-and-Bonus reward formulations, achieving stronger human alignment and substantial downstream gains.
Problem
Current MLLM reward critics for image editing and generation suffer from hallucinations, object neglect, imprecise spatial reasoning, and noisy reward scores.
Method
FIRM constructs specialized editing and generation datasets and reward models, evaluates them on FIRM-Bench, and uses CME and QMA Base-and-Bonus rewards in RL.
Results
FIRM reward models achieve superior human alignment, while FIRM-Qwen-Edit and FIRM-SD3.5 deliver substantial performance gains across editing and generation.
Takeaways & Limitations
Accurate, task-specific critics can guide RL toward more faithful image editing and precise image generation.
Abstract
from arXiv · showhide
Reinforcement learning (RL) has emerged as a promising paradigm for enhancing image editing and text-to-image (T2I) generation. However, current reward models, which act as critics during RL, often suffer from hallucinations and assign noisy scores, inherently misguiding the optimization process. In this paper, we present FIRM (Faithful Image Reward Modeling), a comprehensive framework that develops robust reward models to provide accurate and reliable guidance for faithful image generation and editing. First, we design tailored data curation pipelines to construct high-quality scoring datasets. Specifically, we evaluate editing using both execution and consistency, while generation is primarily assessed via instruction following. Using these pipelines, we collect the FIRM-Edit-370K and FIRM-Gen-293K datasets, and train specialized reward models (FIRM-Edit-8B and FIRM-Gen-8B) that accurately reflect these criteria. Second, we introduce FIRM-Bench, a comprehensive benchmark specifically designed for editing and generation critics. Evaluations demonstrate that our models achieve superior alignment with human judgment compared to existing metrics. Furthermore, to seamlessly integrate these critics into the RL pipeline, we formulate a novel "Base-and-Bonus" reward strategy that balances competing objectives: Consistency-Modulated Execution (CME) for editing and Quality-Modulated Alignment (QMA) for generation. Empowered by this framework, our resulting models FIRM-Qwen-Edit and FIRM-SD3.5 achieve substantial performance breakthroughs. Comprehensive experiments demonstrate that FIRM mitigates hallucinations, establishing a new standard for fidelity and instruction adherence over existing general models. All of our datasets, models, and code have been publicly available at https://firm-reward.github.io.
1 Introduction
FIRM addresses unreliable reward critics in RL for image editing and T2I generation by building task-specific data, benchmarks, reward models, and reward strategies. Experiments report stronger human alignment and substantial performance gains in downstream tasks.
- Current MLLM critics often hallucinate, neglect objects, and lack precise spatial reasoning, producing noisy reward scores for image editing and generation.
- FIRM uses specialized difference-first editing and checklist-guided generation pipelines to create FIRM-Edit-370K and FIRM-Gen-293K and train corresponding 8B reward models.
- FIRM-Bench is a fully human-annotated benchmark for editing and generation, where the reward models show superior human alignment compared with existing models.
- The Base-and-Bonus strategy combines CME for editing and QMA for generation to balance competing rewards and mitigate reward hacking.
- FIRM-Qwen-Edit and FIRM-SD3.5 achieve substantial and consistent improvements in faithful editing and precise generation through extensive RL experiments.
2 Related Works
Related work traces T2I generation and image editing from earlier generative paradigms toward diffusion, flow-based, and autoregressive models. RL now optimizes these systems, but general-purpose MLLMs remain inadequate as precise image-editing and generation rewards.
- T2I generation has progressed from GANs and VAEs to diffusion, flow-based, and autoregressive architectures.
- Image editing evolved from dual-prompt diffusion methods toward large-scale curated datasets, flow matching, and sequential autoregressive approaches.
- RL methods such as DDPO and DPOK formulate diffusion denoising as a multi-step MDP and optimize policies with PPO.
- Recent approaches combine RL with CoT reasoning or MLLM-based rewards, but general-purpose MLLMs frequently fail to provide reliable feedback for image generation and editing.
3 Method
FIRM builds task-specific reward-model data and critics through separate editing and generation pipelines, then integrates them into RL with reward designs intended to prevent shortcut behavior.
- FIRM-Edit Pipeline: Models often identify fine-grained editing details better as problem-solvers than evaluators, producing mismatched final scores.
- FIRM-Edit Pipeline: The FIRM-Edit pipeline uses difference-first analysis of original-edited image pairs to identify obvious and detailed modifications for edit evaluation.
- FIRM-Edit Pipeline: Editing rewards separately measure execution accuracy and consistency preservation, with both dimensions scored from 1 to 5.
- FIRM-Gen Pipeline: The FIRM-Gen pipeline first plans a customized checklist, then performs structured analytical scoring across prompt-specific visual criteria.
- Benchmark and Data: FIRM samples diverse prompts and generated images, and evaluates reward models on an 807-sample benchmark with separate editing and generation subsets.FIRM-Bench contains 301 execution samples, 256 consistency samples, and 250 instruction-following samples.
- RL Reward Design: A linear editing reward caused reward hacking toward nearly unchanged images, whereas CME makes execution necessary for high reward and uses consistency as a shaping signal.The initial equal-weight design used w1 = w2 = 0.5; CME uses w1 = 0.6 and w2 = 0.4.
- RL Reward Design: For generation, QMA treats quality as a constraint and emphasizes it after instruction-following scores become high, mitigating black-shadow shortcut behavior.QMA uses w1 = 0.4 and w2 = 0.6.
4 Experiment Results
Experiments show that FIRM reward models align well with human judgments and guide strong RL performance in image editing and generation. CME further mitigates reward hacking during editing optimization.
- FIRM-Bench evaluation: FIRM-Edit-8B achieves the lowest Execution MAE of 0.53 on FIRM-Bench-Edit and an overall MAE of 0.62, surpassing GPT-5 and open-source baselines.Its Consistency MAE is 0.73.
- FIRM-Bench evaluation: FIRM-Gen-8B achieves an MAE of 0.51 on FIRM-Bench-Gen, surpassing GPT-5 and all evaluated open-source models except Gemini-3-Pro.Gemini-3-Pro achieves the lowest MAE of 0.40.
- Image editing via RL: FIRM-Qwen-Edit reaches 7.84 on GEditBench and 4.42 on ImgEdit after a single RL stage guided by FIRM-Edit-8B.The model uses 2,400 training samples, compared with 27K for UniWorld-Qwen-Image-Edit.
- Image editing via RL: FIRM-Edit-8B improves over Qwen3-VL reward models by +0.30 on GEditBench and +0.07 on ImgEdit.Qwen3-VL-32B instead causes a -0.07 change on ImgEdit.
- Image generation via RL: FIRM-SD3.5 achieves 0.77 on GenEval, 87.16 on DPGBench, 77.12 on TIIF, and 69.56/76.22 on UniGenBench-Short/Long.These results outperform heavily resourced models such as BAGEL and OmniGen2.
- Reward-formulation ablation: CME uniquely produces a consistent and robust increase in reward signals while other reward formulations suffer performance degradation or reward hacking.A weighted-sum formulation can reward unchanged images despite poor instruction following.
5 Conclusion
FIRM presents an end-to-end framework for faithful image editing and generation, spanning data curation, reward datasets, evaluation, reward models, and reward formulations.
- FIRM integrates tailored data curation pipelines, high-quality reward datasets, a rigorous benchmark, robust reward models, and novel reward formulations.
- The framework uses its reward models to guide reinforcement learning for downstream generative tasks.
- The study reports substantial performance improvements and validates accurate critics as indispensable to reinforcement learning for generative models.
A Statistics Analysis of the FIRM Dataset and Benchmark
This section analyzes sample distributions across score ranges for the FIRM datasets and benchmarks, with the counts presented in Table 6.
- Table 6 reports sample counts across score levels for the FIRM datasets and benchmarks.
- The analysis compares score-range distributions between the FIRM dataset and the benchmark.
- The sample-distribution analysis is presented as a table-based comparison.
B Analysis of Reward Function Hacking
Fixed weighted-sum rewards create a trade-off between consistency and execution, enabling reward hacking; CME instead couples the objectives multiplicatively.
- R = 0.5 Consistency + 0.5 Execution initially causes reward hacking by favoring consistency over task execution.
- R = 0.4 Consistency + 0.6 Execution partially improves execution but under-emphasizes consistency with the source image.
- Figure 8 shows that optimizing a fixed weighted sum induces an undesirable trade-off between consistency and execution.
- CME introduces multiplicative coupling to encourage strong execution while preserving alignment with the source image.
C Prompts for FIRM-Edit pipeline
The FIRM-Edit pipeline uses staged prompts to identify image differences, evaluate execution and consistency separately, and produce structured scores for edit assessment.
- Difference analysis: The pipeline prompts an image comparator to identify obvious and detailed differences between original and modified images.
- Judgment prompts: The execution evaluator uses a 1-to-5 scale and returns concise reasoning with a score.
- Judgment prompts: The consistency prompt excludes instructed occlusions and minor image-quality variations from inconsistency judgments.
- Judgment prompts: Separate evaluators assess editing execution and consistency against the edit instruction and original image.
D Prompts for FIRM-Gen pipeline
The FIRM-Gen curation pipeline uses structured prompts to create analysis plans and evaluate generated images for instruction following. Evaluators inspect images against explicit criteria and assign objective ratings on a five-point scale.
- Analysis-plan generation: The pipeline generates an Analysis Plan by decomposing each text-to-image prompt into verifiable visual criteria.The plan covers subjects, attributes or actions, style or composition, and optional negative constraints.
- Analysis-plan generation: The Analysis Plan instructs evaluators to assess whether each criterion is satisfied by the generated image.Evaluators analyze every listed point and provide a brief assessment of whether the image meets or exceeds expectations.
- Instruction-following evaluation: Evaluators produce a structured textual report with criterion-level analyses, a final analysis, and a final alignment rating.The prescribed output includes a rating from 5 to 1, with the boxed rating restricted to those five values.
- Instruction-following evaluation: The prompts require objective checklist-based scoring rather than an unconstrained judgment.The original prompt and analysis plan are supplied in marked sections, followed by the evaluator’s assessment.
- Instruction-following evaluation: Instruction-following evaluation identifies explicit and implicit prompt requirements, including subjects, attributes, actions, relationships, style, composition, and negative constraints.The evaluator first inspects the image, then performs a step-by-step comparison against the original prompt or analysis plan.