Source-linked AI summary
Can We Perform Online RL for Image Editing without Editing Rewards?
Qichao Ma, Jikang Cheng, Ling Liang, Zhaofei Yu, Tiejun Huang, Renye Yan
TL;DR
Image-editing RL lacks mature editing-specific rewards because they require costly triplet supervision and task-dependent calibration. Lever-Edit learns reward-aligned target captions and uses frozen T2I rewards for policy optimization, achieving competitive edit alignment and source preservation while outperforming intuitive transfer baselines.
Problem
Editing-specific rewards require costly triplet supervision and task-dependent calibration, motivating whether mature T2I rewards can support image-editing RL instead.
Method
Lever-Edit learns reward-aligned counterfactual target captions in one stage, then freezes the captioner and optimizes the editing policy using only transferred T2I rewards.
Results
Lever-Edit achieves competitive editing alignment and source preservation while outperforming intuitive transfer baselines without relying exclusively on editing-specific rewards.
Takeaways & Limitations
Effective image-editing RL does not necessarily require separately designed rewards for every editing task.
Takeaways & Limitations
Generic semantically valid captions may be incompatible with a frozen T2I reward because their representations can be poorly calibrated to its evaluation space.
Abstract
from arXiv · showhide
Reinforcement learning (RL) enables direct preference optimization for image editing through editing-specific rewards, which remain less developed due to costly triplet supervision and complex task-dependent calibration. In contrast, text-to-image (T2I) generation benefits from a mature and diverse reward ecosystem spanning semantic alignment, aesthetics, realism, glyph shape, and other visual preferences. Extending this ecosystem to image editing would substantially broaden the range of visual preferences accessible to RL-based optimization, prompting the central question: \emph{Can We Perform Image Editing RL without Editing Rewards?} In this paper, we argue that the standard image editing dimensions have potential to be mapped to the T2I reward space: image quality can transfer directly, prompt following can be aligned through a description of the desired visual state, and reference consistency admits a coarse semantic conversion by encoding the source content to preserve. However, editing instructions specify relative changes, whereas T2I rewards require self-contained target descriptions; moreover, semantically valid captions from generic vision-language models may be incompatible with the frozen reward. Hence, we further introduce Lever-Edit, a two-stage framework that learns a reward-aligned captioner for counterfactual target descriptions, freezes it, and optimizes the editing policy solely with the transferred T2I reward. Experiments show competitive editing alignment and source preservation against editing-reward-based fine-tuning, while outperforming intuitive transfer baselines.
Introduction
The introduction asks whether image-editing RL can use mature T2I rewards instead of costly editing-specific rewards. It proposes Lever-Edit, which converts reference-conditioned instructions into reward-aligned target captions for policy optimization with a frozen T2I reward.
- Motivation: Editing RL must balance instruction execution with preservation of unrelated source content, while costly triplet supervision and task-dependent calibration complicate reward development.Editing quality is characterized by image quality, edit execution, and reference consistency.
- Motivation: T2I-to-editing reward transfer could expose editing RL to mature rewards for semantic alignment, aesthetics, realism, OCR, safety, and other visual preferences.The transfer could also avoid dependence on costly editing-evaluator development and deployment.
- Challenge: Direct transfer is difficult because editing instructions specify relative transformations, whereas T2I rewards require self-contained target descriptions that preserve unchanged source semantics.Using the instruction alone captures the requested change but omits source content that should remain unchanged.
- Method: Lever-Edit uses a two-stage framework in which a prefix-tuned VLM generates counterfactual target captions, then a frozen T2I reward supervises editing-policy fine-tuning.The captioner maps a reference image and editing instruction to a reward-aligned target caption.
- Results: Transferred T2I rewards achieve competitive editing alignment and source preservation without an image-to-image reward model during policy fine-tuning.The paper presents this result as empirical evidence supporting image-editing RL without an editing-specific reward.
Related Works
Related work establishes online RL as a general post-training framework for visual generation, while T2I reward modeling offers diverse alignment criteria. Image-editing RL rewards remain narrower, focusing mainly on edit execution and reference consistency, which limits optimization potential.
- Reinforcement Learning: DDPO and DPOK pioneered RL-based fine-tuning for text-to-image diffusion models, while DanceGRPO and Flow-GRPO extended online RL to contemporary diffusion and flow-matching backbones.These studies established online RL as a general post-training framework for visual generation.
- Image Generation: T2I reward modeling spans broad semantic alignment and visual appeal, alongside specialized criteria for photorealism and visual-text fidelity.Examples include ImageReward, HPS, PickScore, RealGen, and TextPecker.
- Image Editing: Image-editing rewards largely center on edit execution and reference consistency, with limited coverage of broader alignment preferences supported by T2I rewards.This restricted reward space constrains the optimization potential of image-editing RL.
Problem Formulation
The problem is to convert relative editing instructions and reference images into self-contained target descriptions that a frozen T2I reward can evaluate. This transfer can represent core editing semantics, but not exact low-level preservation, and intuitive converters expose conditioning, overhead, and reward-compatibility gaps.
- The Reward-Transfer Gap: A T2I reward cannot directly consume an editing instruction because relative changes depend on the reference image and omit the complete target state and preserved source semantics.For example, “make the car red” specifies a transformation rather than a self-contained description of the resulting image.
- The Reward-Transfer Gap: Reward transfer therefore requires a converter g : I × Q → C that maps the reference image and relative instruction to a self-contained target description.The edited result is then evaluated by the frozen T2I reward using that converted condition.
- Mapping Editing Quality to the T2I Reward Space: Editing quality comprises image quality, edit execution, and reference consistency, and each criterion admits a correspondence to the T2I reward space without requiring a fixed scalarization.Image quality transfers directly because it is intrinsic to the edited image; execution and consistency require reference-conditioned information.
- Mapping Editing Quality to the T2I Reward Space: Expressing the intended outcome as an absolute post-edit description converts edit execution into image–text alignment, while preserved source semantics can be encoded alongside requested changes.A unified target description jointly evaluates execution and consistency through a T2I reward.
- Mapping Editing Quality to the T2I Reward Space: This mapping is a semantic projection rather than an equivalence: text captures principal editing semantics but cannot fully encode low-level identity or pixel-accurate preservation.Thus, direct transfer is ill-posed at the interface level even though core editing objectives remain transferable in principle.
- Intuitive Reward-Transfer Protocols: Direct Edit Prompt sets c_tgt = q_edit with no conversion cost, but omits reference-specific context and explicit preservation content by treating a relative instruction as an absolute description.It retains the requested transformation while failing to specify a complete target state.
- Intuitive Reward-Transfer Protocols: Online VLM rewriting adds reference awareness and fuller post-edit descriptions, but repeated inference incurs overhead and generic captions are not calibrated to the selected T2I reward.Its semantic descriptions are generated for general understanding rather than the evaluation behavior of the frozen reward.
- Intuitive Reward-Transfer Protocols: Offline proprietary-MLLM rewriting can express edits and preserved content accurately for humans, yet remains reward-inflexible, closed-model-dependent, and potentially incompatible with the frozen reward’s conditioning distribution.Human-semantic accuracy does not guarantee that the description emphasizes attributes represented strongly by the reward.
Proposed Method
Lever-Edit uses a two-stage pipeline to transfer multiple T2I rewards to image editing without modifying the editing backbone or VLM. It first learns reward-aligned query embeddings with semantic regularization, then reuses them as a plug-and-play module across editing backbones and fine-tuning paradigms.
- Stage 1: The fixed prompt describes the hypothetical edited image, combining discrete task specification with continuous query-conditioned adaptation.This design preserves the underlying generation task while restricting reward-driven changes to a lightweight prefix space.
- Stage 1: Multiple complementary T2I reward models evaluate sampled target descriptions, whose weighted outputs define the downstream optimization signal.Because target descriptions are sampled as discrete tokens, the query embeddings are optimized with policy gradient.
- Stage 1: Offline MLLM-generated reference descriptions provide token-level semantic supervision that regularizes reward optimization and mitigates semantic drift and reward hacking.The KL regularizer reduces to token-level negative log-likelihood because the target distribution is one-hot.
- Stage 1: Stage 1 freezes the image-editing backbone and VLM while prefix tuning learnable query embeddings against downstream T2I reward preferences.A fixed system prompt specifies the target-description task, while only Equery is optimized.
- Stage 2: In Stage 2, the learned query embeddings become a plug-and-play reward-alignment module usable across different image-editing backbones and fine-tuning paradigms.The corresponding VLM prefill embedding is contextualized for reuse.
Experiments
Experiments evaluate whether transferred T2I rewards can supervise image editing without editing-specific rewards. Lever-Edit improves reward alignment, instruction execution, semantic and source consistency, and image quality over unaligned and prompt-transfer baselines.
- Experimental Setup: Experiments use Lever-Bench, a failure-focused benchmark containing 1,880 curated image-instruction pairs from multiple editing sources.The evaluation also includes the public GEdit benchmark.
- Experimental Setup: HPSv3 and RealGen provide T2I training rewards covering edit execution, consistency, aesthetics, realism, and image quality.Evaluation separately measures perceptual quality, semantic consistency with SigLIP, and instruction execution and consistency with EditScore.
- Reward Transfer: Reward-aligned HPSv3 provides a stronger editing-preference signal than raw edit instructions and improves fine-tuning of the image-editing model.The reward-level comparison includes HPSv3+Ours, HPSv3+Edit, and the editing-specific EditScore on annotated preference pairs.
- Downstream Editing Quality: Lever-based methods consistently outperform the unaligned base model on EditScore Execution while also providing strong semantic consistency.These results show that T2I-only supervision improves core edit-aware dimensions rather than only the optimized T2I reward values.
- T2I Preference Alignment: Lever-Edit achieves the highest HPSv3 score and best RealGen score on Lever-Bench, demonstrating strongest alignment with transferred T2I quality preferences.The HPSv3 gain reflects the learned target descriptions’ role in bridging editing and the T2I preference space.
- Training Dynamics and Qualitative Results: Across training stages, Lever-Edit generally achieves the best aesthetics, semantic alignment, edit consistency, and realism, with edited images better matching intended target semantics.Qualitative comparisons further show strong prompt following, aesthetics, semantic and source consistency, and naturalness.
Conclusion
The paper shows that mature T2I rewards can support image-editing RL without relying exclusively on editing-specific rewards. Lever-Edit bridges relative editing instructions and self-contained T2I target descriptions through a two-stage reward-aligned policy.
- Mature T2I rewards can be transferred to image-editing RL without exclusive reliance on editing-specific rewards.
- Lever-Edit first learns a reward-aligned target-description policy and then freezes it to guide downstream image-editing optimization.The framework addresses the mismatch between relative editing instructions and the self-contained descriptions expected by T2I rewards.
- Experiments evaluate the approach on Lever-Bench and existing benchmarks.