PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning an Image Editing Model without Image Editing Pairs

Nupur Kumari, Sheng-Yu Wang, Nanxuan Zhao, Yotam Nitzan, Yuheng Li, Krishna Kumar Singh, Richard Zhang, Eli Shechtman, Jun-Yan Zhu, Xun Huang

arXiv:2510.14978v2cs.CVcs.LG

TL;DR

Image editing models depend on paired examples that are difficult to curate at scale. This paper trains a few-step editor without pairs using VLM feedback and distribution matching; it reports competitive few-step performance against paired-data models and stronger results than SFT and SFT+RL with the same VLM reward model.

  • Problem

    Image editing requires precise control from a reference image, but supervised input-target edits are difficult to collect at scale.

    Source: §1 · p. 1

  • Method

    NP-Edit trains a few-step editor without paired supervision by unrolling diffusion and combining VLM feedback with distribution matching.

    Source: §1 · p. 2, §4.2 · p. 4

  • Results

    Without paired supervision, the method is competitive with paired-data baselines in few-step generation and surpasses SFT and SFT+RL using the same Llava-OneVision reward model.

    Source: §5.3 · p. 10, §6 · p. 10

  • Takeaways & Limitations

    The paper establishes a pair-free training paradigm that retains competitive performance for efficient few-step image editing.

    Source: §6 · p. 10

    Without pixel-level supervision, edits may alter fine-grained details or fail to fully preserve subject identity; the method also depends on VLM capabilities and requires GPU memory for the VLM.

    Source: §6 · p. 10

Abstract

from arXiv · show

Recent image editing models have achieved impressive results while following natural language editing instructions, but they rely on supervised fine-tuning with large datasets of input-target pairs. This is a critical bottleneck, as such naturally occurring pairs are hard to curate at scale. Current workarounds use synthetic training pairs that leverage the zero-shot capabilities of existing models. However, this can propagate and magnify the artifacts of the pretrained model into the final trained model. In this work, we present a new training paradigm that eliminates the need for paired data entirely. Our approach directly optimizes a few-step diffusion model by unrolling it during training and leveraging feedback from vision-language models (VLMs). For each input and editing instruction, the VLM evaluates if an edit follows the instruction and preserves unchanged content, providing direct gradients for end-to-end optimization. To ensure visual fidelity, we incorporate distribution matching loss (DMD), which constrains generated images to remain within the image manifold learned by pretrained models. We evaluate our method on standard benchmarks and include an extensive ablation study. Without any paired data, our method performs on par with various image editing diffusion models trained on extensive supervised paired data, under the few-step setting. Given the same VLM as the reward model, we also outperform RL-based techniques like Flow-GRPO.

1 INTRODUCTION

Image editing needs image-guided control, but suitable paired training data is difficult to obtain at scale. The paper proposes pair-free training with VLM feedback and distribution matching, and studies how VLMs and dataset scale affect performance.

1 INTRODUCTION

  • Text-only prompts provide limited control for tasks such as changing local attributes, style, or object context, motivating image-guided editing.Such operations require precise control based on an existing image.Source: §1 · p. 1
  • Supervised editing data is difficult to scale: synthetic pairs risk becoming outdated and propagating artifacts, while video-derived and manual pairs have important constraints.Video sequences offer limited transformation diversity and rarely provide pixel-aligned edits; manual collection is laborious.Source: §1 · p. 1
  • NP-Edit trains an image editor without paired supervision by using VLM gradient feedback to assess instruction following and distribution matching to maintain realistic outputs.The framework aims to generalize to arbitrary images and edit instructions.Source: §1 · p. 2
  • The formulation combines VLM feedback with distribution matching to learn an efficient few-step editor that remains competitive with supervised-data baselines.The paper also examines VLM backbones, dataset scale and diversity, and VLM loss formulation; it reports better performance with more powerful VLMs and larger datasets.Source: §1 · p. 2

2 RELATED WORKS

Prior image-editing methods include inference-time and optimization-based approaches, followed by trained editors that still face challenges obtaining paired data. This work relates its pair-free, differentiable VLM-feedback approach to post-training and few-step diffusion research.

Diffusion-based image editing

  • Training-based editors aim to improve efficiency and robustness, but synthetic pairs can become outdated, human annotation is costly, and video or simulation data remain limited in diversity or realism.Earlier approaches also include zero-shot inference-time editing and flexible but slow optimization.Source: §2 · p. 2

Post-training for image generation

  • VLM-as-a-judge post-training has been explored for image generation, while RL-based editing approaches often require paired-data SFT initialization; this method instead uses differentiable VLM feedback.The authors state that their approach avoids an initial SFT stage and synthetic training data.Source: §2 · p. 2
  • Luo et al. (2025) use VLM gradient feedback for image-generation criteria, but their per-example framework requires costly LoRA fine-tuning and does not address editing.The criteria include horizon line, style, and layout.Source: §2 · p. 3

Few-step diffusion models

  • Few-step diffusion methods reduce sampling steps through several strategies, while this framework uses DMD to keep editing outputs on the pretrained teacher's real-image manifold.VLM feedback separately guides instruction following.Source: §2 · p. 3

3 BACKGROUND

The background describes flow-based diffusion training and the VLM architecture used to provide visual-language feedback. In this setup, a denoiser can condition on both an edit prompt and a reference image, while the VLM predicts answers to image-related questions.

3.1 DIFFUSION MODELS

  • In the flow-based formulation, noisy samples are linearly interpolated from data toward noise, and the denoiser predicts velocity while conditioning on text, a reference image, or both.The training objective weights the prediction error by a time-dependent factor.Source: §3.1 · p. 3

3.2 VISION LANGUAGE MODELS (VLMS)

  • A VLM projects vision-encoder tokens into an LLM's word-embedding space and is trained to predict answer tokens; experiments use LLaVa-OneVision-7B with SigLIP and Qwen-2.The described training objective maximizes the probability of the correct answer.Source: §3.2 · p. 3

4 METHOD

NP-Edit trains a few-step image-editing model without ground-truth edited images, combining VLM feedback with distribution matching to keep outputs within the natural-image domain.

4.1 EDIT INSTRUCTION DATASET

  • ∼3M local-editing and ∼600K free-form reference images support instructions generated and validity-checked with Qwen2.5-32B VLM.The dataset covers local operations and more free-form customization or personalization tasks.Source: §4.1 · p. 4

4.2 TRAINING OBJECTIVE

  • Training unrolls two backward-diffusion steps from noise through a sampled intermediate timestep, avoiding a full backward trajectory.Generation uses four steps with a fixed intermediate schedule; intermediate denoised estimates support VLM feedback because VLMs can be unreliable on noisy or blurry inputs.Source: §4.2 · p. 4, Figure 1 · p. 5
  • Category-specific binary VLM questions assess both whether the edit succeeds and whether the output preserves unchanged content.The loss uses yes/no answers and includes an edit-verification question and an identity-preservation question.Source: §4.2 · pp. 4–5
  • DMD matches the fine-tuned generator’s output distribution to the pretrained text-to-image teacher, complementing VLM feedback with a natural-image fidelity constraint.The loss minimizes KL divergence between the teacher-estimated real-image distribution and the fine-tuned model’s output distribution.Source: §4.2 · p. 5

4.3 TRAINING DETAILS

  • The model concatenates the reference-image VAE encoding with noisy-target tokens, begins with identity-reconstruction warmup, then uses weighted VLM-editing and DMD losses.The auxiliary network is updated multiple times for each generator update.Source: §4.3 · pp. 5–6

5 EXPERIMENTS

Experiments evaluate local editing, customization, and ablations of the training objective, data scale, VLM size, and an RL comparison. NP-Edit is competitive in few-step settings and outperforms selected baselines without paired supervision.

5.1 LOCAL IMAGE-EDITING

  • In the few-step setting, the method achieves the best Overall and Perceptual Quality (PQ) scores against baseline methods, and remains competitive with BAGEL and FLUX.1-Kontext despite being ×6 smaller parameter-wise.Source: §5.1 · p. 8

5.2 FREE-FORM EDITING: CUSTOMIZATION

  • On DreamBooth customization, NP-Edit performs comparably to state-of-the-art methods and exceeds OminiControl while retaining object identity in more realistic samples.Other baselines fail at four sampling steps, so comparisons use eight-step baselines.Source: Table 2 · p. 8, §5.2 · p. 9, Figure 3 · p. 9

5.3 ABLATION

  • Ablations show that VLM editing loss supports instruction following and input-output consistency, while DMD prevents unrealistic outputs and training divergence.DMD-only training degrades instruction following and fails on removal; VLM-only training produces unrealistic outputs and eventually diverges. Binary cross-entropy and an explicit consistency question improve overall performance.Source: §5.3 · p. 9, Figure 4 · p. 10, Table 3 · p. 10
  • Performance improves with larger training datasets and larger VLM backbones, with the final dataset size limited by available computational resources.The reported gains suggest further data scaling could improve performance; stronger backbones include InternVL and LLaVA-OneVision.Source: §5.3 · pp. 9–10, Table 4 · p. 10
  • NP-Edit outperforms paired-data SFT and SFT followed by Flow-GRPO using the same Llava-OneVision reward model, despite requiring no paired supervision.Adding paired data before NP-Edit slightly improves pixel-level consistency, while quantitative scores remain similar.Source: §5.3 · p. 10, Table 4 · p. 10

6 DISCUSSION AND LIMITATIONS

The framework trains image editing without paired before-and-after supervision and achieves competitive performance with paired-data baselines in few-step generation. Its limitations include possible fine-detail or identity drift, dependence on VLM capabilities and biases, and GPU-memory overhead.

6 DISCUSSION AND LIMITATIONS

  • The method remains competitive with recent state-of-the-art baselines trained on paired data while enabling few-step generation.It combines differentiable VLM feedback for editing success with distribution matching for visual realism.Source: §6 · p. 10
  • Without pixel-level supervision, edits may alter fine-grained details or fail to fully preserve subject identity.Adding LPIPS similarity can alleviate this to some extent, though often at the cost of editing quality.Source: §6 · p. 10
  • The method depends on VLM capabilities and biases, and keeping the VLM in GPU memory adds VRAM overhead.The authors expect stronger, more efficient VLMs to help address this issue.Source: §6 · p. 10

A ADDITIONAL COMPARISON WITH BASELINE METHODS

Across human evaluation and additional editing benchmarks, the method is competitive with or preferred over baselines under few-step sampling. It also remains comparable to many multi-step baselines on local editing and performs comparably on free-form customization.

Human preference study

  • In 500-response pairwise comparisons per baseline on GEdit-Bench, users preferred the method over Turbo-Edit for editing success and over all baselines for perceptual quality.Editing-success preference was slight over FLUX.1-Kontext and Step-1X Edit, and at random chance against Qwen-Image-Edit.Source: A ADDITIONAL COMPARISON WITH BASELINE… · p. 17, Table 5 · p. 18

Customization or free-form editing

  • On DreamBooth customization, the method performs comparably with other baselines in the few-step regime under the reported additional metrics.These metrics include CLIPScore and TIFA for text alignment, masked DINOv2 identity alignment, and their geometric score.Source: A ADDITIONAL COMPARISON WITH BASELINE… · pp. 17–18

Local image-editing

B ABLATION STUDY

The ablations show complementary roles for VLM-based editing loss and distribution matching, while few-step sampling balances the difficulty of single-step editing against unreliable VLM feedback on noisy intermediate outputs. Removal performance also benefits from a more specific VLM verification prompt.

Training objective

  • VLM-editing-loss-only training gradually degrades image quality, supporting the complementary role of distribution matching loss.The figure also notes that general-purpose VLMs may not reliably assess subjective image fidelity.Source: Figure 6 · p. 19, B ABLATION STUDY · p. 20
  • DMD-only training performs comparably on color change but fails on removal, while VLM-based editing loss supports removal and better input-output consistency.The authors attribute color-change performance to DMD matching text-conditioned scores, and identify VLM-based editing loss as important across diverse editing instructions.Source: Figure 5 · p. 19, B ABLATION STUDY · p. 20

Sampling steps

  • Few-step sampling balances single-step editing difficulty against unreliable VLM responses to blurry, semantically ambiguous intermediate outputs in multi-step diffusion.The comparison describes lower-fidelity outputs from a single-step model; the method uses a four-step model that generates sharp images.Source: Figure 7 · p. 19, Figure 8 · p. 19, B ABLATION STUDY · p. 20

Sensitivity to prompt wording in VLM-based editing loss

  • For removal edits, a specific question asking whether the named object is present performs better than a general instruction-verification question.The authors expect prompt-level sensitivity to diminish as VLM robustness and reliability improve.Source: B ABLATION STUDY · p. 20, Figure 9 · p. 20

C LIMITATION

Without pixel-level supervision, VLM feedback does not ensure exact preservation of unedited details, and regularizers that improve alignment can reduce editing success. VLM feedback is also less reliable for edits requiring complex spatial reasoning.

C LIMITATION

  • On GEdit-Bench, LPIPS and cycle loss score 4.52 and 5.21 Overall VIEScore, respectively, versus 6.10 for Ours.LPIPS can particularly degrade editing quality for Removal; cycle loss performs better than LPIPS but still has lower edit success than Ours.Source: Figure 10 · p. 20, C LIMITATION · p. 21
  • VLM consistency questions do not enforce pixel-level alignment, so edits can deviate from the input in details or spatial alignment.Current VLMs struggle to detect subtle changes and minor details.Source: C LIMITATION · p. 20
  • NP-Edit can handle shape changes but currently fails on tasks requiring complex spatial reasoning, such as moving objects, where VLM feedback is less reliable.The authors expect instruction-following capabilities to improve as unified generative models and VLMs advance.Source: C LIMITATION · pp. 21–22, Figure 12 · p. 22

D DATASET CONSTRUCTION DETAILS

The dataset construction process uses VLMs to generate and validate editing instructions and edited-image captions, then filters examples before assembling local-editing and customization data.

D DATASET CONSTRUCTION DETAILS

  • The pipeline prompts Qwen-2.5-32B to generate editing instructions for reference images, then checks instruction validity and proposes an edited-image caption.The final dataset contains approximately 3M unique reference images and instructions across 10 edit sub-types.Source: D DATASET CONSTRUCTION DETAILS · pp. 22–24
  • Instruction prompts seek balanced coverage across 10 edit types and diverse scenarios while avoiding minute edits and vague or complex requests.The listed types include local color and texture changes, shape adjustment, add, remove, replace, background, style, action, and text manipulation.Source: D DATASET CONSTRUCTION DETAILS · pp. 22–23
  • Instructions are filtered for validity: they must refer to present objects, be feasible, and produce a meaningful image change.Removal captions omit the removed object, and prompts encourage simple, specific, diverse edits with clear placement for additions.Source: D DATASET CONSTRUCTION DETAILS · pp. 22, 24
  • Customization examples are selected for a single, uniquely identifiable, prominent central object, then paired with suggested background contexts.The prompts request diverse, plausible backgrounds without naming the object in the captions.Source: D DATASET CONSTRUCTION DETAILS · pp. 24–25

E.1 LOCAL-IMAGE EDITING

The appendix specifies training and VLM-question settings for local image editing and customization, alongside implementation details for the supervised and RL baselines.

E.1 LOCAL-IMAGE EDITING

  • Local editing uses 10K training iterations, beginning with 4K single-step predictions before two-step unrolling, and combines VLM editing loss with DMD.The VLM questions assess whether the instruction was followed and whether other image content stayed the same; removal uses a target-presence question with a ground-truth answer of No.Source: §E.1 · p. 26

E.2 FREE-FORM EDITING (CUSTOMIZATION)

  • Customization training uses different VLM questions for pose and background change versus object identity, and trains a 4-step model for 10K iterations.An 8-step model is fine-tuned for 5K more iterations; its DMD weight is 2 rather than 0.5 for local editing.Source: §E.2 · p. 27

F OTHER BASELINE DETAILS

  • Flow-GRPO fine-tunes a pre-trained image-editing model for 5K iterations, using the same computational budget as the authors’ method: four A100 GPU nodes (32 GPUs) over 2.5 days.Training collects 16 images per prompt with 12 denoising steps and uses LoRA rank 32; a VLM scores edits from 0 to 9, normalized to 0–1 for the final reward.Source: F OTHER BASELINE DETAILS · p. 27
  • Baseline evaluation follows each method’s open-source sampling setup, with prompt rewriting skipped for DSD and SynCD evaluated using one reference image.Qwen-2.5-32B-VLM supplies edited-image captions for TurboEdit on GEdit-Bench.Source: F OTHER BASELINE DETAILS · p. 28

H REPRODUCILITY OF RESULTS

The appendix points to implementation materials intended to support reproducibility and notes that the approach shares generative-model risks. Qualitative comparisons report competitive performance across editing benchmarks and customization concepts.

H REPRODUCILITY OF RESULTS

  • The paper provides implementation details for dataset construction, training, and hyperparameters, plus Algorithm 1 pseudocode, to facilitate reproducibility.Source: H REPRODUCILITY OF RESULTS · p. 28

I ETHICS STATEMENT

  • The authors note that deepfakes and misleading content are broader generative-model risks that also apply to this approach.They mention watermarking and reliable detection of generated images as possible mitigations.Source: I ETHICS STATEMENT · p. 28
  • On GEdit-Bench, ImgEdit-Bench, and TED-Bench, qualitative comparisons describe NP-Edit as on par with or better than baselines in the few-step setting.The comparisons include the best multi-step method as a reference.Source: Figure 13 · p. 29, Figure 14 · p. 30, Figure 16 · p. 32
  • On DreamBooth, NP-Edit is described as comparable to baselines in identity alignment while achieving better image fidelity across concepts in the few-step setting.Source: Figure 15 · p. 31
Loading 2510.14978v2…