PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantically Aligned Gradient-Driven Context-Preserving Image Editing

Chiranjeev Chiranjeev, Muskan Dosi, Mayank Vatsa, Richa Singh

arXiv:2609.12691v1cs.CV

TL;DR

Instruction-guided editors lack explicit semantic verification that generated images satisfy intended edits. IABEdit adds VLM-derived semantic supervision to training, and reports leading performance across several editing benchmarks, including a proprietary-baseline comparison under heavy occlusion.

  • Problem

    Existing instruction-guided editors lack explicit semantic verification that generated images satisfy intended edits, relying instead on reconstruction or token-level supervision.

    Source: §1 · p. 2

  • Method

    IABEdit adds differentiable VLM-derived semantic verification to training, backpropagating descriptors through the generative backbone to teach what and where to edit.

    Source: §1 · pp. 2–3

  • Results

    IABEdit reports leading performance on RealEdit, MagicBrush, EMU Edit, and D-LORD, and surpasses a proprietary baseline in its hardest tested setting: identity preservation under heavy occlusion.

    Source: §1 · p. 3, §5 · p. 15

  • Takeaways & Limitations

    The training-time supervision works across U-Net and MMDiT backbones without changing inference pipelines or adding deployment-time computational overhead.

    Source: §1 · p. 3

    Instructions containing terms such as “devoid,” “off,” and “less” can lead to incorrect edits.

    Source: §14 · p. 33

Abstract

from arXiv · show

Instruction-guided image editing has a training-time blind spot. Generative editors are never required to semantically verify whether their outputs actually satisfy the instruction. Supervision stops at reconstruction and input textual-level conditioning. This produces incomplete edits, spatial spillover, and poor localization. We present IABEdit, a model-agnostic framework that embeds differentiable semantic verification into training. A frozen vision-language model extracts spatially-aware descriptors from the ground-truth edit. A trainable aligner then reproduces them from the generated output. The residual between the two becomes a gradient that teaches the generator both what to edit and where, with no inference-time VLM cost. IABEdit is compatible with diverse backbones, including U-Net (Stable Diffusion) and MMDiT (FLUX), without altering their inference pipelines. On MagicBrush, it improves structural fidelity by +3.49 DINO-I over the best diffusion baseline and +1.26 over the best overall baseline, while remaining competitive on instruction alignment. It also achieves state-of-the-art instruction adherence performance on RealEdit and EMU Edit benchmarks based on embedding-based metrics. Most consequentially, on the D-LORD surveillance benchmark, it surpasses the proprietary Gemini agent by +5.13 DINO-P under heavy occlusion, where preserving identity is hardest. This shows that gradient-aligned VLM distillation holds up under real-world-like surveillance and occlusion conditions. Human and GPT-4o evaluations confirm perceptually precise, well-localized edits.

1 Introduction

Instruction-guided editors lack explicit semantic verification that outputs satisfy the requested edit, motivating IABEdit’s differentiable semantic supervision during training. The framework combines context-preserving conditioning with gradient-based alignment to guide what changes and where.

1 Introduction

  • IABEdit addresses the absence of explicit semantic verification, which leaves instruction-guided editors vulnerable to incomplete edits, mislocalization, and unintended changes to non-target regions.The paper frames the gap as an objective-level issue, not an architectural one.Source: §1 · p. 2
  • IABEdit reframes instruction-guided editing as semantic alignment and introduces differentiable semantic verification into generative-editor training.VLM reasoning defines the intended edit semantically rather than relying solely on textual conditioning.Source: §1 · p. 2

1.1 Related Work

  • Prior methods use static CLIP embeddings or VLM-based conditioning, but can lack active corrective feedback and incur inference cost.Prompt-level contrastive supervision also leaves instruction semantics out of generation.Source: §1.1 · pp. 3–4

2 IABEdit: Instruction-Aligned Editing

  • IABEdit combines image-context conditioning for structural fidelity with gradient-enabled alignment feedback for instruction-consistent modifications.The framework is described as architecture-agnostic and leaves inference pipelines unchanged.Source: §1 · p. 3, §2 · p. 4

2.1 Editing using context as condition

  • Structured conditioning separates the instruction embedding, source-image embedding, and source latent to guide edits while retaining visual context.The source-image embedding is preserved during training because it encodes identity and contextual cues.Source: §2.1 · p. 5

2.2 Semantic Alignment via Gradient-Enabled VLM Supervision

  • A frozen Descriptive Anchor supplies ground-truth edit descriptors, and a trainable Instruction Aligner predicts them from generated images to produce gradients for the generator.The alignment objective is applied during training; distillation and identity losses are used only at small timesteps, while general-scene editing omits the identity loss.Source: §2.2 · pp. 6, 8
  • For face editing, an identity-coherence loss based on a pretrained face-recognition model encourages retention of biometric features despite occlusions or appearance changes.The paper identifies this added constraint as important for surveillance and security settings.Source: §2.2 · pp. 7–8

3 Experimental Details

The experiments assess scene editing and identity-sensitive facial editing across several benchmarks. They combine feature-based measures with agent and human evaluations, and use LLaVA-based semantic supervision in a model-agnostic framework.

3 Experimental Details

  • IABEdit is trained on Super-40K and evaluated on RealEdit, MagicBrush, and EMU Edit, with D-LORD used for occluded facial images.D-LORD uses disjoint training and test sets.Source: §3 · p. 8
  • The model-agnostic framework supports diffusion and flow-based generators, uses LLaVA-based semantic supervision, and operates as a lightweight diffusion-only model at inference.Its setup uses CLIP instruction embeddings and conditional image features, with longer LLaVA-based descriptors for fine-grained semantic guidance.Source: §3 · p. 8
  • RealEdit qualitative comparisons visualize fine-grained instruction adherence and preservation of non-targeted regions.The caption points readers to supplementary material for additional visual results.Source: Figure 3 · p. 9
  • Evaluation combines preservation, target-similarity, instruction-alignment, and identity metrics with GPT-4o agent assessments and a user study.CS-P measures retention in non-edited regions; DINO-I and L1 assess structural and pixel-level similarity to the edited ground truth, while DINO-P measures facial identity preservation.Source: §3 · p. 9

4 Results and Analysis

IABEdit improves edit fidelity and balances instruction adherence with preservation across image-editing benchmarks. Results also show gains with a FLUX.1 backbone, in surveillance facial editing, and in human and GPT-4o evaluations.

Instruction alignment and localization

  • IABEdit with diffusion achieves the highest CLIP-T among diffusion methods on RealEdit (27.60) and MagicBrush (30.97), and scores 92.41 CLIP-I on MagicBrush.The passage reports competitive instruction alignment relative to FLUX-based and autoregressive methods.Source: §4 · p. 10

MagicBrush results

  • On MagicBrush, IABEdit achieves the best DINO-I score (88.26), beating the strongest diffusion baseline UltraEdit by +3.49 points, and the lowest L1 distance (0.058).These results indicate sharper structural fidelity to the ground-truth edits.Source: §4 · p. 10, Table 1 · p. 11

Balanced editing

  • On RealEdit, IABEdit achieves the highest HM between CLIP-T and CS-P, scoring 42.00 versus 41.67 for Reward InstructPix2Pix and 41.39 for UltraEdit.The reported Wilcoxon signed-rank test used n=560 and found significant results for all models (p < 0.05).Source: §4 · p. 11

Agent-based evaluation

  • On RealEdit GPT-4o evaluations, IABEdit scores 3.63 for instruction following and 4.19 for content preservation; its overall score is 3.78, second-best.SuperEdit is more consistent on binary accuracy, while IABEdit has higher continuous scores in instruction following and preservation.Source: Table 2 · p. 11, Figure 4 · p. 12

Model-agnostic framework

  • IABEdit+FLUX.1 scores 90.20 CS-P, 27.78 CLIP-T, and 42.28 HM on RealEdit, versus 87.64, 27.21, and 41.29 for FLUX.1.The passage reports p < 0.05 for these comparisons and describes improved context preservation after training FLUX.1 with IABEdit.Source: §4 · p. 13

EMU Edit

  • IABEdit outperforms the strongest EMU Edit baseline by +1.41 in CLIP-based instruction following, +0.84 in target-caption alignment, and +9.98 in DINO-based structural preservation.The results cover diffusion and FLUX-based architectures.Source: §4 · p. 13

Real-world surveillance

  • On D-LORD facial edits, IABEdit improves DINO-P by +5.13 points over the Gemini-AI agent and by +3.69 over SuperEdit.The SuperEdit comparison uses a model trained on the D-LORD training set and evaluated on a disjoint-identity test set; the harmonic-mean test gives p < 0.05.Source: §4 · p. 13, Table 3 · p. 13

Human evaluation and ablations

  • The user study finds IABEdit highest in instruction following (50.00%) and non-instructed-region preservation (29.83%); combined denoising and distillation losses improve HM by +2.47 over denoising alone.In the human study, IABEdit also scores 37.58% for image quality, above SuperEdit's 31.45%; LoRA fine-tuning improves HM by +4.42, and context-preservation conditioning improves CS-P by +0.74, CLIP-T by +1.70, and HM by +1.81.Source: §4 · pp. 14–15

5 Conclusion

IABEdit makes semantic instruction alignment an optimization target by distilling spatially aware VLM descriptors into gradients. The conclusion reports transfer across U-Net and MMDiT backbones and a stronger DINO-P result than a proprietary baseline under heavy occlusion.

5 Conclusion

  • The RealEdit user study figure reports human evaluations of instruction following, preservation of non-editing content, and edited-image quality across 30 randomly selected images.Source: Figure 6 · p. 15
  • IABEdit surpasses a proprietary baseline on identity preservation under heavy occlusion, the hardest tested case.The conclusion frames this as a real-world-like surveillance condition.Source: §5 · p. 15
  • IABEdit distills spatially aware descriptors from a frozen VLM into gradients that make instruction satisfaction part of the training objective.The conclusion characterizes this as semantic alignment rather than pixel-only generation.Source: §5 · p. 15
  • The semantic feedback operates only during training, leaving inference untouched while transferring across U-Net and MMDiT backbones.Source: §5 · p. 15

Supplementary Material

The supplementary material collects additional qualitative and quantitative results, implementation details, human-study material, and discussion of cost and limitations.

Supplementary Material

6 IABEdit with FLUX

IABEdit extends its semantic-alignment training approach to FLUX.1-dev, a flow-matching backbone, while retaining structured conditioning and gradient-based feedback. The supplementary figure also presents sample-level user-study preferences across three evaluation criteria.

6 IABEdit with FLUX

  • FLUX interpolates a clean latent z0 and Gaussian noise ε to form zt = (1 − t)z0 + tε.Source: §6 · p. 20
  • The FLUX MMDiT velocity predictor uses instruction embeddings from CLIP and T5, a source-image embedding, and the source latent as context.Source: §6 · p. 20
  • The velocity predictor is trained with the standard flow-matching objective, using the noisy latent concatenated with source context.Source: §6 · p. 20
  • For small timesteps t ≤ τ, the predicted velocity estimates a clean latent that is decoded and passed to LLaVA for gradient-based localized-edit feedback.The passage specifies τ = 200.Source: §6 · p. 20
  • IABEdit operates with flow-based backbones such as FLUX while retaining structured conditioning and gradient-based semantic alignment.The paper presents this as a model-agnostic extension of its framework.Source: §6 · p. 20
  • Figure 7 shows sample images with the percentages of users selecting the best-performing model for instruction following, preservation, and quality.Source: Figure 7 · p. 21

7 Human Study Qualitative Results

The RealEdit human study evaluates instruction following, preservation of non-instructed regions, and edited-image quality. Its qualitative figure pairs representative samples with user preferences for the best-performing model.

7 Human Study Qualitative Results

  • Figure 7 pairs qualitative examples with the percentage of users selecting the best-performing model for instruction following, preservation, and quality.Source: §7 · p. 21
  • Thirty participants aged 20–45 each evaluated 30 randomly sampled RealEdit instances in a blinded, randomized comparison.Model identities were hidden; participants saw each original image, instruction, and edited outputs from different models.Source: §7 · p. 21
  • Participants selected the best output based on instruction following, preservation of non-instructed regions, and overall image quality.Source: §7 · p. 21

8 Facial Image Editing and its impact on Surveillance scenarios

The section evaluates facial disguise editing on D-LORD and its use for recognition under occlusion. Ablations and low-resolution examples assess editing quality, while recognition results test the utility of generated images.

8.1 Ablation Results on Facial Disguise Image Editing

  • 46.74 HM and 55.33 DINO-preserving are the highest reported scores when all denoising, distillation, and coherence losses are used.Compared with diffusion loss alone, the full combination improves CS-P by +11.23, CLIP-T by +2.43, HM by +5.03, and DINO-P by +4.92.Source: §8.1 · p. 22, Table 5 · p. 22

8.2 Surveillance Results in Low-Resolution Setting

  • Low-resolution mask edits cover the lower face, while sunglasses align with the eyes and preserve facial structure and texture outside the edited regions.The examples are from real-world surveillance face images in D-LORD.Source: §8.2 · p. 22, Figure 8 · p. 23
  • The generated occlusions blend naturally with minimal artifacts despite low-resolution inputs, while maintaining instruction adherence and unedited-region characteristics.The paper presents these results as evidence of applicability to real-world face editing scenarios.Source: §8.2 · pp. 22, 24
  • 61.51% accuracy @1% FPR is reached by fine-tuning on combined real and synthetic data, the best of the reported recognition setups.Training from scratch on combined data reaches 59.45%, compared with 58.69% using only real data and 52.7% using only synthetic data.Source: Figure 9 · p. 23, §8.2 · p. 24

9 Implementation Details

Implementation uses a Stable Diffusion latent-diffusion setup for scene and face editing, and also adapts IABEdit to FLUX.1. The section reports a timestep choice, implementation settings, and instruction-based VLM prompts.

9.1 Diffusion Model: Stable Diffusion v1.5

  • The Stable Diffusion v1.5 setup uses LLaVA-7B as descriptive anchor and instruction aligner, CLIP text and image conditioning, and 512 × 512 diffusion inputs.LLaVA processes denoised edited images resized to 336 × 336; inference remains VLM-free, with 30 DDIM sampling steps.Source: §9.1 · pp. 24–25
  • At τ = 200, single-step latent estimation provides the best reported RealEdit balance, with HM=42.00; τ = 300 raises CS-P to 93.77 but lowers CLIP-T to 26.13.The authors select τ = 200 because the latent is less noisy and can be denoised in one step before VLM input.Source: §9.1 · p. 25, Table 6 · p. 25

9.2 Flow Matching Model: FLUX.1

  • The FLUX.1-dev implementation uses an MMDiT backbone trained with LLaVA-derived backward gradient supervision for 26,000 iterations on SuperEdit-40K.The scene-editing setup uses denoising and distillation losses at τ = 200.Source: §9.2 · pp. 25–26

9.3 Editing Instruction-based Prompts

  • VLM prompts are constructed from editing instructions to describe the intended localized changes and the regions that should remain unchanged.Figure 10 shows prompt examples for scene and facial editing.Source: §9.3 · p. 26, Figure 10 · p. 26

10 Additional Qualitative Results

The section examines VLM-derived gradient maps, qualitative MagicBrush edits, and performance on semantically varied instructions. The results show localized edits and context preservation, alongside failures on some complex prompts.

10.1 VLM-focussed Gradient Maps Visualization

  • Grad-CAM activations concentrate on instructed edit regions, including boat, lighting, and plaza changes, indicating token-to-region localization.The maps associate tokens such as “sail,” “sunlight,” and “tower” with the corresponding modified image regions.Source: §10.1 · pp. 27–28, Figure 11 · p. 27
  • Context tokens attend to surrounding structures such as water, trees, buildings, and ground, while the model preserves scene layout and environmental coherence.The reported patterns include attention to reflections around an inserted sailboat and to structures that remain unmodified.Source: §10.1 · p. 28

10.2 Qualitative Results on MagicBrush Benchmark

  • MagicBrush examples show localized edits that preserve surrounding objects and fine detail, though IABEdit fails to replace a baseball with an orange in one case.The paper also notes that IABEdit adds flowers more prominently than the ground truth in one example.Source: §10.2 · pp. 29–30, Figure 12 · p. 29

10.3 Results on Complex Instruction Scenarios

  • Complex instruction scenarios are identified as a future direction for incorporation into the training mechanism across varied prompt-based editing tasks.Source: §10.3 · p. 31
  • 42.00, 41.74, 42.12, and 41.79 HM are reported for IABEdit on direct simple, morphological, syntactic, and lexical instructions, respectively.The paper reports IABEdit as best on direct simple instructions and competitive on lexical instructions, while significantly and consistently outperforming SuperEdit.Source: Table 7 · p. 30, §10.3 · p. 31
  • Both IABEdit and SuperEdit fail on some complex instructions containing “devoid,” “off,” or “less,” despite succeeding on direct simple versions.Source: §10.3 · p. 31

11 Additional Quantitative Results

The section reports masked content-preservation scores on RealEdit and results on the EMU Edit test set. IABEdit (+FLUX.1) leads the reported EMU Edit metrics, while masked-CS-P has a stated precision boundary for add or insert instructions.

11.1 Masked CS-P evaluation on RealEdit dataset

  • IABEdit and SuperEdit achieve masked CS-P scores of 99.68 and 99.60, respectively, for instructions with known edit regions.Source: §11.1 · p. 31
  • Masked-CS-P is less precise for “Add” or “Insert” instructions because the original image lacks the object needed to establish a reference mask.Source: §11.1 · p. 31

11.2 Results on EMU Edit test set

  • +0.84 CLIPdir, +1.41 CLIPout, +5.92 CLIPimg, and +9.98 DINO over EMU Edit; IABEdit (+FLUX.1) achieves the best performance across all four metrics.Its reported scores are CLIPdir 11.21, CLIPout 25.06, CLIPimg 88.81, and DINO 91.02.Source: §11.2 · pp. 32–33, Table 8 · p. 32

12 Discussion on Image Quality

IABEdit does not improve image quality; the evaluation instead focuses on instruction following and context preservation, including quality. The reported preservation score is 4.19, the best on the same metric.

12 Discussion on Image Quality

  • 4.19 is IABEdit’s best preservation score on the same metric used for the baseline.The authors use preservation to evaluate context, including image quality.Source: §12 · p. 33
  • IABEdit does not improve image quality, because the editing task aims to maintain the original image’s quality rather than enhance lighting or appeal.Source: §12 · p. 33
  • The overall evaluation focuses on instruction following and context preservation rather than image-quality improvement.Source: §12 · p. 33

13 Training Cost

IABEdit adds training computation and trainable parameters, while testing remains VLM-free and similar in runtime to SuperEdit. The authors also report incorrect edits for instructions containing certain words and increased training overhead from VLM-based gradient supervision.

13 Training Cost

  • IABEdit training adds 9.03% FLOPs (SD+LLAVA), reaching ∼6624 GFLOPs versus SuperEdit’s ∼733 GFLOPs, and introduces 95.68 M extra trainable parameters versus SuperEdit’s 59.53 M.SmartEdit incurs higher FLOPs, but uses two-stage training with a Q-Former, Bidirectional Interaction Module, and LLaVA-13B with LoRA.Source: §13 · p. 33
  • VLM-free testing takes 3s/image on a single GPU for 30 timesteps, similar to SuperEdit.Source: §13 · p. 33

14 Limitations and Failure Modes

  • Instructions containing “devoid,” “off,” or “less” can lead to incorrect edits, and VLM-based gradient supervision increases training overhead.Source: §14 · p. 33
Loading 2609.12691v1…