PaperlayerView exact revision on arXiv

Source-linked AI summary

RePlan: Reasoning-guided Region Planning for Complex Instruction-based Image Editing

Tianyuan Qu, Lei Ke, Xiaohang Zhan, Longxiang Tang, Yuqi Liu, Bohao Peng, Bei Yu, Dong Yu, Jiaya Jia

arXiv:2512.16864v2cs.CV

TL;DR

Instruction-based editing models still struggle to ground changes when complex instructions meet cluttered or ambiguous scenes. RePlan combines VLM reasoning and region-aligned planning with diffusion editing, using attention injection for one-pass multi-region edits. On IV-Complexity, it improves across MMDiT backbones and outperforms state-of-the-art models trained on far more data.

  • Problem

    Image-editing models struggle to ground edits in complex scenarios, despite producing diverse, high-quality effects [18].

    Source: §1 · pp. 2–3

  • Method

    RePlan has a VLM reason over the image and instruction to produce region-aligned guidance, which a diffusion editor executes using training-free attention injection.

    Source: §1 · p. 3

  • Results

    In IV-Complexity scenarios, RePlan outperforms state-of-the-art models trained on orders of magnitude more data and consistently improves various MMDiT backbones.

    Source: §1 · p. 3, §6 · p. 15

  • Takeaways & Limitations

    IV-Edit provides a benchmark for instruction–visual understanding, fine-grained target localization, and knowledge-intensive reasoning.

    Source: §1 · p. 3, §6 · p. 15

Abstract

from arXiv · show

Instruction-based image editing enables natural-language control over visual modifications, yet existing models falter under Instruction-Visual Complexity (IV-Complexity), where intricate instructions meet cluttered or ambiguous scenes. We introduce RePlan (Region-aligned Planning), a plan-then-execute framework that couples a vision-language planner with a diffusion editor. The planner decomposes instructions via step-by-step reasoning and explicitly grounds them to target regions; the editor then applies changes using a training-free attention-region injection mechanism, enabling precise, parallel multi-region edits without iterative inpainting. To strengthen planning, we apply GRPO-based reinforcement learning using 1K instruction-only examples, yielding substantial gains in reasoning fidelity and format reliability. We further present IV-Edit, a benchmark focused on fine-grained grounding and knowledge-intensive edits. Across IV-Complex settings, RePlan consistently outperforms strong baselines trained on far larger datasets, improving regional precision and overall consistency. Our project page: https://replan-iv-edit.github.io

1 Introduction

Instruction-based editing struggles when cluttered or ambiguous visuals meet complex instructions, because edits require fine-grained grounding and reasoning. RePlan couples a reasoning-based VLM planner with a diffusion editor, and the paper also introduces an attention-injection mechanism and the IV-Edit benchmark.

1 Introduction

  • IV-Complexity combines cluttered or ambiguous visuals with complex instructions, while existing editors struggle to ground and execute edits in such scenarios.The cited challenge includes multiple similar objects, multi-object references, implicit semantics, world knowledge, and causal reasoning.Source: §1 · pp. 2–3
  • RePlan uses VLM chain-of-thought reasoning to produce structured region-aligned guidance that a diffusion model executes for precise edits under IV-Complexity.The framework refines VLM–diffusion interaction from global semantic guidance to region-specific guidance.Source: §1 · p. 3
  • Training-free attention region injection gives a pre-trained editing DiT region-aligned control for efficient multi-region edits in one pass.The mechanism avoids image degradation from multi-round inpainting while reducing computation cost.Source: §1 · p. 3
  • IV-Edit is introduced as a benchmark for instruction–visual understanding, fine-grained target localization, and knowledge-intensive reasoning.The paper describes it as the first benchmark tailored to IV-Complexity and a principled testbed for future research.Source: §1 · p. 3
  • With ∼1k instruction-only examples for GRPO training, RePlan outperforms state-of-the-art models trained on orders of magnitude more data in IV-Complexity scenarios.The attention-region injection mechanism enables region-aligned control and multiple edits in one pass.Source: §1 · p. 3

2 Related Work

Prior work includes direct instruction-to-image editing, mask-guided inpainting, and VLM-guided generation, with limitations in spatial reasoning or fine-grained guidance. Existing editing benchmarks and reasoning-oriented benchmarks also leave complex real-world image-editing scenarios underrepresented.

2 Related Work

  • End-to-end methods such as InstructPix2Pix [13] support strong global editing but have limited spatial reasoning, while inpainting pipelines rely on fragile localization modules and struggle with reasoning-heavy instructions.Inpainting pipelines first localize regions and then apply mask-guided editing.Source: §2 · p. 4
  • VLM-guided generation methods [9] [25] typically use VLMs only at a coarse level, underusing their fine-grained reasoning capabilities.Source: §2 · p. 4
  • Imgedit [27] and GEdit [18] focus on salient subjects and explicit instructions, while KrisBench [26] and RISEBench [30] still use simple image compositions.The passage notes these benchmarks do not reflect the instruction-visual complexity of real-world image editing.Source: §2 · p. 4

3 Method

RePlan separates complex editing into VLM-based planning and diffusion-based execution: the planner produces region-aligned guidance, while training-free attention injection controls edits. A two-stage GRPO strategy trains the planner for valid, reasoned plans and image-level editing quality.

3.1 Overview

  • RePlan couples a VLM planner that reasons over the image and instruction with a DiT decoder that executes region-aligned guidance in one pass.The framework uses reinforcement learning with VLM-based feedback and about 1k instruction-only samples, without paired images.Source: §3.1 · p. 4

3.2 Region-aligned Editing Planner

  • The planner decomposes guidance into global and region-specific edits, outputting structured reasoning and JSON region guidance that users can adjust directly.It uses chain-of-thought reasoning to ground ambiguous or abstract descriptions in concrete visual effects.Source: §3.2 · pp. 4–6

3.3 Training-free Attention Region Injection

  • Full self-attention can let tokens from one region attend to unrelated instructions, causing target confusion or instruction failure in multi-region editing.The method groups image patches by bounding boxes and hints by global or local region, then applies attention masks to control token interactions.Source: §3.3 · pp. 6–7
  • Five attention rules isolate hints, constrain regional and background tokens, and preserve image–latent interaction, enabling parallel region edits while avoiding accumulated errors from multi-round inpainting.The mechanism also supports region-level negative prompts.Source: §3.3 · p. 7

3.4 Structured Planning and Reasoning with GRPO

  • GRPO training proceeds in two stages: first rewarding valid structured plans and reasoning, then using image-level target, effect, and consistency rewards to refine planning.Consistency is reweighted by effect to discourage reward hacking, and a small λ weight preserves format reliability.Source: §3.4 · p. 8
  • Both GRPO stages use 1k complex instruction-editing samples generated for supervised alignment before reinforcement learning.Stage 1 rewards tag and region-format validity and reasoning quality.Source: §3.4 · p. 8

4 Data Construction and Benchmark

IV-Edit is designed to assess fine-grained grounding and reasoning in visually rich editing tasks, including real-world photos and text-related images. It combines diverse referring expressions and edit intents, and uses a four-dimension evaluator for edited images.

4.1 Task Setting

  • IV-Edit targets instructions that require fine-grained language-to-region grounding, with non-subject-dominated images and contextual or knowledge-based reasoning demands.It covers complex real-world photo editing and text-related images such as posters, slides, charts, and tables.Source: §4.1 · pp. 8–9
  • Each IV-Edit instruction combines a referring expression with an editing task, spanning 7 referring types and 16 task types.This composition supports varied combinations of grounding mechanisms and edit intents across visual domains and reasoning requirements.Source: §4.1 · pp. 9–10

4.2 Benchmark Statistics

  • IV-Edit contains around 800 verified instruction–image pairs; 182 involve edits to multiple target regions, and instructions average 21 words.The pairs come from multiple public data sources and cover heterogeneous visual domains and editing effects.Source: §4.2 · p. 10

4.3 Evaluation Protocol

  • Gemini2.5-Pro rates before-and-after image pairs on target, consistency, quality, and effect because global CLIP semantic similarity is not well suited to these inputs.The dimensions assess target localization and modification, preservation of unrelated content and style, visual quality, and fidelity to the requested effect.Source: §4.3 · p. 10

5 Experiments

On IV-Edit, RePlan improves results across MMDiT backbones, while rule-group attention keeps latency efficient as region counts increase.

5.1 Results on IV-Edit Benchmark

  • RePlan consistently improves performance across MMDiT backbones and achieves the best Consistency and Overall scores among open-source models.The evaluation defines Target as recall-oriented localization and Consistency as precision-oriented regional localization.Source: Table 1 · p. 11, §5.1 · p. 11
  • In Fig. 6 editing comparisons, RePlan shows clear advantages in accurately localizing target editing regions and stronger reasoning on indirect instructions, while GPT-4o’s fixed aspect ratios crop non-standard images.Source: §5.1 · p. 12, Figure 6 · p. 12
  • In blinded preference tests, RePlan is preferred in 35% of cases versus 15% for Flux.1-Kontext-dev, with 50% ties.Among non-tie cases, RePlan wins 70%.Source: §5.1 · p. 13
  • Multiple regional edits and a global lighting change run in one pass, with localized edits and coherent image-wide adjustment.The reported results remain localized even across numerous regions.Source: §5.1 · p. 13, Figure 7 · p. 13

5.2 Ablation Study

  • Planner ablations find that RL improves reliability over non-RL planners, removing chain-of-thought reasoning lowers performance, and two-stage RL improves scores and sample efficiency.The non-RL planners show bbox errors or difficulties with hint decomposition and format compliance.Source: §5.2 · pp. 13–14, Table 2 · p. 14

5.3 Inference Latency

  • Rule-group attention gives about 2× speedup over naive masked SDPA; latency rises mildly with more regions, unlike sequential multi-turn editing.With one region, masked RePlan matches the no-mask FlashAttention2 baseline.Source: Table 3 · p. 14, §5.3 · p. 15

6 Conclusion

The conclusion presents RePlan as a region-planning and attention-injection framework for IV-Complexity, alongside IV-Edit as a testbed for complex editing.

6 Conclusion

  • RePlan combines VLM-based region reasoning with training-free attention injection for precise parallel edits, and the paper presents IV-Edit as a benchmark for IV-Complexity.The conclusion reports improvements across MMDiT backbones without compromising efficiency.Source: §6 · p. 15

Appendix Contents

  • The appendices cover IV-Edit statistics and construction, global-rephrase comparisons, multi-region visualizations, attention rules, bbox robustness, and additional benchmarks and comparisons.The listed sections also include overlapping-bbox cases.Source: Appendix Contents · p. 18

A More IV-Edit Statistics

B Data Construction Details

The appendices describe IV-Edit data construction, compare RePlan with global instruction rephrasing, and provide additional multi-region editing visualizations.

B Data Construction Details

  • IV-Edit draws images from several public datasets and filters for complex scenes or structured text images before constructing and screening instructions.The pipeline checks target bbox counts, filters ambiguous or impractical instructions, and manually filters test samples; retained training data is roughly one-third of source data.Source: B Data Construction Details · pp. 20–21

C Comparison with Global Rephrase

D More Multi-Region Editing Visualizations

  • The appendix adds Figs. 10 and 11 as further multi-region examples; the authors report preserving global coherence and local consistency across many regions.They contrast this capability with cut-and-paste and layer-based approaches, which they say fail to maintain global coherence, especially under scene-wide changes.Source: D More Multi-Region Editing… · pp. 21–22, Figure 10 · p. 22, Figure 11 · p. 22

E Attention Rule Discovery

Attention-rule experiments show that region-restricted noise-to-image attention can reference designated objects, while severing cross-region attention harms global consistency and withholding text attention distorts image patches. The injection mechanism also handles overlapping boxes by allowing shared patches to attend to corresponding hint tokens while retaining image-wide self-attention.

E Attention Rule Discovery

  • Cutting attention across image regions creates clear boundaries and loses global consistency.The figure shows the resulting regional boundaries.Source: E Attention Rule Discovery · p. 23, Figure 12 · p. 23
  • Region-restricted noise-to-image attention lets the model generate with reference to objects in designated regions.The experiments decouple image and noise-latent tokens while preserving their respective self-attention.Source: E Attention Rule Discovery · p. 23, Figure 13 · p. 24
  • Image regions without text-modality attention suffer severe distortion and noise; the authors hypothesize text attention facilitates internal image information exchange.The figure also reports severe distortion when background patches cannot attend to the global prompt token.Source: E Attention Rule Discovery · p. 23, Figure 14 · p. 24

F B-boxes Overlapping Case

  • For overlapping boxes, shared image patches can attend to the corresponding hint tokens, while full image-patch self-attention lets the model manage sub-edit interactions.The edited region retains the original content and style in the illustrated attention setup.Source: F B-boxes Overlapping Case · p. 23, Figure 13 · p. 24

G Robustness Against B-box Perturbation

The perturbation experiment tests how random shifts and scaling of VLM-generated bounding-box corners affect RePlan. The method remains robust at a perturbation ratio of 50%.

G Robustness Against B-box Perturbation

  • The experiment adds random scale and shift noise to every corner of VLM-generated bounding boxes.A 10% perturbation shifts each corner randomly by 10% of the box width or height in pixels.Source: G Robustness Against B-box Perturbation · p. 24
  • The mechanism depends primarily on whether a bounding box corresponds to the target instance or region, so pixel-level errors have little impact.The tested perturbations were applied to bounding-box corner points.Source: G Robustness Against B-box Perturbation · p. 24
  • With random scale-and-shift noise applied to all VLM-generated bounding-box corner points, the method still exhibits robustness when the perturbation ratio reaches 50%.Source: G Robustness Against B-box Perturbation · p. 24

H Results on More Benchmarks

On MagicBrush and SmartEdit, RePlan consistently improves over its backbone, including on global, coarse-grained metrics. The section also cautions that such metrics can misrepresent instruction-following detail in IV-Complex tasks.

H Results on More Benchmarks

  • The evaluation compares standard baselines, the MMDiT backbone, RePlan, and attention injection using dataset-provided region guidance.The benchmarks are MagicBrush and SmartEdit, described as more limited in domain and IV-Complexity than IV-Edit.Source: H Results on More Benchmarks · pp. 25–26, Table 6 · p. 26, Table 7 · p. 26
  • Global CLIP similarity can be distorted on IV-Complex tasks, where a higher score may coincide with worse instruction-following details.The paper notes that this metric-hacking phenomenon was also highlighted in SmartEdit.Source: H Results on More Benchmarks · p. 26
  • The section includes additional comparative results using Flux.1 Kontext dev as RePlan's backbone; one example has a slight perspective change.The second column of Figure 17's comparison is noted as having a slight perspective change.Source: Figure 17 · p. 27
  • RePlan consistently improves over its backbone on MagicBrush and SmartEdit, even on global, coarse-grained metrics.Tables 6 and 7 report the benchmark comparisons.Source: H Results on More Benchmarks · pp. 25, 27, Table 6 · p. 26, Table 7 · p. 26

I More Comparative Results

The section supplies additional comparative results, including a text-editing scenario. The figures identify Flux.1 Kontext dev as RePlan's backbone.

I More Comparative Results

Loading 2512.16864v2…