SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models

Introduction

• Complex scenarios involve modifying one object among several using attributes or identifying an object through world knowledge, challenges existing methods often fail to handle.The paper terms these complex understanding and complex reasoning scenarios, respectively, and identifies them as important for practical editing.Source: §1 · p. 2
• SmartEdit jointly optimizes MLLMs and diffusion models to address CLIP’s instruction-understanding limits, while BIM enables bidirectional interaction between input-image information and MLLM outputs.The existing one-way interaction uses image features as queries and MLLM outputs as keys and values; BIM reuses image information from LLaVA’s visual encoder.Source: §1 · p. 2
• To stimulate complex editing without generating extensive costly paired data, SmartEdit trains with perception-related data and fine-tunes on a few high-quality synthetic examples.The strategy aims to strengthen UNet perception and stimulate the model’s ability to handle complex instructions.Source: §1 · p. 2
• On Reason-Edit’s 219 image-text pairs, SmartEdit significantly outperforms previous instruction-based image editing methods in quantitative and qualitative results.The evaluation pairs do not overlap with the synthesized training data.Source: §1 · pp. 2–3

2.1. Image Editing with Diffusion Models.

• Prompt-to-Prompt [13] modifies cross-attention maps, while MasaCtrl [4] uses mutual self-attention to retrieve correlated source-image content and textures for consistency.Source: §2.1 · p. 3
• InstructPix2Pix [1] fine-tunes a UNet on GPT-3- and Prompt-to-Prompt-generated editing data, while MagicBrush [40] provides manually annotated real-image editing data.Source: §2.1 · p. 3
• InstructDiffusion [12] jointly trains across understanding and generative vision tasks, but typically produces inferior results in complex understanding and reasoning scenarios.Source: §2.1 · p. 3

2.2. LLM with Diffusion Models

• LLaVA and MiniGPT-4 [25] [42] improve image-text alignment through instruction tuning, while GILL [20] connects MLLMs with diffusion for text-conditioned image generation.Source: §2.2 · p. 3
• SEED [10] enables joint image-text processing, and SEED-2 [11] aligns generation embeddings with unCLIP-SD image embeddings for richer visual semantics and realistic reconstruction.Source: §2.2 · p. 3
• Emu [33] is a next-token-trained multimodal generalist, while CM3Leon [39] performs text-to-image and image-to-text generation using instruction-style fine-tuning.Source: §2.2 · pp. 3–4

• Instruction-based editing modifies input image x according to instruction cT to produce target y; InstructPix2Pix trains a latent-diffusion UNet to predict added noise conditioned on the encoded image and instruction.The image condition is concatenated with the noisy latent; t denotes the sampling step, and the text condition is cT.Source: §3 · p. 4
• InstructPix2Pix is limited in complex understanding and reasoning scenarios, motivating an MLLM and Bidirectional Interaction Module for information exchange between MLLM output and image information.The module is proposed to enable bidirectional interaction between those inputs.Source: §3 · p. 4
• The section identifies perception-related data and a small amount of complex editing data as crucial to improving model performance.Source: §3 · p. 4

4.1. The Framework of SmartEdit

• SmartEdit appends r [IMG] tokens to the instruction, extracts their LLaVA hidden states, aligns them with a QFormer, and sends BIM-updated features to the diffusion UNet.The updated text feature f′ serves as the UNet key and value, while the updated image feature v′ is added to features before they enter the UNet.Source: §4.1 · p. 4, Figure 3 · p. 5

4.2. Bidirectional Interaction Module

• BIM self-attends over f, uses f to attend to image feature v to produce f′, then uses f′ to attend to v and produce v′.Its two cross-attention blocks enable bidirectional interaction between image and text features.Source: §4.2 · p. 5, Figure 4 · p. 5

4.3. Dataset Utilization Strategy

• To address weak perception and limited reasoning exposure, SmartEdit adds segmentation data and about 476 synthetic edits covering complex object-attribute understanding and world-knowledge reasoning.The understanding scenarios include modifying a specific object by attributes such as location, color, relative size, or whether it is inside or outside a mirror.Source: §4.3 · pp. 5–6

4.4. Reason-Edit for Better Evaluation

• Reason-Edit contains 219 image-text pairs, covers complex understanding and reasoning scenarios, and has no overlap with the training data.The dataset is intended to evaluate instruction-based image editing methods on these scenarios.Source: §4.4 · p. 6

5.1. Experimental Setting

• Two-stage training aligns the MLLM with CLIP before optimizing SmartEdit, which uses perception, editing, VQA, and synthetic editing data.The first stage uses CC12M; the second includes segmentation data and 476 paired synthetic examples. Evaluation includes background PSNR, SSIM, and LPIPS, foreground CLIP Score, and human-rated Instruction-Alignment.Source: §5.1 · pp. 6–7

5.2. Comparison with State-of-the-Art Methods

• SmartEdit improves Instruction-Alignment over prior methods, and SmartEdit-13B scores better than SmartEdit-7B on this metric.For understanding scenarios, PSNR, SSIM, and LPIPS rank SmartEdit-7B below InstructDiffusion and SmartEdit-13B above it, despite both SmartEdit models appearing visually better.Source: §5.2 · p. 7
• In qualitative reasoning examples, both SmartEdit sizes handle the first two cases, but only SmartEdit-13B correctly edits the third without changing the background.The examples include identifying and removing a fruit-cutting knife; the results indicate a difference between model sizes on a reasoning case.Source: §5.2 · p. 7, Figure 5 · p. 8
• On Reason-Edit reasoning scenarios, SmartEdit-7B and SmartEdit-13B outperform the compared methods on all five reported metrics.The comparison fine-tunes InstructPix2Pix, MagicBrush, and InstructDiffusion on the same training set as SmartEdit before evaluation.Source: §5.2 · p. 7, Table 1 · p. 8

5.3. Ablation Study on BIM

• Removing BIM lowers all metrics, while replacing it with one-way cross-attention lowers all but understanding-scenario CLIP Score.Qualitative examples show BIM removes the dog bowl and fork, whereas the no-BIM and SimpleCA variants fail or remove them only partially.Source: §5.3 · p. 7, Table 2 · p. 9, Figure 6 · p. 9

5.4. Ablation Study on Dataset Usage

• Segmentation and synthetic editing data play complementary roles, and combining all training datasets further improves visual results.With editing plus synthetic data, objects are located accurately, but outputs remain mediocre; editing-only and editing-plus-segmentation perform worse in reasoning scenarios.Source: §5.4 · pp. 8–9, Table 3 · p. 9, Figure 7 · p. 9

• SmartEdit outperforms previous methods on the newly constructed Reason-Edit dataset.Source: §6 · p. 9
• The Bidirectional Interaction Module addresses challenges in directly integrating LLMs and diffusion models for complex reasoning.Source: §6 · p. 9
• SmartEdit incorporates visual-input LLMs to enhance understanding and reasoning in instruction-based image editing.Source: §6 · p. 9
• Its data strategy uses perception data and complex instruction editing data to improve handling of complex understanding and reasoning scenarios.Source: §6 · p. 9

• The supplement provides data-production details, further quantitative and visual results on Reason-Edit, MagicBrush results, and comparisons with MGIE [9] and InstructDiffusion [12].Source: Supplementary Material · p. 12
• Unsatisfactory generated images are manually filtered, and Fig. 9’s last row shows reasoning samples included in the training data.Source: Supplementary Material · p. 12

• Approximately 476 synthetic paired examples supplement training data for complex-understanding and reasoning scenarios.Source: §7 · p. 12
• For complex-understanding examples, SAM masks objects, MAT removes the target, and PIH harmonizes a replacement composite to form instruction-image pairs.The pipeline also resizes and filters the replacement image and masks before compositing. It produces pairs such as adding a cat or replacing a smaller cat with a rabbit.Source: §7 · p. 12, Figure 8 · p. 13
• For reasoning scenarios, the pipeline generates an object mask with SAM and uses Stable Diffusion to inpaint according to the instruction.Source: §7 · p. 12

8.1. Instruction-Alignment Metric (Ins-align)

• Ins-align averages four workers’ judgments of whether edits match instructions, addressing cases where CLIP-Score or background PSNR, SSIM, and LPIPS diverge from visual assessment.The passage illustrates these metric disagreements by contrasting SmartEdit and InstructDiffusion outputs.Source: §8.1 · p. 14, Figure 10 · p. 15
• On Ins-align, SmartEdit significantly outperforms previous instruction-based editing methods in complex understanding and reasoning, and SmartEdit-13B scores better than SmartEdit-7B.Ins-align is reported in the main-paper comparisons and ablations.Source: §8.1 · p. 14
• Ablations show that BIM’s bidirectional interaction improves complex instruction editing, while jointly using editing, segmentation, and synthetic editing data yields better complex-scenario results.Source: §8.1 · p. 14

8.2. User Study

• Over 67% of participants judged SmartEdit’s results better aligned with instructions, and more than 72% preferred its edited results.The study used 30 Reason-Edit images, with results from four methods shuffled for 25 participants to select the best instruction-aligned and highest-quality images.Source: §8.2 · p. 14, Figure 11 · p. 15

9. More Visual Results on Reason-Edit

• SmartEdit produces good edits for instructions involving object attributes such as location, relative size, color, and position in or outside a mirror.Figure 12 presents additional SmartEdit results on complex understanding scenarios.Source: §9 · p. 16, Figure 12 · p. 17
• In complex understanding scenarios, SmartEdit edits specified objects more effectively than InstructDiffusion and MagicBrush, which struggle with instruction understanding or image quality.InstructDiffusion can edit specified objects in some examples but produces much worse image quality; MagicBrush modifies some examples but with poor quality, and existing methods struggle in others.Source: §9 · p. 16, Figure 13 · p. 18
• SmartEdit handles complex reasoning instructions better than existing methods, identifying the intended objects and editing them while keeping other areas unchanged.For example, it removes a fork without modifying the cake and replaces the orange—the food with the most vitamins—with an apple; the comparison uses the same fine-tuning data, while SmartEdit adds LLaVA and BIM.Source: §9 · p. 16, Figure 14 · p. 19

10. Results of SmartEdit and Other Methods on MagicBrush

• Figure 15 shows good SmartEdit editing effects for single-turn, two-turn, and three-turn MagicBrush test examples.The figure presents single-turn results in its first two rows, two-turn results in the middle two, and three-turn results in the last row.Source: §10 · p. 20
• In examples, SmartEdit replaces animal stickers and changes hats as instructed, while MagicBrush-168 makes incomplete or minimal changes despite stronger L1, CLIP-I, and DINO-I scores.These examples illustrate why the authors question those metrics and consider CLIP-T potentially more reliable.Source: §10 · p. 20
• In qualitative comparisons, InstructPix2Pix performs poorly, MagicBrush-168 often retains the original or edits poorly, MagicBrush-52 misses instructions, and InstructDiffusion sometimes produces artifacts; SmartEdit adheres effectively.InstructPix2Pix was not trained on MagicBrush.Source: §10 · p. 20
• MagicBrush-168 outperforms SmartEdit on L1, CLIP-I, and DINO-I, but SmartEdit performs better on CLIP-T and comparably to MagicBrush-52.The authors consider L1, CLIP-I, and DINO-I potentially unreliable, while CLIP-T appears more consistent with the actual edits.Source: Table 4 · p. 20, §10 · p. 20

11. Difference between SmartEdit, MGIE and InstructDiffusion

• Compared with MGIE [9], SmartEdit targets complex understanding and reasoning, adds a Bidirectional Interaction Module for image–LLM exchange, and uses a synthetic editing dataset to strengthen perception and reasoning.The paper reports quantitative and qualitative evidence that SmartEdit can handle complex understanding and reasoning scenarios.Source: §11 · p. 20
• Unlike InstructDiffusion’s generic vision-task alignment framework, SmartEdit focuses on instruction-based editing and combines LLaVA, BIM, segmentation data, and synthetic editing data for complex scenarios.The authors say diffusion-model perception is crucial and initialize SmartEdit’s diffusion model with InstructDiffusion weights; despite its extensive perception-data training, InstructDiffusion’s complex-reasoning performance is described as standard.Source: §11 · p. 20