Source-linked AI summary
Guiding Instruction-based Image Editing via Multimodal Large Language Models
Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfei Yang, Zhe Gan
TL;DR
Instruction-based image editing needs more expressive guidance because human commands can be too brief for existing methods. MGIE uses an MLLM with a diffusion model to derive concise visual-aware instructions and latent guidance jointly, and experiments report improved editing across task types while maintaining competitive efficiency.
Problem
Brief and ambiguous human instructions, together with CLIP’s static-description training, limit instruction-based image editing.
Method
MGIE jointly trains an MLLM and diffusion model to derive concise expressive instructions and transform visual tokens into editing guidance.
Results
MGIE improves instruction-based editing across Photoshop-style modification, global photo optimization, and local editing in automatic metrics and human evaluation.
Takeaways & Limitations
Visual-aware expressive instructions are crucial for effective instruction-based image editing, while MGIE maintains competitive inference efficiency.
Takeaways & Limitations
MGIE may inherit bias from its pretrained LLaVA and StableDiffusion foundation models.
Abstract
from arXiv · showhide
Instruction-based image editing improves the controllability and flexibility of image manipulation via natural commands without elaborate descriptions or regional masks. However, human instructions are sometimes too brief for current methods to capture and follow. Multimodal large language models (MLLMs) show promising capabilities in cross-modal understanding and visual-aware response generation via LMs. We investigate how MLLMs facilitate edit instructions and present MLLM-Guided Image Editing (MGIE). MGIE learns to derive expressive instructions and provides explicit guidance. The editing model jointly captures this visual imagination and performs manipulation through end-to-end training. We evaluate various aspects of Photoshop-style modification, global photo optimization, and local editing. Extensive experimental results demonstrate that expressive instructions are crucial to instruction-based image editing, and our MGIE can lead to a notable improvement in automatic metrics and human evaluation while maintaining competitive inference efficiency.
1 INTRODUCTION
Instruction-based editing improves accessibility but brief commands can be ambiguous and insufficient for current methods. MGIE uses MLLMs to derive expressive visual-aware instructions and jointly guides diffusion-based editing across multiple editing tasks.
- Instruction-based editing accepts direct human commands without elaborate descriptions or regional masks, improving practicality and accessibility.
- InsPix2Pix is limited by CLIP’s static-description training and brief, ambiguous instructions that inadequately specify the intended visual transformation.
- MGIE combines an MLLM and diffusion model to derive concise expressive instructions, provide visual-related guidance, and jointly perform editing end to end.
- MGIE evaluates Photoshop-style modification, global photo optimization, and local object alteration using human instructions across multiple datasets.
- Extensive experiments report improved automatic metrics and human evaluation, while identifying visual-aware expressive instructions as crucial to editing performance.
2 RELATED WORK
Prior instruction-based editing methods use direct commands, synthetic triplets, or human feedback, but remain constrained by static text encoders and ambiguous instructions. MLLMs extend vision-language modeling toward image-aware responses, and MGIE applies them to editing rather than generation from scratch.
- Instruction-based Image Editing: Diffusion models support image transformation, while instruction-based editing uses direct commands instead of elaborate descriptions or regional masks.
- Instruction-based Image Editing: Recent editing methods rely on synthetic input-goal-instruction triples or additional human feedback, but frozen CLIP encoders are not trained for editing transformations.
- Large Language Models for Vision: MLLMs align visual features with language and instruction tuning to perceive images and provide visual-aware responses.
- Large Language Models for Vision: Unlike recent MLLM applications that generate images from scratch, MGIE leverages MLLMs to improve editing through derived expressive instructions.
3 METHOD
MGIE converts an input image and instruction into concise expressive guidance and latent visual imagination, then uses a diffusion model to synthesize the edited image. Its end-to-end objective jointly trains instruction derivation and image editing.
- 3.1 BACKGROUND: MULTIMODAL LARGE LANGUAGE MODELS (MLLMS): An MLLM receives visual features and autoregressively generates language responses, but its output remains limited to text.
- 3.2 MLLM-GUIDED IMAGE EDITING (MGIE): MGIE prompts the MLLM with an image and instruction to derive a detailed editing explanation, then trains it to produce concise expressive guidance.
- 3.2 MLLM-GUIDED IMAGE EDITING (MGIE): During inference, MGIE generates concise instructions in 22.7 tokens rather than 64.5-token narrations and external summarization.
- 3.2 MLLM-GUIDED IMAGE EDITING (MGIE): The edit head transforms MLLM visual tokens into semantically meaningful latent guidance for the diffusion model.
- 3.2 MLLM-GUIDED IMAGE EDITING (MGIE): The diffusion model preserves the input latent while generating the goal-image latent under the visual guidance through latent-space denoising and cross-attention.
- 3.2 MLLM-GUIDED IMAGE EDITING (MGIE): Training combines instruction and editing losses in an end-to-end process, with most MLLM self-attention weights frozen for parameter-efficient optimization.
4 EXPERIMENTS
Experiments evaluate MGIE across datasets, editing settings, guidance strategies, visual alignment, human judgments, trade-offs, and inference efficiency. Results consistently support visual-aware expressive instructions and end-to-end training as effective for improving instruction-based image editing.
- Experimental Setup: MGIE is evaluated on Photoshop-style modification, global photo optimization, and local object alteration using multiple datasets and metrics.The evaluation uses EVR, GIER, MA5k, and MagicBrush, with task-specific image and text similarity measures.
- Quantitative Results: Zero-shot results show MGIE improves editing across tasks, including 59.2 SSIM on GIER, 66.3 SSIM on MA5k, and 82.2 DINO similarity on MagicBrush.The reported examples also include 0.3 photo distance on MA5k and 30.4 CTS global caption alignment on MagicBrush.
- Quantitative Results: Fine-tuning further improves MGIE, including 0.24 LPIPS on MA5k and 95.3 CVS on MagicBrush, while remaining superior to LGIE across editing aspects.The authors attribute the gains to domain-related, visual-aware guidance aligned with intended goals.
- Quantitative Results: MGIE surpasses InsPix2Pix across edit relevance and input consistency settings, receives the best human preference, and maintains competitive inference efficiency.Human evaluation covers instruction following, ground-truth relevance, and overall quality; inference takes 10 seconds for a single input on an NVIDIA A100 GPU and uses less than 32 tokens.
- Ablation Study: End-to-end training yields the greatest enhancements among frozen, fine-tuned, and end-to-end expressive-instruction strategies by learning applicable guidance and avoiding propagated instruction errors.The strongest improvements are reported for global optimization on MA5k and local editing on MagicBrush.
- Why MLLM Guidance is Helpful?: MGIE provides more input- and goal-aligned expressive instructions than language-only guidance, supporting clearer and more correlated edit guidance.Higher alignment with goal images is described as providing explicit edit guidance, while language-only guidance can produce irrelevant descriptions.
5 CONCLUSION
MGIE enhances instruction-based image editing by learning expressive instructions that provide explicit visual-aware guidance. Extensive studies show improved performance while maintaining competitive efficiency.
- MGIE learns to produce expressive instructions that enhance instruction-based image editing.
- The framework derives explicit visual-aware intentions from brief or ambiguous human commands to guide reasonable editing.
- Extensive studies across editing aspects demonstrate improved performance with competitive inference efficiency.
A ADDITIONAL RESULTS
Additional experiments examine baselines, model scaling, instruction forms, editing controls, unseen operations, and limitations. MGIE generally benefits from visual-aware expressive guidance, although some complex transfers and targeting cases remain challenging.
- Comparison to More Baselines: MGIE consistently surpasses HIVE without extra human feedback, while InsEdit is stronger for local editing but not global optimization.
- Does Larger LM Help?: Larger and more vision-aligned language models improve MGIE’s visual imagination and editing performance.
- Instruction Forms: Concise summarized expressive instructions provide better image alignment and inference efficiency than full or alternative “How” prompts.
- Visual Tokens: Using fewer than four visual tokens can make visual imagination insufficient, while performance becomes similar beyond four tokens.
- Evaluation: FID differences are limited because edited results often resemble inputs, making FID insufficient for comparing editing quality.
- Instruction Analysis: Expressive instructions contain more adjectives and adverbs than inputs, helping portray concrete edited scenes and clarify challenging verbs.
- Unseen Editing Operation: MGIE handles unseen removal and photo-optimization operations that InsPix2Pix fails on because its training data lacks such examples.
- Additional Editing Results: MGIE supports texture, color, emotion transfer, and object addition, but global colorization and deeper texture or emotion perception remain difficult.
B DETAILED EXPERIMENTAL SETUP
MGIE connects an MLLM to a latent-diffusion editor through a lightweight edit head and classifier-free guidance. Training jointly updates the models using visual imagination and image-conditioned denoising.
- Edit Head: A lightweight four-layer Transformer edit head converts MLLM word embeddings and visual-token hidden states into diffusion guidance.It produces L = 77 visual guidance vectors of dimension 768.
- Editing Loss: The diffusion model predicts noise from the encoded input image and MLLM-derived visual imagination for the goal image.
- Guidance: Classifier-free guidance combines unconditional, image-conditioned, and instruction-conditioned denoising during inference.
- Guidance: The guidance scales αV and αX control the trade-off between input-image similarity and instruction alignment, with defaults αV = 1.5 and αX = 7.5.
- Training Cost: MGIE training uses 26 epochs, about 1.6 hours per epoch on eight NVIDIA A100 GPUs, and completes in two days.
- Human Evaluation: Human evaluation samples 100 examples across four datasets, with three annotators ranking each task.
C ETHICS DISCUSSION AND LIMITATION
The authors discuss inherited foundation-model bias and challenging editing cases. Compositional commands, precise language grounding, and numerical perception remain limitations for practical instruction-based editing.
- Ethics: MGIE may inherit bias from the pretrained LLaVA and StableDiffusion foundation models.
- Ethics: End-to-end updating and concise instruction summarization may reduce harmful expression, while a safety checker can filter offensive outputs.
- Limitations: Compositional commands remain difficult in one step, with later manipulations sometimes failing after an initial successful edit.
- Limitations: Language grounding and numerical perception need improvement for accurate object targeting.