Source-linked AI summary
In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformer
Zechuan Zhang, Ji Xie, Yu Lu, Zongxin Yang, Yi Yang
TL;DR
Instruction-based image editing must balance precise instruction following against the data and computation demands of fine-tuning, while training-free methods have weaker instruction comprehension and layout preservation. ICEdit uses in-context prompts with large-scale DiTs, lightweight fine-tuning, and VLM-based early noise filtering. It reports state-of-the-art editing performance with 0.1% of prior training-data requirements and a competitive VIE score of 78.2 versus SeedEdit’s 75.7.
Problem
Current editing methods trade precision against efficiency: fine-tuning requires large datasets and resources, while training-free approaches struggle with complex instructions and preserving unchanged regions.
Method
ICEdit combines descriptive in-context prompts, minimal parameter-efficient fine-tuning, and VLM-based Early Filter Inference-Time Scaling for large-scale DiT editing.
Results
0.1% of prior training-data requirements accompanies state-of-the-art editing performance, while the method achieves a VIE score of 78.2 versus SeedEdit’s 75.7.
Takeaways & Limitations
ICEdit presents a precision-efficiency balance by leveraging large-scale DiT priors with limited data and lightweight adaptation.
Takeaways & Limitations
Without learned image-to-image editing priors, zero-shot DiT editing is unsatisfactory, particularly for preserving unedited regions.
Abstract
from arXiv · showhide
Instruction-based image editing enables precise modifications via natural language prompts, but existing methods face a precision-efficiency tradeoff: fine-tuning demands massive datasets (>10M) and computational resources, while training-free approaches suffer from weak instruction comprehension. We address this by proposing ICEdit, which leverages the inherent comprehension and generation abilities of large-scale Diffusion Transformers (DiTs) through three key innovations: (1) An in-context editing paradigm without architectural modifications; (2) Minimal parameter-efficient fine-tuning for quality improvement; (3) Early Filter Inference-Time Scaling, which uses VLMs to select high-quality noise samples for efficiency. Experiments show that ICEdit achieves state-of-the-art editing performance with only 0.1\% of the training data and 1\% trainable parameters compared to previous methods. Our approach establishes a new paradigm for balancing precision and efficiency in instructional image editing. Codes and demos can be found in https://river-zhang.github.io/ICEdit-gh-pages/.
1 Introduction
Instruction-based image editing faces a precision-efficiency trade-off: fine-tuning methods require large datasets and resources, while training-free methods often struggle with complex instructions and layout preservation. ICEdit addresses this through in-context generation, minimal fine-tuning, and VLM-based inference-time filtering, achieving strong performance with substantially less data.
- Motivation: Fine-tuning methods use large datasets and architectural changes for precise instruction following, whereas training-free methods are efficient but struggle with complex instructions.Existing approaches therefore trade editing precision against computational efficiency.
- Motivation: Large-scale DiTs offer strong text-to-image alignment and contextual awareness that can support image editing through intrinsic capabilities.Their attention mechanisms enable interactions between reference and generated content.
- ICEdit: ICEdit uses fixed-format in-context prompts to jointly process a source image and editing instruction without architectural modifications.The framework formats instructions descriptively so DiTs can interpret them more effectively.
- ICEdit: Minimal parameter-efficient fine-tuning and a Mixture-of-Experts structure improve editing success, quality, and robustness.The method targets DiTs’ unlearned image-to-image editing priors while retaining a lightweight training strategy.
- ICEdit: Early Filter Inference-Time Scaling uses VLMs to evaluate noise during early denoising and filter samples that conflict with textual instructions.This selects higher-quality candidates to improve output fidelity while maintaining computational efficiency.
- Results: 0.1% of prior training-data requirements accompanies state-of-the-art editing performance on the reported evaluations.The paper presents this result as a balance between editing precision and efficiency.
2 Related Work
Related work spans training-free editing and fine-tuned architectural approaches, while ICEdit explores in-context generation with large-scale DiTs. Prompt formulation is central: direct instructions can destabilize layouts, whereas descriptive in-context prompts improve interpretation.
- Training-free editing techniques: Training-free methods use inversion or attention manipulation to avoid retraining, but their editing performance and instruction handling remain limited.Examples include RF-Solver and StableFlow, which target inversion precision or selected DiT blocks.
- Fine-tuning-based editing methods: Fine-tuning-based methods modify diffusion architectures and train on high-quality datasets to improve image editing.Some methods additionally use multimodal language models to encode and inject instructions.
- Prompt design: Direct editing commands often alter the entire layout, whereas ICEdit’s descriptive diptych prompt significantly improves instruction interpretation.The evaluated alternatives also include global descriptive prompts containing full input and output captions.
- Training-free limitations: Training-free T2I and inpainting frameworks show some editing ability but perform unsatisfactorily, especially when preserving unedited regions.The T2I framework requires time-consuming inversion, while inpainting offers a more straightforward basis for further fine-tuning.
3 Method
ICEdit explores in-context editing with large-scale DiTs through diptych-based T2I and inpainting frameworks, then improves the selected paradigm with lightweight adaptation and inference-time noise filtering.
- 3.1 Exploration of DiT’s In-context Edit Ability: ICEdit tests T2I-DiT and inpainting-DiT training-free frameworks that place the source and edited images in a diptych.The T2I variant uses image inversion and attention injection, while the inpainting variant reconstructs the source beside the edited output using a fixed mask and prompt.
- 3.1 Exploration of DiT’s In-context Edit Ability: ICEdit eliminates intricate position and condition encoding designs by leveraging the diptych structure and DiT processing capabilities.The framework is designed to enable editing without architectural modifications or extensive fine-tuning.
- 3.1 Exploration of DiT’s In-context Edit Ability: The in-context edit prompt reformulates direct commands as descriptive diptych instructions, increasing editing success by approximately 70%.Direct commands often alter the entire layout, whereas the adopted prompt describes the right image as identical to the left except for the requested instruction.
- 3.1 Exploration of DiT’s In-context Edit Ability: The inpainting framework is selected for further fine-tuning because it avoids the time-consuming image inversion required by the T2I framework.Both zero-shot frameworks show limited performance, especially in preserving unedited regions.
- 3.2 Efficient Fine-tuning for Enhanced Editing: A 50K-sample dataset supports LoRA fine-tuning, producing a 150% improvement in editing success despite the compact training set.Because one LoRA structure struggles with diverse tasks such as style transfer and object removal, ICEdit adds parallel LoRA experts with routing based on visual tokens and text embeddings.
- 3.3 Early Filter Inference Time Scaling: Early Filter Inference-Time Scaling samples M initial noises, evaluates m-step preliminary edits with a VLM, and fully denoises the selected candidate.The method uses early edit alignment signals to identify a promising seed before applying n-step refinement.
4 Experiment
ICEdit is evaluated on Emu and MagicBrush benchmarks, where it combines strong editing quality with substantial data and inference efficiency. Ablations attribute gains to the in-context prompt, LoRA-MoE fine-tuning, and Early Filter Inference-Time Scaling.
- Benchmark Results: ICEdit achieves SOTA-comparable performance on Emu and MagicBrush, preserving fidelity and following instructions with fewer samples and parameters.On Emu, GPT-based scores surpass open-source models and rival closed-source Emu Edit despite using 0.5% training data.
- VIE-Score Evaluation: With inference scaling, ICEdit surpasses SeedEdit in overall VIE-Score while SeedEdit retains higher visual-quality scores but weaker unedited-region identity preservation.The VIE-Score combines instruction adherence and unedited-region preservation with visual quality.
- Model Structure: The IC prompt increases training-free GPT scores by 70%, while LoRA-MoE improves GPT scores by 13% over standard LoRA.Restricting adaptation to the output projection layer reduces performance, supporting fine-tuning across all modules.
- Applications: The generative prior produces harmonious, context-aware edits across tasks such as hand refinement and relighting.The method incorporates shadow effects and style alignment without additional tuning in the reported applications.
5 Conclusion
ICEdit presents a DiT-based instructional editing method that combines in-context generation, minimal fine-tuning, and early filtering to improve efficiency and precision. Extensive experiments report superior editing results and a balance between the two objectives.
- Conclusion: ICEdit combines an in-context edit paradigm, minimal fine-tuning with a mixture-of-experts structure, and VLM-based early filtering.The early filter selects optimal early-stage outputs from multiple seeds.
- Conclusion: The method delivers state-of-the-art performance with minimal fine-tuning data while balancing editing efficiency and precision.The conclusion frames this balance as a new paradigm for instructional image editing.
A Preliminary
DiTs use transformer denoising networks that process text and noisy image tokens through multimodal attention. Concatenated text and image tokens enable bidirectional interactions within the model.
- DiT Model: A DiT iteratively refines noisy image tokens using a transformer denoising network.The described models include FLUX.1, Stable Diffusion 3, and PixArt.
- DiT Model: Text condition tokens C_T and noisy image tokens X retain consistent shapes across transformer blocks.M and N denote the numbers of text and image tokens, while d is the embedding dimension.
- DiT Architectures: FLUX and FLUX-Fill use hybrid multimodal and parallel diffusion-transformer blocks scaled to 12B parameters.FLUX-Fill supports inpainting and outpainting using text descriptions and binary masks.
- Multimodal Attention: Multi-Modal Attention projects position-encoded tokens into queries, keys, and values, then attends across concatenated text and image tokens.Concatenation [C_T; X] facilitates bidirectional attention.
B.1 Dataset
The supplementary material describes ICEdit’s 50K-sample training setup, diptych-based in-context pipeline, implementation constraints, evaluation protocol, and comparison baselines. It also documents low-quality dataset samples and training-free editing artifacts.
- Dataset: The fine-tuning set contains 50K samples: 9K from MagicBrush and 40K randomly selected from OmniEdit.The dataset was not rigorously curated, and the public data contains suboptimal samples.
- Dataset: Despite using substantially less data, ICEdit outperforms models trained on MagicBrush and the full 1.2M-sample OmniEdit dataset.This comparison is presented as evidence that the in-context edit methodology contributes beyond the dataset itself.
- In-Context Framework: The T2I framework forms side-by-side diptychs, reconstructing the reference on the left while generating the edited result on the right.The pipeline inverts the reference image, concatenates inversion and random noise, and applies positional encoding to distinguish them.
- In-Context Framework: During denoising, retained inversion value features are injected into the left portion, allowing the right portion to preserve identity while following instructions.The training-free framework shows editing capability but can produce artifacts in identity preservation and layout maintenance.
- Implementation: The fine-tuning implementation uses four rank-32 expert LoRAs with TopK=1, plus standard rank-32 LoRA modules elsewhere.Training uses an effective batch size of 2 on four A800 GPUs for one day; inference uses an A100 GPU.
- Implementation: Training images are resized to 512×512 and formatted as 512×1024 diptychs.Reported VRAM usage with checkpointing is 37 GB at 512×512, 39 GB at 768×768, and 42 GB at 1024×1024 for batch size 1.
- Inference: The IC prompt describes a diptych whose right image keeps the left scene unchanged except for the specified instruction.Inference uses guidance scale 50 and 50 steps; early filtering randomly selects six noise samples.
B.4.2 Metrics
The benchmarks use complementary metrics for ground-truth resemblance, source preservation, instruction adherence, and perceptual quality, while excluding an inconsistent CLIP direction score.
- MagicBrush measures pixel-level differences with L1 and evaluates resemblance to ground truth using CLIP and DINO similarity.
- Emu Edit Test evaluates source-element preservation with CLIP and DINO image similarity and assesses text-image alignment without ground-truth images.
- CLIP text-image direction similarity is excluded because it can assign low scores to successful edits and high scores to failed edits.
- GPT-4o computes VIE-score, combining instruction adherence and editing accuracy with perceptual quality and naturalness.
C Discussion
The discussion identifies failure modes in editing and evaluation, including object movement, semantic ambiguity, VLM efficiency, and potential misuse of generated content.
- Object-relocation instructions may fail because general editing datasets provide insufficient motion-oriented training exposure.
- T5 may confuse polysemous terms such as computer mouse and animal mouse because of limited contextual disambiguation.
- Accurate inference-time quality assessment relies on Qwen-VL 72B, while smaller 7B models often misjudge quality.
- Parameter-efficient design could reduce computational requirements for small-scale developers and individual creators.
- The technology could be misused for deceptive content, motivating provenance standards, human oversight, and domain-specific ethical guidelines.
D.1 Ablation
Ablations examine expert configurations, inference-time filtering, noise seeds, CLIP-based selection, and training-data scale, alongside qualitative comparisons and examples.
- MoE Settings: Increasing expert rank from 8 to 32 or experts from 1 to 4 improves GPT-based evaluation scores, whereas more experts add parameters without notable gains.
- Inference Scaling Settings: Using 10 early inference steps outperforms 4 because four steps can produce low-quality images that mislead VLM judgments.
- Inference Scaling Settings: Increasing initial noise seeds from 1 to 6 significantly improves performance, but gains diminish from 6 to 12.
- Inference Scaling Settings: Full-step CLIP filtering increases computational cost while degrading GPT-based evaluation scores.
- Data Efficiency: Fine-tuning with 10K samples improves over training-free approaches, while gains gradually diminish and 70K adds little over 50K.
- Qualitative Results: Additional figures compare baselines, vary initial noise configurations, and show broader editing results across diverse scenarios.