Source-linked AI summary
Imagic: Text-Based Real Image Editing with Diffusion Models
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, Michal Irani
TL;DR
Text-based image editing has been constrained by edit type, image domain, and auxiliary-input requirements. Imagic uses a pre-trained text-to-image diffusion model with optimized embeddings and image-specific fine-tuning to edit a single real image from one target prompt. The paper reports complex non-rigid edits that align with the prompt while preserving the original image’s structure and composition.
Problem
Existing text-based image editing methods are limited to specific edits, synthetic or restricted domains, or auxiliary inputs such as masks and multiple images.
Method
Imagic optimizes a target-text embedding to match the input image, fine-tunes a pre-trained diffusion model, and interpolates embeddings to produce the edit.
Results
Imagic applies complex non-rigid semantic edits to single real high-resolution images while preserving their overall structure and composition.
Takeaways & Limitations
The method provides a unified framework for varied text-guided edits, including changes to posture, composition, style, color, and object content.
Takeaways & Limitations
Edits may be too subtle, alter extrinsic details such as zoom or camera angle, or inherit generative limitations and biases from the pre-trained diffusion model.
Abstract
from arXiv · showhide
Text-conditioned image editing has recently attracted considerable interest. However, most methods are currently either limited to specific editing types (e.g., object overlay, style transfer), or apply to synthetically generated images, or require multiple input images of a common object. In this paper we demonstrate, for the very first time, the ability to apply complex (e.g., non-rigid) text-guided semantic edits to a single real image. For example, we can change the posture and composition of one or multiple objects inside an image, while preserving its original characteristics. Our method can make a standing dog sit down or jump, cause a bird to spread its wings, etc. -- each within its single high-resolution natural image provided by the user. Contrary to previous work, our proposed method requires only a single input image and a target text (the desired edit). It operates on real images, and does not require any additional inputs (such as image masks or additional views of the object). Our method, which we call "Imagic", leverages a pre-trained text-to-image diffusion model for this task. It produces a text embedding that aligns with both the input image and the target text, while fine-tuning the diffusion model to capture the image-specific appearance. We demonstrate the quality and versatility of our method on numerous inputs from various domains, showcasing a plethora of high quality complex semantic image edits, all within a single unified framework.
1. Introduction
Imagic addresses limitations in text-based image editing by applying sophisticated semantic edits to a single real high-resolution image using only a target text prompt. It combines text-to-image diffusion modeling with image-specific optimization and fine-tuning to preserve the original image while aligning edits with the prompt.
- Motivation: Existing text-based editing methods are limited by edit type, image domain, or requirements for auxiliary inputs.Examples include object addition, style transfer, synthetic-image-only operation, masks, multiple subject images, or source-image descriptions.
- Contribution: Imagic performs sophisticated non-rigid edits on real high-resolution images from only one input image and one target text prompt.The method can edit multiple objects while preserving the original background, structure, and composition.
- Contribution: Imagic supports semantic, style, color, and object-addition edits within a single unified framework.Examples include making parrots kiss and making a person give a thumbs up.
- Method: The method first optimizes a text embedding to reconstruct the input, fine-tunes the diffusion model for fidelity, and interpolates embeddings for the final edit.The interpolation balances fidelity to the input image with alignment to the target text.
- Evaluation: Experiments across varied images report high-quality outputs that resemble the input and align with target text, with stronger editing quality and faithfulness for complex non-rigid edits.Human raters strongly preferred Imagic over other methods on the TEdBench benchmark.
2. Related Work
Related work includes GAN- and diffusion-based approaches for manipulating images, but the cited methods differ in their editing scope, conditioning, and control mechanisms. Imagic is situated among diffusion methods that edit real images while targeting more complex semantic changes.
- GAN-based methods: GAN-based image manipulation methods edit images through latent-space techniques, including optimization-based, encoder-based, and per-input model-adjustment approaches.These methods were developed for a variety of image manipulations, including operations on real images.
- Diffusion-based methods: SDEdit adds noise and denoises an image under an edit condition, but its cited formulation is limited to global edits.The process may also use user-provided brush strokes.
- Diffusion-based methods: Other diffusion approaches use DDIM inversion, text-and-image conditioning, or cross-attention manipulation for edited image synthesis and finer control.Hertz et al. can edit real images when DDIM inversion provides meaningful attention maps.
- Imagic: Imagic uses a probabilistic generative model, so different random seeds produce different edit options for the same input.The paper illustrates these alternatives in Figure 4.
3. Imagic: Diffusion-Based Real Image Editing
Imagic adapts text-to-image diffusion models for real-image editing through embedding optimization, image-specific fine-tuning, and interpolation toward the target text. The framework produces high-resolution edited images while retaining input details and supports multiple diffusion-model backbones.
- Method: Imagic uses a three-stage pipeline: optimize a target-text embedding for the input image, fine-tune the diffusion model, then interpolate toward the target embedding.The optimized embedding preserves proximity to the target text, while fine-tuning closes the reconstruction gap before editing.
- Method: Text embedding optimization matches the input image while keeping the optimized embedding near the original target-text embedding.The method uses relatively few optimization steps so interpolation remains meaningful in embedding space.
- Method: Model fine-tuning shifts the diffusion model to recreate the input image at the optimized embedding and preserves high-frequency details through auxiliary-model fine-tuning.Auxiliary models are conditioned on the target text during inference and are fine-tuned with reconstruction loss.
- Method: Linear interpolation between the optimized and target embeddings balances fidelity to the input image with alignment to the desired edit.The interpolated embedding drives base diffusion, followed by super-resolution using fine-tuned auxiliary models to produce the final high-resolution image.
- Implementation: Imagic is compatible with Imagen and Stable Diffusion, using their diffusion and super-resolution components to generate edited images.The framework is presented as agnostic to the choice of underlying generative model.
- Method comparison: Compared with SDEdit, DDIB, and Text2LIVE, Imagic preserves original image details while applying the desired edit, including complex non-rigid edits.The comparison passage specifically highlights performance on edits such as making a dog sit.
4. Experiments
Experiments show that Imagic supports diverse real-image edits, compares favorably with existing methods, and exposes a controllable fidelity–editability tradeoff. The method also has identifiable failure cases involving weak target-text alignment, camera changes, and inherited generative limitations.
- Qualitative Evaluation: Imagic applies style, appearance, color, posture, and composition edits to high-resolution real images across diverse domains.Experiments use images from Unsplash and Pixabay, generating edits with multiple random seeds before selecting results.
- Qualitative Evaluation: Imagic remains effective with Stable Diffusion, preserving image-specific appearance and producing smooth semantic interpolation as η changes.The method is described as agnostic to the underlying generative model choice.
- TEdBench and User Study: A user study measures preference rates with 95% confidence intervals for Imagic versus SDEdit, DDIB, and Text2LIVE on editing quality.The evaluation collected 9213 answers using a two-alternative forced-choice procedure on TEdBench.
- Comparisons: Imagic maintains high input fidelity while performing desired edits, significantly outperforming prior techniques on complex non-rigid edits such as making a dog sit.Comparisons include Text2LIVE, DDIB, and SDEdit on single real-world images using text prompts.
- Comparisons: Figure 7 compares pre-trained and fine-tuned models across η values, showing that fine-tuning preserves input details and delays divergence from the reconstructed image.Without fine-tuning, the method fails to fully reconstruct the input at η = 0 and loses details as η increases.
- Limitations: The method’s main failures are subtle or absent edits, camera-angle or zoom changes, and detail loss when η is increased to strengthen an edit.Imagic also inherits generative limitations and biases from its underlying diffusion model, and its optimization is slow for direct user-facing deployment.
5. Conclusions and Future Work
Imagic edits a single real image from one target text prompt, preserving image details while supporting sophisticated non-rigid and simpler semantic edits. The authors identify fidelity, identity preservation, random-seed sensitivity, interpolation sensitivity, and societal misuse as future concerns.
- Imagic accepts one image and one target text prompt, then edits pose, geometry, composition, style, color, or objects while preserving image details.The method uses a pretrained text-to-image diffusion model, embedding optimization, fine-tuning, and interpolation.
- The method is presented as the first approach for sophisticated text-based semantic editing of a single real high-resolution image, including multiple-object edits.Examples include changing object pose and composition, alongside style, color, and object-addition edits.
- Future work includes improving fidelity and identity preservation, reducing sensitivity to random seeds and interpolation parameter η, and automating η selection.These concerns are stated as directions for further development rather than as quantified limitations.
- Because it uses text-based generative models, Imagic remains susceptible to societal biases and potential malicious use for misleading synthetic imagery.The paper notes that reliance on the input image may reduce, but does not eliminate, these concerns.
A. Additional Results
Additional results show that Imagic supports diverse edit categories, smooth interpolation between original and edited images, and operation with different diffusion backbones. Together, these examples illustrate the method’s breadth and model agnosticism.
- Imagic handles posture, composition, multiple-object editing, object additions and replacements, style changes, and color changes on 1024 × 1024 image pairs.The figure presents original images alongside edited outputs and target texts.
- Smooth interpolation results show gradual transitions between the input image and edited image when using Imagic with Stable Diffusion.The supplementary material includes animated examples of these transitions.
- Imagic’s formulation is agnostic to the diffusion model choice, with the same requested edits demonstrated using Imagen and Stable Diffusion.The figure compares multiple examples across the two model backbones.
B. Ablation Study
The ablation study examines embedding optimization, fine-tuning, interpolation, optimization length, and random seeds. It finds that fine-tuning and suitable optimization improve reconstruction and editing, while stochasticity can produce failures.
- Ablation scope: Imagic consists of text embedding optimization, model fine-tuning, and interpolation, whose contributions are evaluated through ablation studies.The supplementary analysis focuses especially on embedding optimization and sensitivity to optimization steps and random seeds.
- Text embedding optimization: Text embedding optimization supplies source and target embeddings for interpolation, avoiding an additional user-provided description of the input image.The source embedding is optimized near the target text embedding to match the given image.
- Optimization and fine-tuning: Fine-tuning improves image reconstruction and enables intermediate interpolation values to match both the target text and input image.Without adequate reconstruction, interpolation in the pretrained model may fail to apply the requested edit.
- Optimization and fine-tuning: 100 text embedding optimization steps and 1500 model fine-tuning steps are used for the paper’s examples.Optimizing for 1000 steps offers little post-fine-tuning improvement, can degrade results, and adds runtime cost.
- Random-seed sensitivity: Different random seeds can trigger edits at different interpolation thresholds and can sometimes cause unwanted changes before the desired edit appears.Some image-text inputs exhibit this behavior across all five tested seeds, which the authors classify as failure cases.
C. User Study Details
The user study evaluates complex non-rigid editing on TEdBench while standardizing randomness and hyperparameter goals across methods. Imagic receives more than 70% preference in every baseline comparison.
- Benchmark and evaluation: TEdBench contains 100 image-text input pairs for complex non-rigid image editing, evaluated through a human perceptual study on Amazon Mechanical Turk.Each evaluator sees a random subset of 20 images, including one duplicated pair for consistency checking.
- Evaluation setup: The comparison fixes one random seed for SDEdit, Text2LIVE, and Imagic, while DDIB is deterministic.Imagic’s η and SDEdit’s diffusion timestep are selected to apply the edit while preserving maximal original-image detail.
- Evaluation setup: The compared methods receive different auxiliary-text inputs: DDIB and Text2LIVE use additional descriptions of the original image.This input difference is reported as part of the comparison setup.
- Results: More than 70% preference was recorded for Imagic in every comparison against the baselines.The authors present TEdBench as a benchmark for future comparisons in complex non-rigid text-based real-image editing.