Source-linked AI summary
Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting
Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J. Fleet, Radu Soricut, Jason Baldridge, Mohammad Norouzi, Peter Anderson, William Chan
TL;DR
Text-guided image inpainting needs edits that follow prompts while preserving consistency with the input image. The paper introduces Imagen Editor, which uses object-aware masking and high-resolution conditioning, and EditBench, a systematic evaluation benchmark. Human evaluations find broad alignment gains from object masking and preference for Imagen Editor over DALL-E 2 and Stable Diffusion.
Problem
Text-guided image inpainting lacks reliable prompt faithfulness while maintaining consistency with the input image.
Method
The paper fine-tunes Imagen into Imagen Editor with object-detector-based training masks and image-mask conditioning at each cascaded diffusion stage, and introduces EditBench for systematic evaluation.
Results
Imagen Editor with object masking was preferred in 68% of comparisons against random masking, and was preferred over Stable Diffusion and DALL-E 2 in 78% and 77% of comparisons, respectively.
Takeaways & Limitations
Across EditBench, models render objects better than text and handle material, color, and size attributes better than count and shape.
Takeaways & Limitations
Text-guided image inpainting can amplify social harm and enable convincing misinformation, so the paper uses watermarks and avoids photorealistic human-face generation.
Abstract
from arXiv · showhide
Text-guided image editing can have a transformative impact in supporting creative applications. A key challenge is to generate edits that are faithful to input text prompts, while consistent with input images. We present Imagen Editor, a cascaded diffusion model built, by fine-tuning Imagen on text-guided image inpainting. Imagen Editor's edits are faithful to the text prompts, which is accomplished by using object detectors to propose inpainting masks during training. In addition, Imagen Editor captures fine details in the input image by conditioning the cascaded pipeline on the original high resolution image. To improve qualitative and quantitative evaluation, we introduce EditBench, a systematic benchmark for text-guided image inpainting. EditBench evaluates inpainting edits on natural and generated images exploring objects, attributes, and scenes. Through extensive human evaluation on EditBench, we find that object-masking during training leads to across-the-board improvements in text-image alignment -- such that Imagen Editor is preferred over DALL-E 2 and Stable Diffusion -- and, as a cohort, these models are better at object-rendering than text-rendering, and handle material/color/size attributes better than count/shape attributes.
1. Introduction
The paper addresses text-guided image inpainting by introducing Imagen Editor and EditBench. It uses object masking to improve prompt alignment and evaluates models across diverse editing dimensions.
- Imagen Editor extends Imagen through fine-tuning and adds image and mask context to every diffusion stage.Three convolutional downsampling image encoders provide this conditioning.
- Text-guided image inpainting must produce edits faithful to prompts while remaining consistent with image context.
- Object masking encourages models to rely more on text than randomly masked regions that can be inferred from image context.The method uses detected objects to construct training masks.
- EditBench evaluates inpainting across attributes, objects, and scenes using curated examples with masked images, prompts, and reference outputs.
- 68% of comparisons preferred object-masked Imagen Editor over its random-masking counterpart, while it was preferred over Stable Diffusion and DALL-E 2 in 78% and 77% of comparisons.Improvements covered all object and attribute categories.
- Models perform better on object rendering than text rendering and handle material, color, and size better than count and shape.CLIPScore was identified as the most useful metric for hyperparameter tuning and model selection.
2. Related Work
Related work evaluates text-guided editing through image quality, reconstruction fidelity, and text-image alignment. EditBench extends prior human-evaluation practice with fine-grained, masked-area comparisons across diverse feature axes.
- Text-Guided Image Editing: Prior text-guided editing methods balance consistency between input and edited images with consistency between text guides and outputs.
- Text-Guided Image Editing: Imagen Editor differs from earlier approaches through object-detector masking and architectural changes for high-resolution editing.
- Evaluation of Text-Guided Image Editing: Text-guided image inpainting has been evaluated using image quality, reconstruction fidelity, and text-image alignment.
- Automatic evaluation: Automatic evaluation commonly uses FID for image quality and CLIP-based metrics for text-image alignment.
- Human evaluation: EditBench extends side-by-side human evaluation with attribute, object, and scene axes while focusing judgments on the masked editing region.
3. Imagen Editor
Imagen Editor is a diffusion-based inpainting model fine-tuned from Imagen that uses the image, mask, and prompt to guide edits. Its design combines object-aware training masks with high-resolution conditioning for faithful editing.
- Imagen Editor takes an image, binary mask, and text prompt as inputs, using all three to guide output samples.
- High-Resolution Editing: Imagen Editor conditions its cascaded diffusion stages on high-resolution image and mask inputs through new convolutional image encoders.
- Object Detector Masking Policy: An off-the-shelf object detector generates masks that fully cover identified objects during training.This is intended to increase overlap between masked regions and text prompts.
- Object Detector Masking Policy: Object masking is designed to make inpainting less solvable from image context alone and therefore encourage attention to the prompt.
- High-Resolution Editing: Parameterized downsampling convolutions were found critical for high-fidelity editing, whereas bicubic downsampling produced artifacts along mask boundaries.
4. EditBench
EditBench is a curated benchmark for text-guided image inpainting with varied images, masks, prompts, and difficulty levels. It probes fine-grained editing behavior across attributes, objects, scenes, and mask sizes.
- EditBench contains 240 images, each paired with a mask specifying the region to modify through inpainting.
- Each image-mask pair has three prompts that represent different ways of specifying the edit.The benchmark includes natural and synthetic images.
- Evaluation categories: Its categories cover material, color, shape, size, count, common and rare objects, text rendering, and indoor, outdoor, realistic, and painting scenes.
- Image Collection: Synthetic images target object-attribute-scene combinations that may be unlikely to occur naturally and are manually selected from generated candidates.
- Image Masks: Masks are free-form, completely cover target objects, avoid revealing them through tight segmentation, and span a range of sizes.
- Prompt types: Mask-Simple, Mask-Rich, and Full prompts test basic masked-object description, complex attribute binding, and whole-image description.
5. Evaluation
The evaluation combines fine-grained and side-by-side human judgments on EditBench with automatic-metric comparisons. Imagen Editor leads text-image alignment, while performance varies by prompt complexity, object or attribute type, and metric setup.
- Human evaluation protocol: The evaluation covers four models using single-image and forced-choice side-by-side human judgments of text-image alignment and image quality.Single-image evaluations assess individual objects and attributes; side-by-side evaluations compare relative performance, with image quality evaluated only side by side.
- Human evaluation results: Imagen Editor receives the highest aggregated alignment ratings, 10-13% higher than the second-highest model across prompt types.For the remaining models, the usual order is IMRM > DL2 > SD, except that IMRM trails by 4-8% on Mask-Simple prompts.
- Human evaluation results: Models perform better on object-rendering than text-rendering, while Imagen Editor leads common, rare, and text-rendering object types by 10%, 11%, and 11%.Stable Diffusion’s text-rendering performance is 26%, compared with 59% for common objects and 44% for rare objects.
- Human evaluation results: Imagen Editor is preferred for text alignment against SD, DALL-E 2, and IMRM in 78%, 77%, and 68% of Mask-Rich comparisons, respectively.These alignment gains accompany similar image-quality performance, with differences of 0-6%.
- Human evaluation results: Imagen Editor is rated 13-16% higher than the second-highest model across most attribute types, but count remains comparatively difficult.Count is the exception: DALL-E 2 is 1% behind Imagen Editor, and Imagen Editor’s improvement over IMRM is 14%.
- Automatic evaluation metrics: CLIPScore based on text-to-image similarity agrees most with human judgments, identifying the best image in 68-76% of pairs and the best hybrid model in 39-48% of instances.Agreement is higher for Mask-Simple than Mask-Rich prompts, and CLIPScore works best when the image representation matches the prompt region.
6. Societal Impact
Text-guided image inpainting can create social harms, including amplified bias, convincing misinformation, and risks involving recognizable human likenesses.
- Training data can cause image-editing models to recapitulate and amplify social biases.
- Inpainting may enable the scaled creation of convincing misinformation by altering images of political figures.The paper gives editing an image of a political figure to include a controlled substance as an example.
- The authors report using distinctive watermarks and avoiding photorealistic human-face generation as mitigation approaches.
- The paper identifies robust guardrails as necessary to prevent recognizable likenesses of people from being generated and exposed to users.
7. Conclusion
The paper presents Imagen Editor for text-guided image inpainting and EditBench for systematic evaluation. Imagen Editor outperforms DALL-E 2 and Stable Diffusion on EditBench in human and automatic evaluations.
- Imagen Editor is a text-guided image-inpainting model fine-tuned from Imagen, using convolutional layers for high-resolution editing and object masking during training.
- EditBench is a comprehensive benchmark evaluating text-guided image inpainting across attributes, objects, and scenes.
- Imagen Editor outperforms DALL-E 2 and Stable Diffusion on EditBench in both human evaluation and automatic metrics.
A.1. Human Evaluation Results
The appendix analyzes human evaluations on EditBench by rendering criteria, mask size, comparison format, and statistical reliability. Object masking improves rendering and attribute binding, especially for smaller masks.
- Human Evaluation Results: Over 85% of Imagen Editor comparisons correctly render at least one of three object-attribute pairs in Mask-Rich prompts.
- Human Evaluation Results: Object masking improves object rendering, attribute rendering, and attribute binding compared with random masking.Attribute binding means that both the object and its attribute are correct.
- Impact of Mask Size: Object masking is more beneficial for small and medium masks than for large masks.
- Human Evaluation Results: Single-image evaluation supports fine-grained judgments and avoids the combinatorial explosion of comparing many model pairs.
- Human Evaluation Results: Randomized batch presentation to a large annotator pool controls for calibration biases in single-image evaluation.
- Human Evaluation Results: Imagen Editor differed significantly from other models across prompt types, with reported p-values of 3.5 × 10^-10, 2.0 × 10^-8, and 5.2 × 10^-7.
- Human Evaluation Results: The evaluation rated 11,520 outputs from 240 prompts, three prompt types, four image samples, and four models.
A.2. Imagen Editor object masking
Imagen Editor uses bounding-box-based masks during training by combining a random mask with an object-detection bounding box.
- The training mask is the union of a random mask and an object-detection bounding box.
A.3. Examples and Failure Cases
Examples show Imagen Editor’s object-masking strategy improves robustness to detailed prompts, while failures remain concentrated in difficult attribute types, especially count and shape.
- Examples: Object masking makes Imagen Editor more robust to prompts specifying multiple object details and attributes.Additional examples compare object-masking with random masking on Mask-Simple and Mask-Rich prompts.
- Failure Cases: Imagen Editor is relatively strong on common color and material attributes, but uncommon objects or colors can produce failures.Examples include butter-colored letters and a silver llama.
- Failure Cases: Size and shape attributes are more challenging because they can be ambiguous, and some size specifications are ignored.The cited examples include a tiny octopus and a pentagon-shaped block rendered instead of a cube-shaped block.
- Failure Cases: Count is the clearest failure case, with models usually rendering too few objects rather than too many.The failure-case discussion describes undercounting as occurring almost always, while overcounting is rare.
- Examples: Mask sizes are grouped into Small (5.7–21.5%), Medium (21.5–36.9%), and Large (>36.9%) buckets by mask-to-image area ratio.The buckets are formed from three quantiles of mask-to-image area ratios.