Source-linked AI summary
Overpainting: Localized Context-aware Diffusion Image Editing
Sam Sartor, Iliyan Georgiev, Michael Fischer, Valentin Deschaintre, Pieter Peers
TL;DR
Precise, context-aware localization remains difficult for prompt-driven image editing because masks may be treated as suggestions rather than constraints. Overpainting addresses this with trimaps, adapted diffusion components, and curated training data, demonstrating versatile edits while retaining acknowledged quality and task-scope limitations.
Problem
Existing image editing models struggle to provide precise user-directed localization while preserving the previous content in the edited region.
Method
Overpainting uses white, gray, and black trimap regions with joint attention, cross-image low-rank adaptation, attention-dropout, and automated curated training-pair generation.
Results
The model demonstrates versatile editing operations that are difficult for prior editing models, with 50%-dropout balancing mask adherence and edit quality.
Takeaways & Limitations
Trimaps let users trade off region-of-interest accuracy against edit-localization precision, while the combined model balances mask adherence and edit quality.
Takeaways & Limitations
The model can lose quality relative to the base model and sometimes fails at object removal or adding human subjects, especially when mask shape constrains subject shape and pose.
Abstract
from arXiv · showhide
We present "overpainting", an image editing operation which offers both control over the location of the edit and awareness of the previous content in that location. The overpainted area is given by a trimap, where white-annotated pixels must be edited, gray-annotated pixels may be edited, and black-annotated pixels must not be edited. This enables both precise and loose control, depending on user intent. We implement overpainting by adapting a pretrained image editing diffusion model using a combination of joint attention and low-rank adaption across input images with attention-dropout to balance the information flow between noise, source and mask images. We present a novel, automated, training data generation pipeline that (1) generates a set of candidate image pairs leveraging existing language-based editing models, (2) carefully curates those pairs, and (3) extracts a trimap from each usable pair. We demonstrate the versatility of our overpainting model on a wide range of editing tasks.
1 Introduction
Overpainting addresses the difficulty of making prompt-driven local edits while preserving existing image content. It uses trimaps to balance localization control and editing freedom, supported by adapted diffusion modeling and automated training-data construction.
- Overpainting task: Overpainting makes prompt-driven local alterations while respecting previous content, letting users specify both which pixels change and how they change.The operation classifies pixels as must-edit, may-edit, or must-not-edit through a trimap.
- Trimap control: Trimaps assign white pixels to required edits, gray pixels to optional edits, and black pixels to protected regions, enabling precise or loose control.Binary masks provide exact control, while gray margins allow natural blending around uncertain or intricate boundaries.
- Model: The model adapts a pretrained editing diffusion model using joint attention, cross-image low-rank adaptation, and attention-dropout to balance edit quality with mask adherence.Joint attention and Teamwork provide complementary information-sharing strategies across source, mask, and noise streams.
- Data: The training pipeline generates candidate image pairs with existing editing models, filters under-editing, over-editing, and shifting failures, and extracts trimaps from usable pairs.The paper also introduces a hand-crafted benchmark containing 180 exemplars.
- Demonstration: The paper demonstrates overpainting across diverse editing examples that existing image editing diffusion models cannot readily attain.The contribution summary identifies overpainting, balanced information sharing, and automated training-pair generation as the central components.
2 Related Work
Related work spans diffusion-based image editing, inpainting, localized editing, and texture or material painting. Overpainting combines localized control with a trimap to address finer-grained region specification and existing-content awareness.
- Diffusion-based Image Editing: Diffusion-based editing models extend image generation toward prompt-conditioned, multimodal edits with increasing controllability and transformer-based architectures.This work builds on the broader shift from natural-image generation toward image editing.
- Inpainting: Inpainting synthesizes masked regions from surrounding content and prompts, but the related passage notes that it ignores existing content within the masked area.Overpainting is related to inpainting but targets edits that preserve awareness of the prior content in the edited region.
- Localized Editing: Localized-editing methods provide regional control through blending, agents, or semantic attention, but can suffer from latent drift, boundary inconsistencies, or difficulty with fine-grained edits.Mask-free methods are inherently limited to recognizable parts, motivating explicit trimap-based region specification.
- Texture and Material Painting: Texture and material painting methods change an object's appearance within a region, including approaches that preserve underlying fine details or transfer exemplar textures.These methods form a neighboring application area for region-based appearance editing.
3 Method
Overpainting adapts a pretrained editing diffusion model to perform localized, content-aware edits from trimaps while training on automatically generated and curated image pairs. The method combines joint attention, cross-stream low-rank adaptation, and attention-dropout to balance edit quality with mask adherence.
- Overpainting formulation: Trimaps specify pixels that must change, may change, or must remain unchanged, enabling precise or loose control over localized edits.The model takes a source image, noisy latent, edit prompt, and trimap, and outputs the edited image.
- Model adaptation: Joint attention and Teamwork provide complementary benefits: flexible edits versus stronger positional mask adherence.Joint attention can connect different spatial positions across streams, while Teamwork exchanges information through linear layers with same-position bias.
- Model adaptation: Attention-dropout balances the two information-sharing mechanisms because naïvely combining them does not improve performance.The method randomly drops attention between streams so Teamwork is sometimes the only coordination mechanism.
- Automated training data generation: The training-data pipeline generates candidate edits with Qwen and FLUX.1 Kontext, filters under-editing, over-editing, and shifting, then extracts masks and trimaps.Approved source–edited pairs are passed to a mask-estimation network, and estimated alpha masks are converted into trimaps during training.
- Automated training data generation: Randomized gray regions are generated around binary-mask boundaries during training to mimic user-drawn trimaps.This augments precise estimated masks with variable-width regions that may be edited.
4 Evaluation
Evaluation shows that overpainting balances mask adherence and editing quality across curated examples, while preserving context and identity better than related local-editing approaches. The model also supports trimap workflows and inpainting, with quality trade-offs depending on training configuration.
- Prior Work Comparison: Overpainting better retains the background and edits only the intended foreground object compared with Layered Diffusion Brushes.The competing method adds a blue glow around the tower and changes the trees at its base.
- Overpainting strikes a balance between mask adherence and editing quality on 180 manually curated overpainting examples.The evaluation measures alignment, editing rate, F1, CLIP-IQA, EditCLIP, AlphaCLIP, and EditReward.
- Prior Work Comparison: SDEdit requires careful denoising-strength tuning because lower strength makes surfaces too flat, while higher strength degrades identity preservation.The comparison uses FLUX.1 Dev, a solid-color edit in white-mask regions, and fixed latents in black regions.
- Edit Quality vs. Mask Adherence: 50% attention-dropout provides a practical balance between mask adherence and edit quality without catastrophic cases where the mask is ignored.Increasing dropout improves mask adherence but reduces editing quality, producing a Pareto frontier rather than one universally best setting.
- Inpainting Comparison: Overpainting better retains the identity of masked regions than inpainting models in the illustrated face-tiling example.The combined inpainting/overpainting model achieves comparable inpainting quality to FLUX-Fill, but specialized overpainting quality is higher without inpainting training data.
- User Study: Seven of 11 participants preferred the trimap workflow, compared with three preferring binary masks and one neutral participant.Preferences were averaged across three test cases.
5 Conclusion
The paper concludes that overpainting enables prompt-driven localized edits that preserve the edited region’s context through trimap control. Its combined Teamwork–joint-attention model and automated training-data pipeline support difficult editing operations while exposing a trade-off between localization and edit quality.
- Overpainting makes prompt-driven localized edits while taking the context of the edited region into account.Its trimap lets users trade off region-of-interest accuracy against edit-localization precision.
- The model balances mask adherence and edit quality by combining Teamwork, joint attention, and attention-dropout.The conclusion identifies this trade-off as inherent to overpainting.
- An automated pipeline generates candidate image pairs, filters low-quality pairs, and extracts masks or trimaps for training.The pipeline leverages a pretrained diffusion editing model and retains high-quality training pairs.
- The model demonstrates varied editing operations that are difficult to achieve with prior editing models.
A Architecture Details
The architecture extends FLUX.1 Kontext with image-stream conditioning for source, mask, and noisy latents. It uses shared transformer processing, Teamwork LoRAs, and reshaping procedures that enable cross-stream attention and positional coordination.
- FLUX Kontext processes image patches and text tokens through 57 transformer blocks as a unified sequence.The first 19 are Double Blocks with separate text and image streams, while the remaining 38 are Single Blocks with shared parameters.
- The Teamwork batch trick reshapes coordinated image patches between batch-wise and sequence-wise representations before attention.The implementation separately computes text and image features in Single Blocks and applies attention dropout with a random block matrix.
- Source, mask, and noisy image inputs are distinguished with 3D RoPE image indices, using index 1 for the source and index 2 for the mask.The noisy latents retain index 0, and the mask is VAE encoded like other condition images.
B Automated Training Data Generation – Extended Description
The paper builds training data by generating edited image pairs, filtering failures, estimating edit masks, and converting them into randomized trimaps. This pipeline addresses degradation, misalignment, and the need to simulate imprecise user-drawn boundaries.
- Paired Image Generation: The usable training examples consist of an original image, edited image, estimated mask, and prompt describing the edit operation.The mask-estimation stage is required because the image-editing synthesis pipeline does not directly produce masks.
- Paired Image Generation: The pipeline generates paired exemplars by prompting Qwen2.5-VL-72B and FLUX.1 Kontext from 11,323 source photographs, supplemented by 60 manually selected images.The resulting edits include diverse prompts and seed-dependent positional variations.
- Filtering: 9,999 good training pairs remain after filtering under-editing, over-editing, and misaligned outputs.The filtering pipeline rejects pairs with insufficient or excessive local change and uses LightGlue keypoint motion to remove misaligned images.
- Mask Estimation: A head-expanded BiRefNet estimates edited pixels from original and edited images, outperforming thresholded pixel differences after finetuning on synthetic pairs.The expanded head is initialized as [W, −W] to resemble pixel-wise differencing, while finetuning improves robustness to degradation.
- Trimap Generation: Randomized low-frequency perturbations expand precise binary masks into trimaps with variable gray bands that mimic imprecise user specifications.The procedure uses signed distance fields, random thickness, and positive offsets so the true boundary does not always lie at the gray region’s center.
C Usability Study Details
The usability study compares binary masks with trimaps for reproducing fixed target edits. Participants generally preferred trimaps and achieved better self-rated similarity, although preferences varied across participants and examples.
- Setup: The study asked 11 participants to reproduce four target images once with a binary mask and once with a trimap, using fixed prompts and seeds.The first image served as a warm-up and its results were discarded, keeping the session under 15 minutes.
- Results Analysis: 7 of 10 participants leaned toward trimaps over binary masks, and trimaps were most preferred for every example.Three participants preferred binary masks, while preference varied with image and mask complexity.
- Results Analysis: Participants obtained better self-rated matches with trimaps than with binary masks.Similarity ratings were collected while participants reproduced targets using the same prompt and seed.