Source-linked AI summary
SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Model
Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, Kun Zhang
TL;DR
Generic inpainting offers limited control over newly generated content, motivating text- and shape-guided object inpainting. SmartBrush combines diffusion modeling, precision-controlled masks, foreground-mask prediction, and joint text-to-image training, with experiments reporting stronger overall quality, controllability, and background preservation than baselines. Its main stated limitation is handling shadows that extend beyond the available object bounding-box mask.
Problem
Traditional inpainting mainly borrows surrounding pixels and lacks flexible control over the generated object’s content and shape.
Method
SmartBrush uses diffusion-based object inpainting conditioned on text and masks with adjustable precision, predicts foreground masks for background preservation, and jointly trains text-to-image generation.
Results
Experiments report state-of-the-art object-inpainting results, improved background preservation, and user preference over DALLE-2 and Stable Inpainting across shape, text alignment, and realism.
Takeaways & Limitations
SmartBrush provides flexible control over how closely generated objects follow input masks while preserving surrounding background content.
Takeaways & Limitations
Long shadows extending beyond an object bounding box may not be generated because the coarsest mask is the bounding box.
Abstract
from arXiv · showhide
Generic image inpainting aims to complete a corrupted image by borrowing surrounding information, which barely generates novel content. By contrast, multi-modal inpainting provides more flexible and useful controls on the inpainted content, \eg, a text prompt can be used to describe an object with richer attributes, and a mask can be used to constrain the shape of the inpainted object rather than being only considered as a missing area. We propose a new diffusion-based model named SmartBrush for completing a missing region with an object using both text and shape-guidance. While previous work such as DALLE-2 and Stable Diffusion can do text-guided inapinting they do not support shape guidance and tend to modify background texture surrounding the generated object. Our model incorporates both text and shape guidance with precision control. To preserve the background better, we propose a novel training and sampling strategy by augmenting the diffusion U-net with object-mask prediction. Lastly, we introduce a multi-task training strategy by jointly training inpainting with text-to-image generation to leverage more training data. We conduct extensive experiments showing that our model outperforms all baselines in terms of visual quality, mask controllability, and background preservation.
1. Introduction
SmartBrush targets text- and shape-guided object inpainting, addressing text misalignment, mask misalignment, and background distortion through precision-controlled masks and foreground-mask prediction. It also uses localized supervision and joint text-to-image training to improve control and data utilization.
- Multi-modal object inpainting conditions generation on both a text description and the desired object shape, unlike traditional inpainting conditioned mainly on surrounding pixels.Diffusion models are explored for their ability to model complex image distributions and generate high-quality images.
- Existing methods can suffer from text misalignment, mask misalignment, and distorted background surrounding the synthesized object.Global captions may not match local masked content, while CLIP-based guidance may not encourage alignment with the mask.
- SmartBrush adds a precision factor so users can choose whether generated objects follow detailed masks closely or fit loosely within coarse masks such as bounding boxes.Masks are produced from fine to coarse by applying Gaussian blur to accurate instance masks.
- Foreground-mask prediction and replacement during sampling are used to preserve background content around objects, especially with coarse input masks.The model predicts an instance mask and uses it at test time to reduce unnecessary changes inside the input region.
- Experiments report state-of-the-art object-inpainting results, stronger background preservation, and user preference over DALLE-2 and Stable Inpainting across shape, text alignment, and realism.
- Training uses instance segmentation masks with local text descriptions rather than random masks and captions describing entire images.
- SmartBrush jointly trains object inpainting and text-to-image generation to leverage more training data.
2. Related Work
Diffusion models have become important for image synthesis and can be adapted to text-guided inpainting, but early adaptations introduce artifacts and require specialized fine-tuning to use image context.
- Diffusion models learn image distributions by reversing a Markov noising process and have shown strong performance in image synthesis compared with GANs.
- A direct text-to-image adaptation replaces random noise with noisy background outside the mask, but can generate partial objects or inconsistent background content.
- GLIDE fine-tunes a pretrained text-to-image model using randomly masked images and masks so the model learns to use information outside the missing region.
3. Preliminary: Diffusion Model
Diffusion models progressively add noise to images and train a network to reverse that process. Conditional information can be supplied during denoising, while iterative sampling reconstructs images from Gaussian noise.
- The forward diffusion process adds scheduled Gaussian noise over time, producing increasingly noisy latents and an approximately isotropic Gaussian at the final step.The total number of steps is denoted by T.
- The reverse process learns a neural network approximation to the unknown conditional distribution of a less-noisy latent given the current noisy latent.
- Training can predict the added noise instead of directly predicting the reverse-process mean, yielding the diffusion-model training objective.
- At test time, sampling starts from random Gaussian noise and repeatedly applies the learned network until the clean image is obtained.DDIM, PNDM, and classifier-free guidance can improve sampling efficiency or sample quality.
- For conditional diffusion, text or inpainting conditions are fed into the network without changing the loss function, enabling conditional image generation.
4. Our Approach
SmartBrush performs object inpainting conditioned on text and mask shape, with a precision control that spans accurate to coarse masks. Its diffusion training adds noise only inside the mask, predicts both noise and the accurate object mask, and uses the prediction during sampling to preserve background.
- Text and Shape Guided Diffusion: SmartBrush generates an inpainted image whose masked region aligns with both a text prompt and a specified object shape while preserving the surrounding background.The task uses image x, text prompt d, and binary mask m as inputs, targeting text- and mask-consistent content.
- Text and Shape Guided Diffusion: The model trains on instance or panoptic segmentation masks paired with object class labels, ensuring each masked region contains one object and enabling local text descriptions.Unlike random erasures, the training data uses annotated object regions and corresponding labels such as hat or cat.
- Text and Shape Guided Diffusion: Noise is added only within the masked region, while the model receives the noisy input, mask, and class text to recover the object using clean background information.At inference, random Gaussian noise is initialized inside the mask and the diffusion process is reversed for T sampling steps.
- Shape Precision Control: Gaussian-blurred masks create precision levels from accurate instance shapes to bounding boxes, and the diffusion model is conditioned on the precision indicator.The highest precision level preserves the original mask; the coarsest level becomes a bounding box without detailed shape information.
- Background Preservation: An auxiliary segmentation objective makes the diffusion network predict an accurate instance mask from the coarse input mask, which is then used during denoising to preserve background.The model adds an extra output channel for the instance-mask prediction and combines the segmentation and diffusion losses with λ = 0.01.
- Training Strategy: SmartBrush jointly trains inpainting and text-to-image generation with segmentation data and LAION-Aesthetics image-text pairs to support diverse content and descriptions.For text-to-image training, the input mask covers the entire image as a special inpainting case.
5. Experimental Evaluation
SmartBrush is evaluated against established inpainting baselines on text-guided object and scene inpainting, mask precision control, and background preservation. Across qualitative, quantitative, and user-study evidence, it follows text and mask guidance while preserving surrounding pixels.
- Experimental Setup: SmartBrush is evaluated on OpenImages and MSCOCO against Blended Diffusion, GLIDE, Stable Diffusion, Stable Inpainting, and limited DALLE-2 comparisons.Testing uses segmentation-derived masks and class-label prompts; results are assessed with local-region FID and text-alignment metrics.
- Quantitative Results: SmartBrush achieves the best performance on all reported metrics for both bounding-box and object-layout mask tasks.The paper attributes this result to training with text and shape guidance.
- Qualitative Results: Qualitatively, SmartBrush generates objects and scenes that follow both text and mask shape more reliably than the baselines.The comparison includes short and long text descriptions, while existing methods frequently fail at mask following or text alignment.
- User Study: More than 50% of surveyed users selected SmartBrush as best for mask alignment, text alignment, and realism.The user study involved more than 300 Amazon Mechanical Turk workers.
- Mask Precision Control: Finer masks make SmartBrush follow the specified shape more closely, whereas coarser masks permit greater freedom within the masked region.With box-like masks, generation can occur anywhere inside the box; Stable Inpainting changes object size but does not follow mask shape.
- Background Preservation: SmartBrush better preserves background pixels than DALLE-2 and its variant without background-preservation regularization.Foreground-mask prediction during sampling helps avoid changing non-object pixels inside coarse masks.
6. Conclusion, Limitation, and Future Work
The paper identifies mask misalignment, text misalignment, and background preservation as central challenges and proposes training and loss designs to address them. It reports superiority in quantitative and qualitative evaluations but notes a limitation for large shadows extending beyond the object mask.
- Conclusion: Existing text- and shape-guided inpainting models face mask misalignment, text misalignment, and background-preservation challenges.SmartBrush uses segmentation-derived text and shape guidance to address these problems.
- Conclusion: The proposed method creates fine-to-coarse masks for precision control and adds an object-prediction loss to avoid unnecessary changes inside the mask.The predicted object mask is used during sampling to preserve surrounding content.
- Conclusion: The paper reports that quantitative and qualitative results demonstrate the superiority of SmartBrush.
- Limitation: Large shadows extending beyond the object mask remain a limitation, because the coarsest mask is the object bounding box.The paper gives long human shadows as an example of this boundary.