Source-linked AI summary
BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion
Xuan Ju, Xian Liu, Xintao Wang, Yuxuan Bian, Ying Shan, Qiang Xu
TL;DR
Diffusion-based inpainting methods can suffer from semantic inconsistency, reduced quality, limited transferability, and burdensome backbone-specific adaptation. BrushNet addresses this gap with a plug-and-play dual-branch architecture that hierarchically injects pixel-level masked-image features into pretrained diffusion models. Across EditBench and the introduced BrushBench, it reports superior performance across seven metrics spanning image quality, masked-region preservation, and text alignment.
Problem
Existing diffusion inpainting approaches have limitations including incoherent results, text-influenced masked-image features, single-branch burden, and limited transferability across diffusion backbones.
Method
BrushNet adds a separate masked-image branch using VAE encoding, hierarchical UNet feature injection, and removed text cross-attention for plug-and-play pixel-level control.
Results
BrushNet achieves state-of-the-art performance across seven metrics covering image quality, masked-region preservation, and text alignment.
Takeaways & Limitations
BrushNet provides a plug-and-play inpainting architecture with flexible unmasked-region controllability and dedicated resources for segmentation-based training and evaluation.
Takeaways & Limitations
Performance depends heavily on the chosen base model and remains poor for unusually irregular masks or text that conflicts with the masked image.
Abstract
from arXiv · showhide
Image inpainting, the process of restoring corrupted images, has seen significant advancements with the advent of diffusion models (DMs). Despite these advancements, current DM adaptations for inpainting, which involve modifications to the sampling strategy or the development of inpainting-specific DMs, frequently suffer from semantic inconsistencies and reduced image quality. Addressing these challenges, our work introduces a novel paradigm: the division of masked image features and noisy latent into separate branches. This division dramatically diminishes the model's learning load, facilitating a nuanced incorporation of essential masked image information in a hierarchical fashion. Herein, we present BrushNet, a novel plug-and-play dual-branch model engineered to embed pixel-level masked image features into any pre-trained DM, guaranteeing coherent and enhanced image inpainting outcomes. Additionally, we introduce BrushData and BrushBench to facilitate segmentation-based inpainting training and performance assessment. Our extensive experimental analysis demonstrates BrushNet's superior performance over existing models across seven key metrics, including image quality, mask region preservation, and textual coherence.
1 Introduction
BrushNet addresses shortcomings in diffusion-based inpainting by separating masked-image processing from generation and hierarchically injecting pixel-level features into pretrained diffusion models. The paper also introduces BrushData and BrushBench for segmentation-based evaluation, reporting state-of-the-art performance across seven metrics.
- Challenges: Existing diffusion inpainting methods either modify sampling or use dedicated models, but both can produce incoherent results or require costly backbone-specific fine-tuning.Sampling modifications have limited mask-boundary and unmasked-context awareness, while dedicated models have limited transferability.
- Challenges: Dedicated models fuse noisy latent, masked image latent, mask, and text early, allowing text to influence masked-image features and burdening a single UNet branch.
- BrushNet: BrushNet adds a separate branch that uses a VAE encoder, hierarchical UNet feature injection, and text-cross-attention removal for pixel-level masked-image control.The design also supports plug-and-play use and flexible unmasked-region controllability.
- Datasets and Evaluation: BrushData and BrushBench support training and evaluation for segmentation-based inpainting, complementing EditBench for random brush masks.The evaluation spans image quality, masked-region preservation, and text alignment.
- Results: BrushNet achieves state-of-the-art performance across seven metrics covering image quality, masked-region preservation, and text alignment.
2 Related Work
Diffusion-based inpainting evolved from sampling modifications toward dedicated, content- and shape-aware models, but transferability to arbitrary pretrained backbones remains limited. BrushNet is presented as a plug-and-play alternative with flexible inpainting control.
- Diffusion-Based Inpainting: Diffusion methods gained popularity for inpainting because they provide high-quality generation, fine-grained control, and diverse outputs.
- Sampling Strategy Modification: Sampling-based methods such as Blended Latent Diffusion combine generated masked regions with copied unmasked regions during denoising.
- Dedicated Inpainting Models: Dedicated models fine-tune diffusion backbones for content- and shape-aware inpainting, including mask-boundary guidance and specialized masked-image inputs.
- Transferability: These specialized approaches are difficult to transfer to arbitrary pretrained models, while ControlNet-based inpainting has limited perceptual understanding of masks and masked images.
- BrushNet: BrushNet is characterized as plug-and-play, content-aware, shape-aware, and capable of flexible preservation control.
3 Preliminaries and Motivation
Diffusion models denoise noisy latent representations through iterative forward and backward processes, while prior inpainting adaptations expose limitations in blending accuracy, feature purity, computational burden, and transferability. These limitations motivate a dedicated masked-image branch.
- Diffusion Models: Diffusion models add Gaussian noise to clean samples in a forward process and iteratively denoise from z_T to z_0 in a backward process.The denoiser estimates noise at each timestep conditioned on C.
- Diffusion Models: The diffusion denoiser is trained by minimizing the difference between sampled noise and its prediction conditioned on the noisy feature, timestep, and C.
- Sampling Strategy Modification: Blended Latent Diffusion encodes the masked image with a VAE, adds noise over T steps, and blends masked-image and generated latents using a resized binary mask.
- Motivation: Blended Latent Diffusion has suboptimal unmasked-region preservation and content alignment because mask resizing impairs blending and the model lacks mask-boundary and unmasked-context knowledge.
- Motivation: Dedicated inpainting models fuse conditions early, burden one UNet branch, and require extensive fine-tuning across diffusion-backbone variations.
- Motivation: These limitations motivate an additional branch dedicated to masked-image processing, although directly applying ControlNet remains unsuitable for pixel-level inpainting feature injection.
4 Method
BrushNet separates masked-image feature extraction from image generation through an additional branch, then inserts those features hierarchically into a frozen pretrained diffusion model. Pixel-space blending with a blurred mask improves boundary coherence while preserving the unmasked region.
- 4.1 Masked Image Guidance: BrushNet uses a dual-branch strategy that separates masked-image feature extraction from the image-generating process.The additional branch receives noisy latent, masked-image latent, and a downsampled mask as concatenated input.
- 4.1 Masked Image Guidance: The masked image is encoded with a VAE, while cubic interpolation resizes the mask to align with latent dimensions.The VAE aligns the masked-image latent with the pretrained UNet distribution.
- 4.1 Masked Image Guidance: BrushNet extracts features with a cross-attention-free clone of the pretrained diffusion model and inserts them layer-by-layer into the frozen diffusion network.Removing cross-attention keeps the additional branch focused on image information, while layer-wise insertion enables dense per-pixel control.
- 4.1 Masked Image Guidance: The feature-insertion equation adds zero-convolution-transformed BrushNet features to each pretrained UNet layer, scaled by a preservation factor w.The concatenated inputs are noisy latent, masked-image latent, and resized mask; w adjusts BrushNet’s influence.
- 4.2 Blending Operation: After denoising, BrushNet blends the generated and masked images in pixel space using a blurred mask to improve mask-boundary coherence.The method accepts a slight loss of boundary-detail accuracy because the resulting error is nearly imperceptible.
- 4 Method: BrushNet is designed for plug-and-play integration because it does not modify the pretrained diffusion model’s weights.The architecture supports integration with community fine-tuned diffusion models and flexible control through the added scale.
5 Experiments
The experiments introduce BrushBench and BrushData for segmentation-based inpainting, evaluate seven metrics across mask types, and compare BrushNet with diffusion-based baselines. BrushNet shows strong quantitative and qualitative performance, broad domain coverage, flexible control, and design choices supported by ablations.
- 5.1 Evaluation Benchmark and Metrics: BrushBench contains 600 balanced natural and artificial images with human-annotated masks and captions across humans, animals, indoor scenes, and outdoor scenes.Its balanced construction supports evaluation across image categories.
- 5.1 Evaluation Benchmark and Metrics: The experiments distinguish random brush masks from segmentation-based masks, with inside-inpainting and outside-inpainting treated as separate segmentation tasks.EditBench is used for random brush-mask comparisons, while BrushBench evaluates segmentation-based masks.
- 5.1 Evaluation Benchmark and Metrics: Seven metrics assess image generation quality, masked-region preservation, and text alignment, including Image Reward, HPS v2, Aesthetic Score, and CLIP Similarity.The image-quality metrics are selected to better reflect human perception of varied modern text-to-image outputs than FID and KID.
- 5.3 Quantitative Comparison: BrushNet outperforms the compared methods across image quality, masked-region preservation, and image-text alignment on BrushBench.The comparison includes Blended Latent Diffusion, Stable Diffusion Inpainting, HD-Painter, PowerPaint, and ControlNet-Inpainting.
- 5.3 Quantitative Comparison: Performance on EditBench is roughly consistent with BrushBench, indicating strong results across random, inside-inpainting, and outside-inpainting mask types.The paper reports BrushNet’s superior overall performance on both benchmarks.
- 5.4 Qualitative Comparison: Qualitative comparisons show coherent generated and unmasked regions, including background awareness when a goldfish is already present in the masked image.The authors attribute this background awareness to dual-branch decoupling.
- 5.4 Qualitative Comparison: The untouched pre-trained diffusion branch supports coverage across natural images, pencil paintings, anime, illustrations, digital art, and watercolor.The reported qualitative comparisons span multiple image domains.
- 5.5 Flexible Control Ability: BrushNet provides flexible control through base diffusion-model selection and a control-scale parameter that adjusts unmasked-region protection.These controls allow users to adapt the model to different inpainting requirements.
6 Discussion
BrushNet shows superior inpainting performance across image quality, masked-region preservation, and image-text alignment, while remaining subject to base-model, mask-shape, and text-alignment limitations. The paper also highlights social risks from biased training data and persuasive image misinformation.
- BrushNet achieves superior performance on BrushBench and EditBench across image generation quality, masked region preservation, and image-text alignment.
- BrushNet’s quality and content depend heavily on the selected base model, with domain mismatch producing incoherent results.The paper cites anime-focused MeinaMix applied to a natural image as an example.
- Unusually shaped or irregular masks and text that poorly aligns with the masked image remain challenging cases for BrushNet.
- Internet-collected training data may amplify social biases in image inpainting models.
- Manipulating human images with offensive elements creates a specific risk of persuasive misinformation.