Source-linked AI summary
A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpainting
Junhao Zhuang, Yanhong Zeng, Wenran Liu, Chun Yuan, Kai Chen
TL;DR
High-quality versatile inpainting must handle both context-aware filling and text-guided object synthesis, which existing approaches do not readily combine. PowerPaint uses task-specific learnable prompts and tailored training within one model, achieving state-of-the-art performance across multiple inpainting tasks while also supporting object removal and controllable shape-guided inpainting.
Problem
Existing inpainting approaches typically target either context-aware filling or text-guided object synthesis, making it challenging to achieve both tasks simultaneously.
Method
PowerPaint introduces distinct learnable task prompts with tailored fine-tuning strategies for multiple inpainting tasks, plus negative-prompt use for removal and prompt interpolation for shape control.
Results
PowerPaint achieves state-of-the-art performance across multiple inpainting tasks, including text-guided object inpainting, context-aware inpainting, object removal, and shape-guided object inpainting.
Takeaways & Limitations
Task prompts provide a unified way to support diverse inpainting targets and extend the model to object removal and controllable shape-fitting applications.
Takeaways & Limitations
Shape-guided object inpainting is motivated by a limitation that prompts the authors to propose task prompt interpolation for more controllable results.
Abstract
from arXiv · showhide
Advancing image inpainting is challenging as it requires filling user-specified regions for various intents, such as background filling and object synthesis. Existing approaches focus on either context-aware filling or object synthesis using text descriptions. However, achieving both tasks simultaneously is challenging due to differing training strategies. To overcome this challenge, we introduce PowerPaint, the first high-quality and versatile inpainting model that excels in multiple inpainting tasks. First, we introduce learnable task prompts along with tailored fine-tuning strategies to guide the model's focus on different inpainting targets explicitly. This enables PowerPaint to accomplish various inpainting tasks by utilizing different task prompts, resulting in state-of-the-art performance. Second, we demonstrate the versatility of the task prompt in PowerPaint by showcasing its effectiveness as a negative prompt for object removal. Moreover, we leverage prompt interpolation techniques to enable controllable shape-guided object inpainting, enhancing the model's applicability in shape-guided applications. Finally, we conduct extensive experiments and applications to verify the effectiveness of PowerPaint. We release our codes and models on our project page: https://powerpaint.github.io/.
1 Introduction
PowerPaint addresses the challenge of combining context-aware filling with text-guided object synthesis in one versatile inpainting model. Learnable task prompts and tailored training strategies support multiple tasks, including object removal and shape-controlled object inpainting.
- Motivation: Context-aware inpainting produces coherent completions but struggles to synthesize novel objects, while text-guided approaches can lose context-aware filling ability.The differing task requirements make versatile high-quality inpainting challenging.
- Approach: PowerPaint uses distinct learnable prompts and tailored training strategies to support text-guided object inpainting and context-aware image inpainting within one model.Pobj is trained with object bounding-box masks and text descriptions, whereas Pctxt uses random masks and itself as the text prompt.
- Object removal: Using Pctxt as a positive prompt and Pobj as a negative prompt enables object removal by suppressing unwanted objects and promoting seamless background filling.This design uses classifier-free guidance during sampling.
- Shape-guided inpainting: Prompt interpolation enables controllable shape-fitting in object inpainting by balancing text-guided synthesis in the mask center with context-aware filling near its periphery.The method introduces Pshape and interpolates it with Pctxt according to the expanded mask area ratio.
- Contributions: PowerPaint is presented as the first versatile model achieving state-of-the-art results across multiple inpainting tasks, supported by quantitative and qualitative evaluations.The experiments cover a range of inpainting tasks and applications.
2 Related Work
Prior inpainting methods either reconstruct masked regions coherently from image context or use text-to-image models to synthesize specified objects. These approaches face limitations when required to support both context-aware filling and object synthesis.
- Image Inpainting: Early inpainting methods randomly mask image regions and optimize reconstruction, producing content coherent with the surrounding image context.These approaches include methods based on generative adversarial networks and related training schemes.
- Text-guided Inpainting: Text-to-image diffusion methods fine-tune pretrained models with random masks and image captions, improving text-guided inpainting.SD-Inpainting and ControlNet-Inpainting are described as examples built on Stable Diffusion.
- Limitations: Methods trained with paired object-description data tend to assume objects occupy missing regions, losing the ability to perform context-aware image inpainting.PowerPaint addresses this limitation by learning different task prompts for different inpainting tasks.
3 PowerPaint
PowerPaint fine-tunes a text-to-image diffusion model with task-specific prompts and training strategies for versatile inpainting. It supports context-aware filling, text-guided object synthesis, object removal, and controllable shape-guided inpainting.
- Task Prompts: PowerPaint introduces three learnable task prompts for text-guided object inpainting, context-aware image inpainting, and shape-guided object inpainting.Pobj, Pctxt, and Pshape are combined with tailored training strategies for different inpainting targets.
- Model Adaptation: The inpainting model extends a pretrained text-to-image denoising network with masked-image and mask channels, while retaining text conditioning.Its input concatenates the noisy latent, masked image, and mask, and training uses a noise-prediction objective.
- Context and Object Inpainting: Pctxt enables coherent context-aware filling without explicit content descriptions, whereas text guidance supports synthesis of novel objects that context alone cannot infer.Context-aware training uses randomly masked image regions and optimizes a learnable prompt; object synthesis uses paired object captions and descriptions.
- Object Removal: PowerPaint uses Pctxt as a positive prompt and Pobj as a negative prompt in classifier-free guidance to decrease the likelihood of generating objects for removal.The guidance scale w controls the combination of the two prompt-conditioned denoising predictions.
- Shape-Guided Inpainting: Prompt interpolation between Pctxt and Pshape provides user control over how closely generated objects fit the mask shape.Training randomly dilates segmentation masks, computes an area ratio α, and linearly interpolates the corresponding text embeddings.
- Training: PowerPaint is fine-tuned on Stable Diffusion with task prompts and segmentation data, using local text descriptions from segmentation labels and BLIP captions.Training uses the semantic segmentation subset of OpenImage v6 as the main dataset for multi-task prompt tuning.
4 Experiments
PowerPaint is evaluated against recent inpainting baselines across object, context-aware, outpainting, removal, and shape-guided tasks. The experiments report strong quantitative, qualitative, and user-preference results, while ablations examine task prompts and unified modeling.
- Experimental setup: PowerPaint is compared with recent competitive approaches across text-guided object, shape-guided, context-aware, and outpainting benchmarks.The evaluation uses OpenImages, MSCOCO, Places2, and Flickr-Scenery, with task-specific masks and metrics.
- Comparisons with State-of-the-Art: PowerPaint achieves state-of-the-art visual quality and text alignment for text-guided object inpainting while satisfying text and shape constraints.Bounding box masks are used for object inpainting and object layout masks for shape-guided inpainting.
- Comparisons with State-of-the-Art: PowerPaint outperforms baselines for context-aware inpainting, producing fewer random artifacts and more coherent backgrounds aligned with image context.The comparison includes text-free models and SD-Inpainting with default text prompts, while PowerPaint uses Pobj as a negative prompt.
- Comparisons with State-of-the-Art: PowerPaint demonstrates superior image and aesthetic quality for outpainting and achieves state-of-the-art qualitative results across multiple inpainting tasks.The reported comparisons cover text-guided object inpainting, context-aware image inpainting, and outpainting.
- User study: Users preferred PowerPaint across object inpainting, object removal, and outpainting, with 2,995 valid votes collected in the study.Object-inpainting preferences included shape, text alignment, and realism.
- Ablation Study: Learnable task prompts outperform unlearnable identifiers, while one unified PowerPaint model matches or sometimes exceeds task-specific models.The ablations support task prompts as a way to unify multiple tasks without compromising performance.
- Applications: Using Pobj as a negative prompt enables effective object removal, and interpolating Pctxt with Pshape controls shape-fitting to the mask.Shape-guided results are reported as faithful to both the mask shape and the text prompt.
5 Conclusions
PowerPaint is presented as a versatile inpainting model with state-of-the-art performance across multiple tasks. The authors attribute its effectiveness to task prompts and tailored training strategies, supported by extensive experiments and applications.
- 5 Conclusions: PowerPaint achieves state-of-the-art performance across multiple inpainting tasks through task prompts and tailored optimal training strategies.The paper verifies the model through extensive experiments and applications including object removal and controllable shape-fitting.
6 Appendix
The appendix provides supplementary code, models, materials, qualitative comparisons, and additional application results.
- 6 Appendix: Supplementary materials include code, models, qualitative comparisons, and applications for object removal and shape-guided inpainting with controllable fitting.The appendix also discusses combining the approach with another component, though the supplied passage is truncated.
A Qualitative Comparisons
PowerPaint is qualitatively compared with recent inpainting baselines across text-guided, shape-guided, context-aware, and outpainting tasks. The comparisons report strong alignment, coherence, visual quality, and versatility across these settings.
- Evaluation setup: The evaluation compares PowerPaint with state-of-the-art baselines using randomly sampled results to avoid cherry-picking.Baselines include Stable Diffusion, CN-Inpainting, SD-Inpainting, SmartBrush, LaMa, and LDM-Inpainting across the evaluated tasks.
- Text-guided object inpainting: PowerPaint achieves comparable text-guided object inpainting results that match both the text descriptions and input images.Stable Diffusion can lack context coherence, while CN-Inpainting and SD-Inpainting can misalign with prompts describing the inpainting region.
- Shape-guided object inpainting: PowerPaint and SmartBrush generate pleasing objects that match the prompt and image context while aligning with free-form object layouts.CN-Inpainting and SD-Inpainting may produce coherent completions but fail to satisfy the prompt, including a sea-turtle example.
- Context-aware image inpainting: PowerPaint provides high-quality, context-aware results without requiring input text prompts.The task is intended for reasonable completions coherent with image context, including automatic restoration or batch object removal.
- Image outpainting: PowerPaint expands input images with more reasonable and visually pleasing results across varied outpainting masks.LaMa struggles with large masks, while diffusion-based baselines generally produce pleasing results but can overlook image context.
B Application Results
The application results cover object removal and other extensions of PowerPaint. In object removal comparisons, PowerPaint is reported to outperform Adobe Firefly.
- Additional applications: The additional applications include controllable shape-guided object inpainting and integration with ControlNet.These applications are reported alongside the object-removal experiments.
- Object removal: PowerPaint outperforms Adobe Firefly for object removal in crowded image contexts.The comparison follows Adobe Firefly’s object-removal guidelines and reports superior results for PowerPaint.
B.1 Object Removal
PowerPaint applies its learned task prompts to object removal by using context-aware behavior positively and object-generation behavior negatively during classifier-free guidance.
- Prompt-based object removal: Removing objects from crowded image contexts is challenging because attention may focus excessively on surrounding context and copy nearby content.This issue is described as affecting Adobe Firefly in the comparison.
- Prompt-based object removal: PowerPaint designates Pctxt as the positive prompt and Pobj as the negative prompt for object removal.This classifier-free guidance setup suppresses undesired object-generation effects while using the learned task-specific patterns.
B.2 Controllable Shape-Guided Object Inpainting
PowerPaint interpolates between context-aware and shape-guided task prompts to control how closely an inpainted object fits its mask. The fitting degree trades contour conformity against a more flexible object shape.
- Controllable shape-guided object inpainting: Prompt interpolation enables a trade-off between context-aware background filling near mask contours and text-guided object inpainting at the mask center.The interpolation is between Pctxt and Pshape and provides a controllable shape-fitting degree.
- Controllable shape-guided object inpainting: At α = 0.95 with an accurate object layout, PowerPaint precisely follows both the text prompt and mask shape.With a rough bounding-box mask and α = 0.5, it generates a reasonable shape without excessively conforming to the mask.
B.3 PowerPaint with ControlNet
PowerPaint extends inpainting to additional controllable conditions by integrating four ControlNets, including edges, depth, boundaries, and pose. The resulting images align with the supplied conditions while maintaining high quality.
- B.3 PowerPaint with ControlNet: PowerPaint integrates canny edge, depth, hed boundary, and human pose ControlNets to guide the inpainting process.These experiments test compatibility with four types of external conditioning.
- B.3 PowerPaint with ControlNet: The model effectively generates high-quality images aligned with the provided ControlNet conditions.The reported visual results cover the ControlNet experiments shown in Figures 17 to 20.