Source-linked AI summary
Inpaint Anything: Segment Anything Meets Image Inpainting
Tao Yu, Runseng Feng, Ruoyu Feng, Jinming Liu, Xin Jin, Wenjun Zeng, Zhibo Chen
TL;DR
Existing inpainting systems often require fine masks and mainly fill removed regions with contextual content. Inpaint Anything composes SAM, inpainters, and AIGC models into click-based removal, prompt-based filling, and background replacement, with experiments indicating robust performance across varied images.
Problem
Existing inpainting methods typically require fine mask annotations, while conventional filling primarily uses contextual content rather than newly generated content.
Method
Inpaint Anything composes SAM for mask generation, state-of-the-art inpainters for removal, and AIGC models for prompted filling or background replacement.
Results
Experiments indicate that Inpaint Anything effectively handles diverse image content, resolutions, and aspect ratios across removal, filling, and background-replacement cases.
Takeaways & Limitations
IA provides mask-free image inpainting with a user-friendly workflow of clicking for removal and prompting for filling.
Abstract
from arXiv · showhide
Modern image inpainting systems, despite the significant progress, often struggle with mask selection and holes filling. Based on Segment-Anything Model (SAM), we make the first attempt to the mask-free image inpainting and propose a new paradigm of ``clicking and filling'', which is named as Inpaint Anything (IA). The core idea behind IA is to combine the strengths of different models in order to build a very powerful and user-friendly pipeline for solving inpainting-related problems. IA supports three main features: (i) Remove Anything: users could click on an object and IA will remove it and smooth the ``hole'' with the context; (ii) Fill Anything: after certain objects removal, users could provide text-based prompts to IA, and then it will fill the hole with the corresponding generative content via driving AIGC models like Stable Diffusion; (iii) Replace Anything: with IA, users have another option to retain the click-selected object and replace the remaining background with the newly generated scenes. We are also very willing to help everyone share and promote new projects based on our Inpaint Anything (IA). Our codes are available at https://github.com/geekyutao/Inpaint-Anything.
1. Motivation and Observation
Existing inpainting methods require detailed masks and usually restore removed regions only from context. Inpaint Anything combines SAM, state-of-the-art inpainters, and AIGC models for click-based removal, prompt-based filling, and background replacement.
- State-of-the-art inpainting methods can handle large regions and complex structures but typically require fine mask annotations for training and inference.
- SAM produces high-quality object masks from point or box prompts, offering a basis for simplifying mask selection.
- Inpaint Anything combines SAM, state-of-the-art inpainters, and AIGC models into a pipeline for object removal, content filling, and background replacement.
- For removal, users click an object, SAM generates a mask, and a state-of-the-art inpainter fills the resulting hole with contextual data.
- Stable Diffusion can generate new objects from text prompts to fill removed regions, while alternative prompts can generate replacement backgrounds around retained objects.
2. Methodology
Inpaint Anything composes foundation models into three schemes: Remove Anything, Fill Anything, and Replace Anything. The pipeline coordinates segmentation, inpainting, and generation through mask refinement, resolution preservation, and prompting practices.
- SAM, state-of-the-art inpainters, and AIGC models are composed to support removal, filling, and replacement workflows.
- Remove Anything: SAM-based Remove Anything follows clicking, segmenting, and removing, with an inpainter filling the masked hole using surrounding background context.
- Fill Anything: Fill Anything adds text prompting and uses Stable Diffusion to generate requested content inside the selected object hole.
- Replace Anything: Replace Anything prompts an AIGC model to generate a visually consistent background outside the selected object.
- Practice: Dilation refines discontinuous or hole-filled SAM masks, and larger masks give AIGC models more space for content creation during filling.
- Practice: Cropping or aspect-ratio preservation helps avoid fidelity loss when models requiring fixed 512 × 512 inputs process images.
- Practice: Simple prompts often produce satisfactory text-prompt inpainting results, whereas longer prompts may be less user-friendly despite sometimes impressive outputs.
3. Experiment
The authors evaluate Remove Anything, Fill Anything, and Replace Anything on images from COCO, the LaMa test set, and phone photographs. They report robust inpainting across varied content, resolutions, and aspect ratios.
- Experiments cover object removal, object filling, and background replacement using COCO images, the LaMa test set, and photographs taken by phone.
- The reported results indicate that Inpaint Anything is general and robust across diverse image content, resolutions, and aspect ratios.
4. Conclusion
Inpaint Anything is presented as a versatile, composable pipeline for mask-free, click-and-prompt image inpainting. The authors report support for varied high-quality inputs, including arbitrary aspect ratios and 2K resolution, with future extensions planned.
- IA combines Remove Anything, Fill Anything, and Replace Anything to enable mask-free image inpainting through clicking for removal and prompting for filling.
- The system is described as handling high-quality images with arbitrary aspect ratios and 2K resolution.
- The authors plan to extend IA toward practical functions such as fine-grained image matting and editing.