PaperlayerView exact revision on arXiv

Source-linked AI summary

Inst-Inpaint: Instructing to Remove Objects with Diffusion Models

Ahmet Burak Yildirim, Vedat Baday, Erkut Erdem, Aykut Erdem, Aysegul Dundar

arXiv:2304.03246v2cs.CV

TL;DR

Traditional image inpainting relies on user-drawn masks, and the paper addresses object removal specified instead through natural-language instructions. It builds GQA-Inpaint and presents Inst-Inpaint, reporting that its framework outperforms existing methods on this task.

  • Problem

    Traditional inpainting requires user-specified masks at inference, which can be tedious and prone to errors.

    Source: §1 · p. 1

  • Method

    The paper builds GQA-Inpaint and develops Inst-Inpaint, a latent-diffusion framework that removes objects from images using textual instructions.

    Source: §1 · p. 1, §4 · p. 5

  • Results

    Inst-Inpaint outperforms existing methods on the instruction-based inpainting task.

    Source: §6 · p. 11

  • Takeaways & Limitations

    The results demonstrate the potential of textual instructions as a more user-friendly and natural way to control image inpainting.

    Source: §6 · p. 11

    On complex GQA-Inpaint scenes, the autoencoder may reconstruct details such as text poorly, reducing output quality even when the target object is correctly removed.

    Source: §5.5 · p. 10

Abstract

from arXiv · show

Image inpainting task refers to erasing unwanted pixels from images and filling them in a semantically consistent and realistic way. Traditionally, the pixels that are wished to be erased are defined with binary masks. From the application point of view, a user needs to generate the masks for the objects they would like to remove which can be time-consuming and prone to errors. In this work, we are interested in an image inpainting algorithm that estimates which object to be removed based on natural language input and removes it, simultaneously. For this purpose, first, we construct a dataset named GQA-Inpaint for this task. Second, we present a novel inpainting framework, Inst-Inpaint, that can remove objects from images based on the instructions given as text prompts. We set various GAN and diffusion-based baselines and run experiments on synthetic and real image datasets. We compare methods with different evaluation metrics that measure the quality and accuracy of the models and show significant quantitative and qualitative improvements.

1 INTRODUCTION

Traditional inpainting requires user-drawn masks, while text-conditioned editing may be less suited to object removal. The paper introduces instruction-based inpainting, a real-image dataset, and a framework, and reports improvements over state-of-the-art text-based inpainting methods.

1 INTRODUCTION

  • Traditional inpainting requires user-specified binary masks at inference, which can be tedious and error-prone to draw.Inpainting must also account for global image context so filled regions harmonize with the rest of the image.Source: §1 · p. 1
  • Text-conditioned inpainting can be adapted from image editing, but the cited approach is better suited to changing object attributes or adding elements than removing objects.The passage gives Blended Latent Diffusion as an example.Source: §1 · p. 1
  • Instructional image inpainting specifies unwanted objects through text, without requiring users to provide binary masks.Inst-Inpaint is presented as an end-to-end framework that removes objects based on textual instructions.Source: §1 · p. 1
  • GQA-Inpaint is a real-image dataset built from GQA scene graphs and state-of-the-art instance segmentation and inpainting methods.It provides paired training data for the proposed task.Source: §1 · p. 1
  • Figure 1 presents sample Inst-Inpaint results on a new real-image benchmark dataset for instructional inpainting.The figure caption names the dataset GQA-Instruct.Source: Figure 1 · p. 2
  • The authors report significant improvements over state-of-the-art text-based image inpainting methods.They evaluate baselines with several metrics, including a novel CLIP-based inpainting score.Source: §1 · pp. 1–2

2 RELATED WORK

Prior work spans diffusion-based image editing, mask-conditioned inpainting, GAN-based inpainting, and diffusion-based inpainting. The paper situates its text-instruction removal task among editing methods and notes limits in existing approaches for this use case.

2 RELATED WORK

  • Text-prompt editing methods manipulate attention or fine-tune diffusion models on paired data, while SDEdit edits real images through brush-stroke modifications and noise-denoising.The paper identifies its instruction-based inpainting task as more related to editing than image synthesis.Source: §2 · p. 2
  • GAN-based inpainting commonly uses binary masks and reconstruction losses, while diversity-oriented variants described here are limited to single-object datasets such as faces.The passage notes that GANs are difficult to train for diverse scenes.Source: §2 · p. 2
  • RePaint conditions an unconditional diffusion model on unerased content by modifying denoising, while X-Decoder can segment prompted objects for erasure when combined with diffusion models.The paper describes X-Decoder as concurrent work on referring segmentation.Source: §2 · p. 2

3 DATASET GENERATION

GQA-Inpaint uses GQA images and scene graphs to create a real-image dataset for instruction-based object removal. Its construction selects suitable objects, derives masks and removal prompts, and generates inpainted targets.

3 DATASET GENERATION

  • GQA-Inpaint addresses the limitation of CoDraw and CLEVR, whose synthetic scenes are not directly applicable to real-image editing.Source: §3 · p. 2
  • GQA-Inpaint contains 147165 unique images and 41407 instructions, built from GQA’s 85K real-world images and scene graphs without human annotation.The scene graphs supply objects, attributes, relations, positions, and sizes for constructing paired examples.Source: §3 · p. 2, Figure 2 · p. 3
  • Objects are selected for plausible removal and unambiguous reference, while large, tiny, and part-like targets are excluded.The size thresholds are greater than half the image area and smaller than 0.0025%; the authors set them empirically by visual inspection.Source: §3 · pp. 3–4
  • Multiple Detectron2 and Detic segmentation setups cover COCO and LVIS categories, and the mask with the highest bounding-box IoU is selected.Selected masks are refined with CascadePSP, expanded by 11×11-pixel dilation, and inpainted with CRFill.Source: §3 · p. 4
  • Prompts identify a sole instance by object name and use scene-graph relations to distinguish multiple same-category instances.Relations are filtered by frequency, spatial position supplies a relation when none is defined, and one object attribute is randomly selected for augmentation.Source: §3 · pp. 4–5

4 METHOD

Inst-Inpaint is a conditional latent diffusion model that takes an image and textual instruction as inputs. It conditions denoising on the source image and instruction, with a frozen encoder and jointly optimized denoiser and transformer.

4 METHOD

  • Inst-Inpaint uses latent diffusion for computational efficiency, with an encoder and decoder that compress and reconstruct images and are trained separately in the first stage.The encoder is fixed during later training.Source: §4 · p. 5
  • The diffusion process progressively adds noise to the encoded target image, and a denoising network learns to predict that noise from noisy latents and the timestep.The model uses a U-net architecture.Source: §4 · p. 5
  • In the second stage, source-image latents are concatenated with noisy target latents, while cross-attention conditions U-net activations on text embeddings.A Transformer trained with the U-net processes BERT tokens to form in-domain instruction representations.Source: §4 · pp. 5–6
  • The encoder remains fixed during training, while the noise-prediction network and text transformer are optimized together.Source: §4 · p. 5
  • The authors train separate Inst-Inpaint models on GQA-Inpaint and CLEVR; CLEVR enables comparison with GAN methods they could not successfully train on real GQA-Inpaint.The GQA-Inpaint model uses a pretrained VQGAN encoder, whereas the CLEVR model uses an autoencoder with KL regularization.Source: §4 · pp. 5–6
  • Figure 7 sorts GQA-Inpaint object classes by occurrence and distinguishes removable or relational objects in blue from objects usable in referring expressions in orange.Source: Figure 7 · p. 6

5 EXPERIMENTS

Experiments compare Inst-Inpaint with diffusion and GAN-based alternatives on real GQA-Inpaint and synthetic CLEVR data, using image-quality and removal-related metrics. Results favor Inst-Inpaint in the reported comparisons, while its latent autoencoder can impair output quality on complex scenes.

5.1 Baselines

  • Inst-Inpaint is compared with Instruct X-Decoder, CLIPSeg, and InstPix2Pix; TIM-GAN and cMANIGAN comparisons are conducted on CLEVR.The authors report that they could not train the GAN-based instruction-erasing methods successfully on GQA-Inpaint.Source: §5.1 · pp. 6–7

5.2 Datasets

  • GQA-Inpaint has 173,715 unique training and 18,883 unique test source-target-prompt pairs, built from 44,500 unique training and 4,811 unique test source images.A subset using remove instructions and flipped source-target pairs with modified text for add instructions contains 5,404 training and 2,362 test images.Source: §5.2 · p. 7

5.3 Evaluation Metrics

  • Evaluation combines FID for image-distribution photorealism with CLIP-based measures of removal accuracy; RelSim is used on CLEVR but is difficult to apply to real images because it requires effective object detectors.CLIP Distance checks whether similarity to the prompted object decreases, while CLIP Accuracy checks whether its predicted class disappears from the top predictions.Source: §5.3 · pp. 7–8

5.4 Experimental Analysis

  • On GQA-Inpaint, Inst-Inpaint achieves better FID and CLIP scores than X-Decoder, InstPix2Pix, and CLIPSeg.The authors also report successful erasure in challenging scenarios; competing methods sometimes fail to find or fully erase the target.Source: Table 2 · p. 8, §5.4 · pp. 8–9
  • Average cross-attention maps focus on the prompted objects, and a separately trained UNet using those maps achieves better mask scores than CLIPSeg and X-Decoder.The authors note that a more powerful decoder could produce more detailed masks; the simple UNet is used for analysis, not as Inst-Inpaint’s explicit mask predictor.Source: §5.4 · p. 9, Table 3 · p. 9
  • On CLEVR, Inst-Inpaint achieves a better FID and comparable RelSim to TIM-GAN, with fewer artifacts in the reported qualitative comparison.The authors describe CLEVR as an easier setup than GQA-Inpaint and exclude real-image-trained diffusion baselines from this comparison.Source: Table 4 · p. 10, §5.4 · p. 10

5.5 Model Limitations

  • Because Inst-Inpaint uses latent diffusion with an autoencoder, complex GQA-Inpaint scenes, especially text, can be poorly reconstructed even when the target object is correctly removed.The authors suggest more powerful autoencoders or different optimization methods as possible ways to prevent these reconstruction problems.Source: §5.5 · p. 10

6 CONCLUSION

The paper presents instruction-based real-image inpainting without binary masks, Inst-Inpaint, and the GQA-Inpaint benchmark. Its experiments report that Inst-Inpaint outperforms existing methods, and the authors identify textual instructions as a more user-friendly, natural way to control inpainting.

6 CONCLUSION

  • The paper introduces real-image inpainting in which textual instructions specify unwanted objects without binary masks.It presents Inst-Inpaint, a diffusion-based framework, and the GQA-Inpaint benchmark.Source: §6 · p. 11
  • The authors report that Inst-Inpaint outperforms existing methods on the proposed task.They also suggest textual instructions offer a more user-friendly and natural way to control image inpainting.Source: §6 · p. 11
  • The authors plan to release GQA-Inpaint and Inst-Inpaint to facilitate research in this domain.Source: §6 · p. 11

Related papers

Loading 2304.03246v2…