Source-linked AI summary
Image inpainting: A review
Omar Elharrouss, Noor Almaadeed, Somaya Al-Maadeed, Younes Akbari
TL;DR
Image inpainting is an important computer-vision task, but comparing methods is difficult because datasets, masks, and distortion types vary. This paper reviews three method categories, catalogs datasets and evaluations, and concludes that no method handles every distortion, although learning-based approaches show promising case-specific results.
Problem
Image inpainting research lacks widely available datasets, and differing masks and distortion types make comparisons between methods difficult.
Method
The paper reviews sequential-based, CNN-based, and GAN-based approaches, organizing methods by distortion type and discussing datasets and evaluation metrics.
Results
No method can inpaint all distortion types, while learning techniques produce promising results for particular analyzed cases.
Takeaways & Limitations
The review provides a reference for comparing image-inpainting methods and datasets across distortion types and evaluation settings.
Takeaways & Limitations
GAN-based methods require substantial computational resources and have slower training, while dataset and mask differences constrain comparisons.
Abstract
from arXiv · showhide
Although image inpainting, or the art of repairing the old and deteriorated images, has been around for many years, it has gained even more popularity because of the recent development in image processing techniques. With the improvement of image processing tools and the flexibility of digital image editing, automatic image inpainting has found important applications in computer vision and has also become an important and challenging topic of research in image processing. This paper is a brief review of the existing image inpainting approaches we first present a global vision on the existing methods for image inpainting. We attempt to collect most of the existing approaches and classify them into three categories, namely, sequential-based, CNN-based and GAN-based methods. In addition, for each category, a list of methods for the different types of distortion on the images is presented. Furthermore, collect a list of the available datasets and discuss these in our paper. This is a contribution for digital image inpainting researchers trying to look for the available datasets because there is a lack of datasets available for image inpainting. As the final step in this overview, we present the results of real evaluations of the three categories of image inpainting methods performed on the datasets used, for the different types of image distortion. In the end, we also present the evaluations metrics and discuss the performance of these methods in terms of these metrics. This overview can be used as a reference for image inpainting researchers, and it can also facilitate the comparison of the methods as well as the datasets used. The main contribution of this paper is the presentation of the three categories of image inpainting methods along with a list of available datasets that the researchers can use to evaluate their proposed methodology against.
1. Introduction
Image inpainting addresses the removal or restoration of image distortions and has become an important computer-vision research topic as editing tools have advanced. The review organizes applications, methods, and evaluations across distortion types.
- Applications: Image inpainting applications include security-related object removal, image restoration, and other image-processing purposes.The supplied figures identify distortion types and application purposes, while the text describes restoration and object-removal uses.
- Problem and scope: Object removal matters because sophisticated editing can undermine public trust and make forgery detection difficult.The introduction identifies automatic recovery or extraction of the real image as an urgent need.
- Problem and scope: Image inpainting removes added objects or restores damaged regions, including text, blocks, noise, and scratches.It can restore texture and structure coherently without leaving visible artifacts.
- Review organization: The review covers sequential-based, CNN-based, and GAN-based image-inpainting methods.Its literature overview presents these categories alongside applications and distortion types.
2. Related works
Image inpainting approaches are reviewed across patch-based and diffusion-based methods, followed by sequential, CNN-based, and GAN-based techniques for completing missing regions or removing objects.
- Overview: Image inpainting completes missing regions or removes added objects, with applications including scratch, text, block, and crack removal.The operation varies by application domain, including restoration, photo editing, image coding, transmission, and virtual painting restoration.
- Patch-based and diffusion-based methods: Patch-based methods fill holes by finding matching patches in undamaged image regions and copying them into corresponding locations.Examples include Markov Random Field search and low-rank structured-matrix approaches.
- Patch-based and diffusion-based methods: Diffusion-based methods smoothly propagate image content from the hole boundary toward its interior.Some approaches localize diffusion and use intra-channel and inter-channel local variances to identify inpainted regions.
- Sequential-based approaches: Sequential-based methods can fill texture details, but capturing global image structure remains difficult.The review presents sequential-based methods in its literature tables alongside CNN- and GAN-based approaches.
- CNN-based approaches: CNN-based approaches use large-scale training data and include encoder-decoder, VGG, and text-erasing architectures for different restoration tasks.Reported applications include completing line drawings, repairing scanned data, addressing UAV blindspots, and removing text.
- GAN-based approaches: GAN-based methods use a generator and discriminator, with coarse-to-fine networks and contextual attention reported as helpful for inpainting.Their main limitation is slower training and substantial computational-resource requirements.
3. Image inpainting datasets
The review collects public datasets used to evaluate image inpainting across natural, artificial, face, RGB, RGB-D, and depth-image categories, while noting the scarcity of damaged-painting data.
- Dataset overview: The review compiles commonly used datasets including Paris StreetView, Places, depth image, Foreground-aware, Berkeley segmentation, ImageNet, and others.These datasets support evaluation and comparison across image categories and data types.
- Natural and scene datasets: Paris StreetView contains 15,000 street images from several cities, with a reported resolution of 936 × 537 pixels.The dataset is collected from Google StreetView.
- Natural and scene datasets: Places contains 10 million images across many scene categories, supporting large-scale deep-learning training.Examples include bedrooms, streets, synagogues, and canyons.
- Depth and mask datasets: Depth image datasets include RGB-D and grayscale depth images across 14 scene categories, with textual and random missing masks.They are used to evaluate depth-image inpainting methods.
- Depth and mask datasets: Foreground-aware provides 100,000 irregular-hole masks for training and 10,000 for testing, usable with arbitrary images.Each 256 × 256 mask marks hole pixels with 255 and valid pixels with 0.
- Other dataset categories: The review also describes datasets spanning segmentation, recognition, stereo, RGB-D, handwritten, vehicle, and face imagery.Examples include Berkeley segmentation, ImageNet, CelebA, Cityscapes, Middlebury, ICDAR 2013, SceneNet, and Stanford Cars.
4. Evaluation and discussion
The review catalogs datasets, distortion types, evaluation metrics, and reported performance across sequential-based, CNN-based, and GAN-based inpainting methods. Comparisons remain difficult because researchers use different datasets, masks, and distortions.
- Datasets: A lack of large damaged-image datasets leads researchers to modify existing datasets such as USC-SIPI, Paris StreetView, Places, and ImageNet.These datasets are damaged to train image-inpainting models and algorithms.
- Metrics: PSNR and SSIM are commonly used to evaluate repaired images, although metric scores do not always correspond to better qualitative results.The review notes that this mismatch is related to the requirement for a unique ground truth.
- Coverage: The literature covers text, noise, objects, scratches, RGB images, historical images, and RGB-D images, but methods differ in which distortions and image types they address.Some methods analyze three distortion types, whereas others process only text and noise.
- Evaluation design: The review organizes evaluations by datasets, distortion types, metrics, and image resolution across sequential-based, CNN-based, and GAN-based methods.Tables summarize method performance and evaluation settings for the different approach categories.
- Performance patterns: Performance depends on distortion type: block-damaged images have lower PSNR, while text- or line-based damage can be handled well by several algorithms.Accuracy also varies with noise percentage and dataset diversity.
- Limitations: Comparisons between methods are constrained by dataset unavailability and by authors’ use of different masks and distortion types.These differing evaluation conditions limit direct comparison across reported results.
5. Conclusions
The paper reviews three categories of image-inpainting approaches, their datasets, distortion types, and evaluation metrics. It concludes that no method handles every distortion, while learning-based techniques show promising results in analyzed cases.
- Contributions: The review covers sequential-based, CNN-based, and GAN-based approaches across text, objects, scratches, noise, RGB, RGB-D, and historical images.It also collects commonly used training datasets and tabulates evaluation settings.
- Conclusion: No method can inpaint all image-distortion types, but learning techniques produce promising results for each analyzed category.The conclusion is scoped to the cases analyzed in the review.