Source-linked AI summary
Contextual Residual Aggregation for Ultra High-Resolution Image Inpainting
Zili Yi, Qiang Tang, Shekoofeh Azizi, Daesik Jang, Zhan Xu
TL;DR
Learning-based inpainting is constrained by memory and training-data demands when processing images approaching 8K, and naive upsampling produces blurry results. The paper introduces Contextual Residual Aggregation to add contextual high-frequency residuals to a low-resolution prediction, enabling high-resolution completion. It reports 3x∼6x faster processing than state-of-the-art methods on 1K–2K images and an 82% FID reduction, while supporting end-to-end inpainting from 4K to 8K.
Problem
Existing learning-based inpainting methods face intractable memory use at ultra-high resolutions, worsening quality with larger holes, and difficult access to high-resolution training data.
Method
Contextual Residual Aggregation upsamples a low-resolution inpainted result and adds weighted high-frequency residuals aggregated from contextual patches.
Results
3x∼6x faster processing and an 82% FID reduction are reported versus state-of-the-art methods on high-resolution images between 1K and 2K, with end-to-end inpainting from 4K to 8K.
Takeaways & Limitations
The framework supports high-resolution inpainting using low-resolution network processing and training on small images.
Takeaways & Limitations
The model is prone to fail when the majority of a background object is missing.
Abstract
from arXiv · showhide
Recently data-driven image inpainting methods have made inspiring progress, impacting fundamental image editing tasks such as object removal and damaged image repairing. These methods are more effective than classic approaches, however, due to memory limitations they can only handle low-resolution inputs, typically smaller than 1K. Meanwhile, the resolution of photos captured with mobile devices increases up to 8K. Naive up-sampling of the low-resolution inpainted result can merely yield a large yet blurry result. Whereas, adding a high-frequency residual image onto the large blurry image can generate a sharp result, rich in details and textures. Motivated by this, we propose a Contextual Residual Aggregation (CRA) mechanism that can produce high-frequency residuals for missing contents by weighted aggregating residuals from contextual patches, thus only requiring a low-resolution prediction from the network. Since convolutional layers of the neural network only need to operate on low-resolution inputs and outputs, the cost of memory and computing power is thus well suppressed. Moreover, the need for high-resolution training datasets is alleviated. In our experiments, we train the proposed model on small images with resolutions 512x512 and perform inference on high-resolution images, achieving compelling inpainting quality. Our model can inpaint images as large as 8K with considerable hole sizes, which is intractable with previous learning-based approaches. We further elaborate on the light-weight design of the network architecture, achieving real-time performance on 2K images on a GTX 1080 Ti GPU. Codes are available at: Atlas200dk/sample-imageinpainting-HiFill.
1 Introduction
The paper targets the memory and data demands that limit learning-based inpainting on ultra-high-resolution images. It introduces CRA and a lightweight architecture that reconstruct high-frequency details from contextual residuals while operating on low-resolution inputs.
- Existing learning-based methods become memory-intensive at 8K inputs, while quality deteriorates as hole size increases and high-resolution training data is difficult to obtain.
- Contextual Residual Aggregation predicts a low-resolution inpainted image, upsamples it, and adds weighted high-frequency residuals aggregated from contextual patches.The residual-enhanced result restores sharp details while the generator operates only on low-resolution images.
- The method enables ultra-high-resolution inpainting up to 8K with considerable hole sizes and limited memory and computing resources.
- The model can be trained on small images and applied to large images, alleviating the need for high-resolution training datasets.
- The architecture combines slim and deep layers, attention score sharing, multi-scale attention transfer, and Light-Weight Gated Convolutions to improve quality, computation, and speed.
- The paper reports better inpainting quality than existing methods even when evaluated on low-resolution images.
2 Related Works
Related work spans copying-based, modeling-based, and learned mechanisms for handling irregular holes and contextual information. The paper builds on contextual attention while transferring residuals and reusing attention scores across abstraction levels.
- Vanilla convolutions can produce color inconsistency, blurriness, and boundary artifacts because they treat invalid hole pixels like valid pixels.
- Partial convolution masks and renormalizes convolutional responses using valid pixels, while gated convolution provides a learnable dynamic feature selection mechanism.
- Contextual attention matches patches inside and outside holes, then copies and aggregates weighted contextual patches to fill missing regions.
- Unlike prior contextual attention extensions, the method computes attention scores once and reuses them at multiple abstraction levels, reducing parameters and computation.
- The method decomposes images into low- and high-frequency components, with the low-frequency component obtained by averaging neighboring pixels.
3 Method
The method combines low-resolution image completion with contextual attention and residual transfer to produce sharp ultra-high-resolution inpainting while limiting computation. A lightweight coarse-to-fine generator and simplified training objectives support efficient inference.
- Overall Pipeline: The CRA pipeline downsamples a high-resolution input to 512 × 512, predicts a completed image, and upsamples it into a blurry large image.Contextual residuals are computed from the raw input and blurry image for later transfer.
- Contextual Attention: CRA computes patch affinities between masked and contextual regions, then transfers weighted contextual information across multiple feature-map levels.Attention scores are computed from high-level features and reused during multi-scale attention transfer.
- Residual Aggregation: Weighted contextual residuals are aggregated for hole patches and added to the upsampled blurry result to recover sharp missing details.The residual patches are sized to cover pixels consistently without overlap in the primary aggregation step.
- Network Architecture: The generator uses a two-stage coarse-to-fine architecture in which the coarse network hallucinates rough contents and the refine network predicts finer results.Both generator input and output are expected to be 512 × 512, with additional downsampling in the coarse network to enlarge receptive fields and reduce computation.
- Light Weight Gated Convolution: Light Weight Gated Convolutions reduce parameters and processing time relative to original gated convolutions while maintaining their dynamic feature-selection mechanism.The variants are depth-separable, pixelwise, and single-channel LWGC, differing in how the gate branch is computed.
- Training Objectives: Training uses adversarial and reconstruction losses, with WGAN-GP enforcing global consistency and L1 reconstruction encouraging consistency with the original image.The coarse network uses reconstruction loss, while the refinement network uses a weighted combination of reconstruction and GAN losses.
4 Experimental Results
Experiments evaluate CRA across datasets, resolutions, sampling choices, lightweight convolutions, learning-based methods, super-resolution techniques, and traditional PatchMatch-based tools. CRA maintains strong quality at larger image and hole sizes while reducing computational demands.
- Experimental Setup: The model is trained on 512×512 images from Places2, CelebA-HQ, and DIV2K, then tested from 512 to 8K resolutions.The final model has 2.7M parameters and uses two NVIDIA 1080 Ti GPUs during training.
- CRA Design Analysis: Averaging performs best for down-sampling in experiments on an HD dataset, while the two up-sampling operations must use consistent methods.Nearest-Neighbor, Bilinear, Bicubic, and Averaging were evaluated for down-sampling; three methods were evaluated for up-sampling.
- Lightweight Convolutions: The LWGCsc+LWGCpw configuration matches the quality of other tested configurations while requiring fewer parameters.LWGCsc+LWGCsc produces visible artifacts, whereas the other five configurations perform equally well in quality.
- Learning-based Comparisons: On 512×512 images, CRA performs equally well or slightly better than prior learning-based methods, while larger holes expose artifacts in several competitors.Other methods deteriorate more dramatically than CRA on larger images with bigger hole sizes.
- Super-resolution Comparisons: CRA produces sharper and more visually consistent hole regions than the compared super-resolution methods after low-resolution inpainting and up-sampling.The comparison includes SRGAN, Nearest Neighbor, and Bicubic up-sampling.
- Traditional Comparisons: PatchMatch-based methods generate clear textures but can distort structures relative to surrounding regions.The comparison includes Photoshop, Inpaint, and an open-source PatchMatch implementation.
5 Conclusion
The paper presents CRA as an efficient, high-quality approach to ultra-high-resolution inpainting. It reports stable quality as resolution and hole size increase, faster 1K–2K inference, lower FID, and end-to-end 4K–8K capability.
- Conclusion: CRA enables more efficient and high-quality inpainting of ultra high-resolution images.The conclusion characterizes the technique as Contextual Residual Aggregated.
- Conclusion: 3x∼6x faster inference and 82% lower FID than the state-of-the-art are reported for 1K–2K images.The reported speed applies to images of the same size on high-resolution tests.
- Conclusion: CRA is presented as the only learning-based technique enabling end-to-end inpainting on 4K∼8K images.The conclusion also states that increased resolution and hole size do not deteriorate quality or considerably increase processing time in this framework.
Network Architectures
The architecture combines coarse-to-fine generator networks, attention branches, and a discriminator, with detailed layer configurations specified for each component.
- Notation: The architecture notation defines K, S, C, and D as kernel size, stride size, channel number, and dilation rate.D is omitted when its value is 1.
- Coarse Network: The coarse network uses repeated convolutions, dilated layers, and three up-sampling stages after an initial 2× down-sampling.Its listed channels progress through 32 and 64 channels before producing a three-channel output.
- Attention Branches: The attention branches compute attention and transfer it at multiple abstraction levels before concatenating transferred features.The branches are specified for P l=1, P l=2, and P l=3.
- Discriminator: The discriminator applies six stride-2 convolutional layers before a fully connected scalar output.The listed channel counts are 64, 128, and 256.
More Test Results on Places2
Additional Places2 test results cover input sizes from 512×512 through 2048×2048.
- More Test Results on Places2: Additional Places2 results are presented for 512×512, 1024×1024, and 2048×2048 inputs.These results appear in Figures 8, 9, and 10, respectively.
Sources of High-Resolution Images
The paper identifies the internet-crawled sources of the high-definition images used in the main paper in Table 3.
- Table 3 presents the sources of the high-definition images used in the main paper.
- The high-definition images were crawled from the internet.
- The source information concerns images presented in the main paper.
Failure Examples & Limitation
The paper reports failure examples and visual test results across 512 × 512, 1024 × 1024, and 2048 × 2048 inputs, alongside ultra-high-resolution comparisons.
- Failure Examples & Limitation: The model is prone to fail when most of a background object is missing.The paper cites the bicycle and dog face examples as instances of this failure mode.
- Failure Examples & Limitation: Figure 11 presents failure examples of the model.