Source-linked AI summary
Aggregated Contextual Transformations for High-Resolution Image Inpainting
Yanhong Zeng, Jianlong Fu, Hongyang Chao, Baining Guo
TL;DR
High-resolution image inpainting struggles with distorted structures and blurry textures when filling large missing regions. AOT-GAN combines aggregated contextual transformations in its generator with a tailored mask-prediction task for discriminator training. It reports a 38.60% relative FID improvement over GatedConv in the most challenging Places2 setting and promising practical completions.
Problem
High-resolution inpainting remains challenged by reasoning from distant contexts and synthesizing fine-grained textures across large missing regions.
Method
AOT-GAN stacks AOT blocks for aggregated contextual transformations and trains its discriminator with a tailored mask-prediction task.
Results
38.60% relative improvement in FID over GatedConv is reported in the most challenging Places2 setting with a 50–60% ratio of holes.
Takeaways & Limitations
AOT-GAN achieves state-of-the-art high-resolution inpainting performance and promising completions in practical applications.
Takeaways & Limitations
AOT-GAN depends on masks that cover all affected pixels; incomplete masks can leave boundaries and produce noticeable artifacts.
Abstract
from arXiv · showhide
State-of-the-art image inpainting approaches can suffer from generating distorted structures and blurry textures in high-resolution images (e.g., 512x512). The challenges mainly drive from (1) image content reasoning from distant contexts, and (2) fine-grained texture synthesis for a large missing region. To overcome these two challenges, we propose an enhanced GAN-based model, named Aggregated COntextual-Transformation GAN (AOT-GAN), for high-resolution image inpainting. Specifically, to enhance context reasoning, we construct the generator of AOT-GAN by stacking multiple layers of a proposed AOT block. The AOT blocks aggregate contextual transformations from various receptive fields, allowing to capture both informative distant image contexts and rich patterns of interest for context reasoning. For improving texture synthesis, we enhance the discriminator of AOT-GAN by training it with a tailored mask-prediction task. Such a training objective forces the discriminator to distinguish the detailed appearances of real and synthesized patches, and in turn, facilitates the generator to synthesize clear textures. Extensive comparisons on Places2, the most challenging benchmark with 1.8 million high-resolution images of 365 complex scenes, show that our model outperforms the state-of-the-art by a significant margin in terms of FID with 38.60% relative improvement. A user study including more than 30 subjects further validates the superiority of AOT-GAN. We further evaluate the proposed AOT-GAN in practical applications, e.g., logo removal, face editing, and object removal. Results show that our model achieves promising completions in the real world. We release code and models in https://github.com/researchmm/AOT-GAN-for-Inpainting.
1 INTRODUCTION
High-resolution image inpainting must recover plausible structures and clear textures across large missing regions. AOT-GAN addresses context reasoning and texture synthesis with aggregated contextual transformations and tailored discriminator training, achieving a 38.60% relative FID improvement.
- Motivation: Existing diffusion- and patch-based approaches can handle narrow holes but often fail to hallucinate plausible contents in complex semantic scenes.Their limitation is insufficient reasoning for missing contents and textures in complex scenes.
- Motivation: High-resolution inpainting often produces distorted structures and blurry textures when reasoning over large missing regions.The challenges involve using distant image contexts and generating fine-grained presentation across large regions.
- Approach: AOT-GAN uses a generator with aggregated contextual transformations to capture distant contexts and rich patterns for context reasoning.The generator repeats AOT blocks that apply transformations at various dilation rates and aggregate their outputs.
- Approach: A tailored mask-prediction task trains the discriminator to distinguish detailed appearances of real and synthesized patches, facilitating fine-grained texture synthesis.This design accounts for whether patches originate inside or outside missing regions.
2 RELATED WORK
Image inpainting research progressed from diffusion and patch-based methods to deep feature learning and adversarial training. These approaches differ in how they propagate context, synthesize missing content, and encourage realistic textures, while each retains limitations for large or complex regions.
- Non-learning based Image Inpainting: Diffusion-based methods propagate boundary information into holes and can fill narrow sketches, but often introduce blur in large missing regions.They use boundary conditions and pixel propagation along isophotes.
- Non-learning based Image Inpainting: Patch-based methods copy and paste similar patches from known contexts or external databases to synthesize missing contents.Research has explored patch size, offset, filling order, and matching algorithms.
- Learning-based Image Inpainting: Deep image inpainting models synthesize more plausible contents for complex scenes than non-learning-based approaches.These models commonly combine deep feature learning with adversarial training.
- Learning-based Image Inpainting: Context-based deep models use distant image features, but non-local matching can repeat patterns and serialized dilated convolutions can encode predefined gridding patterns.These issues can distort structures or limit the richness of patterns captured for reasoning.
- Learning-based Image Inpainting: Style and perceptual losses reduce feature-space differences between inpainted and original images, yet clear texture generation remains challenging.GAN-based minmax training improves texture clarity by training a discriminator to distinguish real and inpainted images.
3 APPROACH
AOT-GAN addresses high-resolution inpainting by combining an AOT-block generator for distant-context reasoning with an SM-PatchGAN discriminator for texture synthesis.
- Generator overview: AOT-GAN uses an encoder, stacked AOT blocks, and a decoder to transform a masked image and binary missing-region mask into a completed image.The AOT blocks are the generator’s repeated building blocks.
- Aggregated Contextual Transformations: AOT blocks split convolution kernels, apply sub-kernels with different dilation rates, and merge their transformations to capture distant contexts and local patterns.Larger dilation rates expand the receptive field, while smaller rates focus on local patterns.
- Aggregated Contextual Transformations: Compared with standard residual blocks, AOT blocks enrich pathway combinations without adding model parameters or computation costs.The design is intended to capture patterns of interest for context reasoning while retaining computational efficiency.
- Gated residual connection: A gated residual connection updates features inside missing regions while retaining known features outside them.This spatially variant aggregation addresses discrepancies between input pixels inside and outside missing regions.
- Soft Mask-Guided PatchGAN: AOT-GAN trains its discriminator with a mask-prediction task that distinguishes synthesized missing-region patches from real context patches.SM-PatchGAN uses downsampled, Gaussian-filtered soft patch-level masks, and only synthesized missing-region predictions optimize the generator.
4 EXPERIMENTS
The experiments evaluate AOT-GAN on high-resolution inpainting benchmarks using free-form masks, established baselines, quantitative metrics, qualitative comparisons, user studies, ablations, and practical applications.
- Experimental program: The experimental program includes quantitative and qualitative evaluations, a user study, ablation studies, and practical applications.The paper also discusses limitations in a dedicated section.
- Datasets and evaluation: Evaluations cover Places2, CELEBA-HQ, and QMUL-OpenLogo, with 512 × 512 resizing and free-form masks used for training and testing.Places2 contains over 1.8 million images across 365 scene categories; CELEBA-HQ evaluates facial texture details, while QMUL-OpenLogo contains 27,083 logo images.
- Baselines: The benchmark compares AOT-GAN with CA, PEN-Net, PConv, EdgeConnect, GatedConv, HiFill, and MNPS.The compared methods use diverse designs, including contextual attention, cross-layer non-local filling, partial convolution, edge completion, gated convolution, contextual residual aggregation, and multi-scale patch synthesis.
4.3 Evaluation Metrics
The evaluation uses four objective metrics to assess pixel accuracy, image quality, structural similarity, and perceptual rationality, while also incorporating subjective assessment.
- Objective metrics: L1 error calculates mean absolute error between inpainted and original images to evaluate per-pixel reconstruction accuracy.
- Objective metrics: PSNR is a classical image-quality assessment used by many image-inpainting approaches.
- Objective metrics: SSIM compares inpainted and original images using luminance, contrast, and structure.
- Objective metrics: FID measures the distance between distributions of real and fake image features and is considered closer to human perception.
- Subjective assessment: Qualitative evaluations and a user study complement objective metrics for comprehensive comparison.
4.4 Quantitative Comparisons
Quantitative comparisons evaluate AOT-GAN against competing inpainting models on Places2 and ImageNet under high-resolution masking settings. AOT-GAN achieves the strongest reported Places2 performance, particularly in FID.
- Places2 protocol: Places2 testing uses 100 evaluation images per scene, randomly sampled free-form masks, and identical image-mask pairs across approaches.Hole-to-image ratios include settings such as 20-30%.
- Places2: 38.60% relative improvement in FID over GatedConv is reported for AOT-GAN in the 50-60% hole-ratio setting on Places2.GatedConv is identified as the most competitive model in this comparison.
- Places2: AOT-GAN outperforms state-of-the-art methods on all reported metrics in the Places2 quantitative comparison.The authors highlight FID because it has been verified as closer to human perception.
- ImageNet: On ImageNet, AOT-GAN is compared with MNPS using 2,000 test images and a center 224 × 224 square mask.Both models are trained at 512 × 512 under aligned center-square masking settings.
4.5 Qualitative Comparisons
Qualitative comparisons on complex Places2 scenes show differences in structure and texture reconstruction across AOT-GAN and several baselines. The accompanying table and figure materials frame comparisons through visual examples and user-study results.
- Baseline comparisons: CA, PEN-Net, and PConv tend to generate blurry textures, while EdgeConnect produces checkerboard artifacts.
- User study: The user-study table reports pair-wise comparisons with PConv, GatedConv, and real data on CELEBA-HQ and Places2.“Percentage” denotes the proportion of cases in which the AOT-GAN result is judged better than the comparison.
- Qualitative comparisons: AOT-GAN reconstructs more plausible contextual structures and clearer textures than the compared baselines across complex Places2 scenes.Examples include valley, street, field wild, and alcove, with enlarged patches shown alongside results.
- Baseline comparisons: GatedConv and HiFill distort structures through unreasonable repeating patterns in the qualitative results.
4.6 User Study
A pair-wise user study compared AOT-GAN with real data and two strong baselines, using anonymous trials and participant judgments of structure and texture quality.
- More than 30 participants cast 951 valid votes across 400 randomly sampled cases each from Places2 and CELEBA-HQ.Participants compared AOT-GAN with real data, PConv, or GatedConv without being told the missing-region locations.
- AOT-GAN outperformed other methods in most user-study comparisons.The study asked participants to select images with better structures and more realistic textures.
- 22.47% of CELEBA-HQ comparisons against real data were won by AOT-GAN.
4.7 Ablation Studies
Ablation studies evaluate aggregated contextual transformations, gated residual connections, and SM-PatchGAN through quantitative and qualitative comparisons on CELEBA-HQ.
- Aggregated Contextual Transformations: Aggregated contextual transformations improve L1 error, PSNR, SSIM, and FID over a single-branch module.More branches with more diverse dilation rates produce larger improvements; the final model uses four branches with rates 1, 2, 4, and 8.
- Aggregated Contextual Transformations: The final aggregated-transformations setting completes large facial holes with better structures and clearer hair textures than the single-branch comparison.
- Ablation Studies: The full AOT-GAN combining aggregated contextual transformations, gated residual connections, and SM-PatchGAN performs best in the component ablations.The ablations use CELEBA-HQ images with 40–50% mask-to-image ratio masks.
- Gated Residual Connections: Gated residual connections outperform identical residual connections, GatedConvolution, and feature concatenation across all reported metrics.Qualitative comparisons also show more realistic facial textures.
- Soft Mask-guided PatchGAN: SM-PatchGAN performs best especially in FID among PatchGAN, HM-PatchGAN, and SM-PatchGAN.Its mask-guided optimization distinguishes detailed appearances of synthesized missing-region patches from contextual patches, yielding clearer eye textures.
4.8 Real Applications
AOT-GAN is evaluated for logo removal, face editing, and object removal at high resolution, with reported visual results across complex and irregular missing regions.
- Face Editing: AOT-GAN generates coherent facial structures and realistic textures for 512 × 512 face-editing images.
- Object Removal: AOT-GAN shows promising results for filling user-provided irregular holes in 512 × 512 object-removal images.The object-removal evaluation trains on Places2 with irregular masks and tests user-provided masks.
- Failure Case: When a logo mask leaves boundaries outside the masked region, AOT-GAN tends to propagate them and produce noticeable artifacts.The limitation can require multiple interactive operations and refinements to include all affected pixels.
4.9 Limitations
The paper identifies two limitations: AOT-block settings are manually tuned, and performance depends on masks covering all affected pixels.
- Customized AOT Block: The number of AOT-block branches and dilation rates is empirically set, so the optimal configuration may need to be searched again when image size changes.The authors plan to investigate adaptive mechanisms.
- Mask Selection: Incomplete masks can leave boundaries outside the inpainting region, making them difficult to remove and causing propagated boundaries and noticeable artifacts.Multiple interactive operations and refinements may be required to include all affected pixels.
5 CONCLUSIONS
AOT-GAN combines aggregated contextual transformations with an enhanced discriminator for high-resolution image inpainting. Evaluations, ablations, user studies, and practical applications support its effectiveness.
- AOT-GAN learns aggregated contextual transformations and uses an enhanced discriminator for high-resolution image inpainting.The approach targets context reasoning and texture synthesis through complementary generator and discriminator designs.
- AOT-GAN outperforms state-of-the-art approaches by a significant margin in extensive evaluations, including a user study.The paper also conducts ablation studies to analyze each component.
- Results on logo removal, face editing, and object removal demonstrate promising completions in real-world applications.These applications are used to verify the model’s effectiveness beyond benchmark evaluation.