Source-linked AI summary
High-Fidelity Pluralistic Image Completion with Transformers
Ziyu Wan, Jingbo Zhang, Dongdong Chen, Jing Liao
TL;DR
Image completion needs both realistic textures and globally coherent, diverse structures, but CNNs struggle with global modeling and pluralism while transformers incur quadratic attention cost. The paper uses a transformer to reconstruct sampled low-resolution appearance priors and a CNN to replenish high-resolution textures, reporting higher fidelity and diversity with strong performance on large masks and ImageNet.
Problem
Image completion requires diverse, photorealistic results, but CNNs have limited global-structure modeling and deterministic outputs, while transformers have quadratic attention cost that hampers high-resolution processing.
Method
The method samples low-resolution appearance priors with a bi-directional transformer, then uses an upsampling CNN guided by those priors and visible pixels to render fine textures.
Results
The method outperforms state-of-the-art deterministic and pluralistic approaches in fidelity, diversity, and generalization, with ImageNet FID improved by more than 41.2 for large masks over PIC.
Takeaways & Limitations
Combining transformer structure modeling with CNN texture refinement yields high-fidelity pluralistic completion across regular evaluations, large masks, and large-scale datasets.
Abstract
from arXiv · showhide
Image completion has made tremendous progress with convolutional neural networks (CNNs), because of their powerful texture modeling capacity. However, due to some inherent properties (e.g., local inductive prior, spatial-invariant kernels), CNNs do not perform well in understanding global structures or naturally support pluralistic completion. Recently, transformers demonstrate their power in modeling the long-term relationship and generating diverse results, but their computation complexity is quadratic to input length, thus hampering the application in processing high-resolution images. This paper brings the best of both worlds to pluralistic image completion: appearance prior reconstruction with transformer and texture replenishment with CNN. The former transformer recovers pluralistic coherent structures together with some coarse textures, while the latter CNN enhances the local texture details of coarse priors guided by the high-resolution masked images. The proposed method vastly outperforms state-of-the-art methods in terms of three aspects: 1) large performance boost on image fidelity even compared to deterministic completion methods; 2) better diversity and higher fidelity for pluralistic completion; 3) exceptional generalization ability on large masks and generic dataset, like ImageNet.
1. Introduction
Image completion must balance realistic texture synthesis with global structural coherence and diverse outputs. This method combines transformer-based appearance-prior reconstruction with CNN-based texture refinement to address those challenges.
- CNNs learn rich texture patterns efficiently but struggle with global image structures because convolution provides local inductive priors.
- Spatial-invariant CNN filters can produce duplicated patterns or blurry artifacts in masked regions.
- CNN-based models are inherently deterministic, complicating the generation of diverse completion outputs.
- Transformers model long-term interactions and naturally support pluralistic outputs by optimizing the underlying data distribution.
- The proposed method decouples completion into transformer-based pluralistic appearance-prior reconstruction and CNN-based low-resolution upsampling for fine textures.
- Compared with deterministic and pluralistic state-of-the-art methods, the method improves fidelity, diversity, and generalization to large masks and ImageNet.The reported ImageNet FID improvement over PIC reaches 41.2 at most.
2. Related Works
Transformers provide the architectural basis for dense visual relationships, while pluralistic completion methods seek diverse outputs through learned distributions or shared latent spaces. Figure 2 presents this paper’s two-network pipeline as a related architectural combination.
- Transformers use stacked self-attention and point-wise feed-forward layers to model dense relationships among sequence elements.
- Figure 2’s bi-directional transformer produces a missing-region probability distribution from which diverse appearance priors are sampled.
- A second CNN upsamples each appearance prior to the original resolution under guidance from the masked input image.
- Prior pluralistic completion methods couple reconstructive and generative paths by jointly maximizing a variational lower bound.
3. Method
The method separates pluralistic appearance-prior reconstruction from texture replenishment, using a transformer for coherent structures and a CNN for high-resolution detail refinement.
- Method overview: The completion procedure factors into appearance-prior reconstruction p(X|Im) and texture-detail replenishment p(I|X, Im).The appearance prior is treated as a coarse intermediate representation before final image reconstruction.
- Appearance Priors Reconstruction: The transformer models p(X|Im) over low-resolution discretized appearance priors containing global structure and coarse textures.Low-resolution representations reduce attention cost while preserving structural information and coarse texture cues.
- Appearance Priors Reconstruction: Appearance priors use 32 × 32 or 48 × 48 resolutions and a 512-element visual vocabulary to reduce quadratic attention complexity and representation dimensionality.Masked regions are replaced with a special [MASK] token for transformer prediction.
- Appearance Priors Reconstruction: Bi-directional attention lets each token attend to all positions, while masked-language modeling predicts missing tokens from whole-image context.The transformer output is projected to a distribution over the visual vocabulary, and Gibbs sampling generates diversified complete token sequences.
- Guided Upsampling: A guided CNN upsamples the reconstructed appearance prior to original resolution using the masked input to render high-fidelity local details and preserve boundary consistency.The upsampling network combines an L1 loss with adversarial training; instance normalization is removed because it causes color inconsistency and severe artifacts.
4. Experiments
Experiments on FFHQ, Places2, and ImageNet evaluate implementation choices, qualitative and quantitative performance, diversity, robustness to large holes, and transformer confidence patterns. Across these evaluations, the method is reported to improve fidelity and diversity over deterministic and pluralistic baselines, including under large masks.
- Implementation Details: Experiments use 256 × 256 images from FFHQ, Places2, and ImageNet, with diversified irregular masks for training and evaluation.FFHQ reserves 1K images for testing; the other datasets use their original splits.
- Qualitative Comparisons: Qualitatively, the method is superior in photorealism and diversity, while CNN-based methods struggle with global context and large masks on ImageNet.The comparison includes deterministic and pluralistic baselines, including EC, MED, DeepFillv1/v2, and PIC.
- Quantitative Comparisons: Top-1 sampling achieves superior results against competitors in almost all metrics, while top-50 sampling slightly improves FID for relatively large mask regions.The experiments report PSNR, SSIM, MAE, and FID across FFHQ, Places2, and ImageNet comparisons.
- Quantitative Comparisons: On ImageNet, the method outperforms PIC by more than 41.2 FID points for large masks.The comparison is reported in Table 2 for ImageNet.
- User Study: The user study found the method 73.70% more likely to receive the first rank for photorealism.The study used 30 masked test images and responses from 28 participants.
- Diversity: The method achieves better diversity in all tested cases while maintaining better perceptual quality than PIC at the maximum Places2 mask ratio.Diversity is measured with average LPIPS distance and assessed alongside FID to account for meaningless variation.
- Robustness for the completion of extremely large holes: Despite training with masks capped at a 60% maximum ratio, the method generalizes fairly well to extremely large holes and produces high-quality, diversified completions.In this setting, almost all baselines fail with large missing regions.
- Visualization of probability map: Transformer probability maps show increasing uncertainty from mask boundaries toward interiors, with lower uncertainty for face regions than hair and high confidence near a symmetric cheek.Higher confidence corresponds to lower uncertainty.
5. Concluding Remarks
The paper addresses the dilemma between diversity and photorealistic quality by combining transformer structural understanding and pluralism support with CNN texture enhancement and efficiency. Experiments report gains over state-of-the-art fully convolutional approaches, including more diverse and vivid results and generalization to large masks and datasets.
- The method targets the longstanding difficulty of achieving both sufficient diversity and photorealistic quality in image completion.
- It combines transformers’ structural understanding and pluralism support with CNNs’ local texture enhancement and efficiency for high-fidelity free-form pluralistic completion.
- Experiments show large performance gains over state-of-the-art fully convolutional approaches in regular evaluation settings.
- The method produces more diversified and vivid results than the compared fully convolutional approaches.
- The method demonstrates exceptional generalization ability on large-scale masks and datasets.