Large Scale Image Completion via Co-Modulated Generative Adversarial Networks

Shengyu Zhao; Jonathan Cui; Yilun Sheng; Yue Dong; Xiao Liang; Eric I Chang; Yan Xu

Source-linked AI summary

Large Scale Image Completion via Co-Modulated Generative Adversarial Networks

Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong, Xiao Liang, Eric I Chang, Yan Xu

arXiv:2103.10428v1Mar 18, 2021cs.CVcs.GRcs.LG

TL;DR

Large-scale image completion remains difficult, and existing metrics inadequately capture perceptual fidelity. The paper introduces co-modulated GANs, which combine conditional and stochastic styles, and P-IDS/U-IDS for discriminative evaluation; experiments report superior quality and diversity with image-to-image translation generalization.

Problem
Large-scale missing regions remain challenging for image completion, while existing metrics often favor blurry results and user studies are costly and variable.
Method
Co-modulated GANs jointly embed conditional and stochastic style representations, and P-IDS/U-IDS measure perceptual fidelity through linear separability in feature space.
Results
The method demonstrates superior quality and diversity in free-form image completion and easy generalization to image-to-image translation.
Takeaways & Limitations
Co-modulation bridges conditional and unconditional modulated architectures, while P-IDS/U-IDS provide proposed quantitative benchmarks for perceptual fidelity.
Takeaways & Limitations
The model sometimes fails to recognize surrounding semantic information and produces strange artifacts, especially on the challenging Places2 dataset.

Abstract

from arXiv · show

Numerous task-specific variants of conditional generative adversarial networks have been developed for image completion. Yet, a serious limitation remains that all existing algorithms tend to fail when handling large-scale missing regions. To overcome this challenge, we propose a generic new approach that bridges the gap between image-conditional and recent modulated unconditional generative architectures via co-modulation of both conditional and stochastic style representations. Also, due to the lack of good quantitative metrics for image completion, we propose the new Paired/Unpaired Inception Discriminative Score (P-IDS/U-IDS), which robustly measures the perceptual fidelity of inpainted images compared to real images via linear separability in a feature space. Experiments demonstrate superior performance in terms of both quality and diversity over state-of-the-art methods in free-form image completion and easy generalization to image-to-image translation. Code is available at https://github.com/zsyzzsoft/co-mod-gan.

1 INTRODUCTION

Image completion methods struggle with large missing regions, while existing metrics often favor blurry results. The paper introduces co-modulated GANs and P-IDS/U-IDS to address generation and evaluation.

Image-conditional GANs have spawned task-specific approaches because vanilla U-Net-like generators perform poorly, especially for free-form completion.
Co-modulated GANs jointly embed conditional and stochastic style representations, bridging image-conditional and modulated unconditional architectures.
The approach supports diverse, consistent completion from small-scale inpainting to extremely large missing regions, including regular and irregular masks with limited conditional information.
Similarity metrics such as L1, L2, PSNR, and SSIM heavily prefer blurry outputs, while user studies are costly and have high variance.
P-IDS/U-IDS provide scalable assessment of perceptual fidelity and are reported to capture subtle differences and correlate with human preferences.

2 RELATED WORK

Prior work spans image-conditional translation, specialized image completion, and GAN evaluation, but existing approaches leave gaps in stochastic completion and suitable metrics.

Image-conditional GANs support paired and unpaired image-to-image translation, including methods that embed learned style representations from inputs or reference images.
Image completion is a constrained image-to-image translation problem requiring semantically consistent contents beyond the low-level features used by traditional methods.
Recent completion methods specialize semantic context, texture, edges, contours, or architecture to address the task’s challenges.
Inception Score and FCN-Score are task-specialized, while FID is broadly acceptable but few metrics target image completion effectively.
P-IDS/U-IDS formulate paired and unpaired discriminability without relying on classifier generalizability or separate classifier training and testing sets.

3 CO-MODULATED GENERATIVE ADVERSARIAL NETWORKS

The model combines conditional image information with stochastic modulation to retain diversity in weakly conditioned completion. Co-modulation adapts unconditional generative capability through a joint style transformation.

Image-conditional generation maps an input image y and latent vector z to output x, while image completion constrains known pixels to remain unchanged.
Modulated generators derive style vectors from latent representations and use them to modulate intermediate decoder activations.
Conditional modulation derives style from encoded image features but lacks stochastic generative capability, limiting diversity and generalization under weak conditioning.
Co-modulation adapts unconditional modulation to conditional generation by jointly conditioning the style representation on conditional and stochastic inputs.
The paper uses a joint affine transformation of both style representations and reports that this linear correlation supports quality–intra-conditioning-diversity trade-offs without external losses.

4 PAIRED/UNPAIRED INCEPTION DISCRIMINATIVE SCORE

P-IDS/U-IDS evaluate perceptual fidelity through linear separability in a pretrained feature space, addressing weaknesses of existing image-completion metrics. Experiments show faster convergence, greater sensitivity to subtle differences, and stronger correlation with human preferences than FID and KID.

Metric Design: P-IDS/U-IDS fit a linear SVM on pretrained Inception features of real and generated images to measure their separability.P-IDS uses paired real/fake samples, while U-IDS supports unpaired distributions through SVM misclassification rate.
Advantages and Cost: P-IDS/U-IDS offer robustness to sampling size, sensitivity to subtle differences, and correlation with human preferences as stated advantages over FID.The paper also reports mild computational overhead beyond feature extraction for the SVM-based metrics.
Robustness to Sampling Size: P-IDS/U-IDS converge within a small number of samples, whereas FID fails to converge within 10k samples when distinguishing masking manipulations.The comparison uses FFHQ images at 512×512 resolution and a 50k-sample reference distribution for FID.
Effectiveness of Capturing Subtle Differences: P-IDS reflects the number of manipulated pixels, while FID and KID fail to respond within 29 noisy pixels.This test evaluates subtle partial-region differences using 10k samples.
Correlation to Human Preferences: 0.870 is the correlation coefficient between P-IDS and human preference rate, compared with −0.765 for FID.The correlation analysis uses 20 data points from the user study.

5 EXPERIMENTS

Experiments evaluate co-modulated GANs for 512×512 image completion and 256×256 image-to-image translation, including quality, diversity, stochasticity, and user-facing fidelity. The method supports large-scale and free-form completion while generalizing to edges-to-photos and labels-to-photos settings.

Image Completion: Experiments use 512×512 FFHQ and Places2 image-completion datasets, including a 10k-image FFHQ validation split and Places2’s 36.5k-image validation set.Training uses 25M images on FFHQ and 50M images on Places2.
Image Completion: Co-modulated GANs produce diverse outputs with fixed inputs and masks, while truncation ψ enables a quality–diversity trade-off.The stochastic style branch is learned without external losses and is explicitly amplified by tuning ψ.
Image Completion: Qualitative comparisons assess PatchMatch, DeepFillv2, and the proposed method on large-scale completion using FFHQ and Places2 images.The original images are sampled at 512×512 resolution.
Image Completion: Co-modulation dominates conditional modulation, especially as the masked ratio increases, while the vanilla version completely fails in the ablation study.The comparison is conducted on FFHQ under the same image-completion setting.
Image-to-Image Translation: Co-modulated GANs extend to 256×256 image-to-image translation, with FID measuring fidelity and LPIPS measuring intra-conditioning diversity on edges-to-photos datasets.FID samples once per validation image, while LPIPS uses 2k pairs.
Image-to-Image Translation: A reported limitation in edges-to-photos is that high fidelity may result from strong learned correspondence between edge maps and color information from the limited training set.The authors state this as a hypothesis explaining the observed behavior.
Image-to-Image Translation: On COCO-Stuff labels-to-photos, the method matches SPADE’s FID while significantly outperforming it on P-IDS and U-IDS without direct supervision.The evaluation uses 5k validation outputs.

6 CONCLUSION

The conclusion presents co-modulated GANs as a bridge between conditional and unconditional modulated architectures for large-scale free-form completion and image-to-image translation. It also introduces P-IDS/U-IDS for perceptual-fidelity assessment.

Co-modulated GANs bridge conditional and unconditional modulated architectures while improving free-form large-scale image completion and generalizing to image-to-image translation.
P-IDS/U-IDS is proposed as an intuitive metric for robustly assessing GAN perceptual fidelity and providing quantitative benchmarks.

APPENDIX A IMPLEMENTATION DETAILS

The implementation largely follows StyleGAN2, adding a conditional encoder and stochasticity-related dropout while training a 109M-parameter model with procedurally generated free-form masks.

Network Architecture: The network borrows StyleGAN2 details including two convolutions per level, resolution-dependent channels, an 8-layer mapping MLP, noise injection, style mixing, logistic loss, and R1 regularization.Style mixing uses a probability of 0.5 instead.
Conditional Encoder: The conditional encoder produces a conditional style representation by flattening its final 4×4 feature map into a 1024-channel fully connected layer with dropout.Dropout remains enabled during testing because it partially correlates with inherent stochasticity.
Training: The model contains 109M parameters and trains on eight NVIDIA Tesla V100 GPUs with a total batch size of 32 for 25M images unless specified otherwise.At 512×512 resolution, training takes about one week.
Mask Generation: Free-form training masks combine random brush strokes and rectangles with uniformly sampled geometric properties.Brush widths range from 12 to 48, vertices from 4 to 18, and stroke counts from 0 to 20.

APPENDIX C USER STUDY

The user studies compare generated and real images or competing methods through timed forced-choice judgments, with “don’t know” responses scored as 0.5. Results include 14,336 image-completion judgments and a COCO-Stuff preference comparison favoring the proposed method.

Image Completion Study: The image-completion user study samples 256 validation images, masks, and outputs for each dataset and masked-ratio range.
COCO-Stuff Study: 319 participants preferred the proposed method, 189 preferred SPADE, and 212 selected “don’t know” in the COCO-Stuff comparison.

APPENDIX D MORE QUANTITATIVE RESULTS

Quantitative experiments compare image-completion methods across masked ratios, with the proposed method achieving superior performance at all evaluated ratios.

The proposed method achieves superior image-completion performance across all masked ratios.Table 5 reports comparisons across methods and masked ratios.

APPENDIX E MORE QUALITATIVE RESULTS

Qualitative results cover image-to-image translation and free-form image completion, including a fidelity–diversity trade-off controlled by ψ in Edges2Handbags.

On Edges2Handbags, ψ = 1 achieves superior fidelity while ψ = 3 achieves superior diversity.Additional qualitative comparisons are shown for COCO-Stuff and free-form image completion.
Extensive qualitative examples are provided for free-form image completion.The examples appear in Figs. 18-23.

APPENDIX F DISCUSSION

The discussion presents qualitative completion and translation examples while noting that large-scale completion can still produce semantic artifacts, especially on challenging Places2 scenes.

Discussion and limitations: Large-scale completion sometimes produces strange artifacts when the model fails to recognize semantic information in surrounding areas.The issue is especially noted for the challenging Places2 dataset containing diverse scenes, styles, and image quality.
Discussion and limitations: User-study and quantitative comparisons evaluate image completion against PatchMatch and DeepFillv2 across different masked ratios.The quantitative comparison reports results averaged over five runs.
Discussion and limitations: Ablation results show co-modulation dominating conditional modulation at all masked ratios, especially as the masked ratio increases.The comparison is performed on FFHQ over five runs; Vanilla completely fails.
Qualitative results: Examples also demonstrate image-to-image translation from labels to photos on COCO-Stuff.The comparison is against SPADE.
Qualitative results: The method bridges small-scale inpainting and large-scale completion across regular and irregular masks.Examples span differently conditioned masks on FFHQ and Places2.
Qualitative results: At 1024×1024 resolution, qualitative image-completion examples are shown using FFHQ validation images.The figures use images sampled from a 10k validation split.
Qualitative results: Qualitative comparisons include PatchMatch, DeepFillv2, and the proposed method on FFHQ and Places2.The comparisons use 512×512 validation images.