PaperlayerView exact revision on arXiv

Source-linked AI summary

Pluralistic Image Completion

Chuanxia Zheng, Tat-Jen Cham, Jianfei Cai

arXiv:1903.04227v2cs.CV

TL;DR

Image completion must accommodate multiple plausible outcomes, but single-instance conditional training can suppress diversity. The paper addresses this with linked reconstructive and generative paths and short+long-term attention, reporting diverse, high-quality completions across datasets.

  • Problem

    Most image-completion methods generate one result, while partial-image conditions usually have only one ground-truth instance, limiting meaningful variation.

  • Method

    The framework couples a VAE-based reconstructive path with a generative path that predicts missing-region priors, supported by GANs and short+long-term attention.

  • Results

    The method generated higher-quality completion results with multiple diverse plausible solutions across multiple datasets.

  • Takeaways & Limitations

    Pluralistic image completion can produce varied, plausible outputs for a single masked input, including for large holes.

  • Takeaways & Limitations

    Single-path conditional VAEs remain vulnerable to negligible diversity when each partial-image condition has only one training instance.

Abstract

from arXiv · show

Most image completion methods produce only one result for each masked input, although there may be many reasonable possibilities. In this paper, we present an approach for \textbf{pluralistic image completion} -- the task of generating multiple and diverse plausible solutions for image completion. A major challenge faced by learning-based approaches is that usually only one ground truth training instance per label. As such, sampling from conditional VAEs still leads to minimal diversity. To overcome this, we propose a novel and probabilistically principled framework with two parallel paths. One is a reconstructive path that utilizes the only one given ground truth to get prior distribution of missing parts and rebuild the original image from this distribution. The other is a generative path for which the conditional prior is coupled to the distribution obtained in the reconstructive path. Both are supported by GANs. We also introduce a new short+long term attention layer that exploits distant relations among decoder and encoder features, improving appearance consistency. When tested on datasets with buildings (Paris), faces (CelebA-HQ), and natural images (ImageNet), our method not only generated higher-quality completion results, but also with multiple and diverse plausible outputs.

1. Introduction

Image completion is subjective because multiple detailed fillings can share high-level semantic agreement. The paper introduces pluralistic image completion, using linked reconstructive and generative paths plus short+long-term attention to produce diverse, plausible, consistent results.

  • Motivation: Experts may agree on high-level semantics while independently creating substantially different details for the same damaged image.Completion requires semantic inference, structural continuity, and visually realistic filling.
  • Problem: Pluralistic image completion generates multiple diverse plausible results for one masked image instead of a single guess.The task addresses the one-to-many nature of completion.
  • Problem: Conditional VAEs often underestimate variation because each partial-image condition typically has only one matching training instance.Sampling therefore remains close to the single ground-truth reconstruction.
  • Method: The proposed framework links a VAE-based reconstructive path using full ground truth with a generative path that predicts and samples missing-region priors from visible pixels.The generative path is not trained to reconstruct instance-specific hidden pixels and uses an auxiliary discriminator to assess reasonableness.
  • Results: Experiments report higher-quality completion results alongside multiple, substantially diverse plausible solutions.The method was evaluated across multiple datasets and demonstrated varied completions for the same mask.
  • Method: A short+long-term self-attention layer uses distant context and encoder-decoder feature relations to improve appearance consistency.The layer lets the network attend to finer encoder features or more semantically generative decoder features.

2. Related Work

Prior image-completion methods primarily copy or realign visible-region content, which works for background completion but struggles with new content and distant context.

  • Intra-Image Completion: Traditional intra-image methods assume holes share similar content with visible regions and match, copy, or realign background patches.These approaches include diffusion-based and patch-based completion methods.
  • Intra-Image Completion: These methods perform well for background completion such as object removal but do not use distant information effectively for new content differing from visible regions.The paper motivates extending self-attention to harness abundant context.

3. Approach

The approach models pluralistic completion as sampling hidden regions from a conditional distribution, using linked reconstructive and generative paths plus short+long term attention. The reconstructive path uses full images during training, while the generative path samples from visible regions at test time.

  • Probabilistic Framework: Classical deterministic completion produces one solution, whereas the proposed framework samples hidden content from p(Ic|Im).Im denotes visible pixels and Ic the ground-truth hidden pixels.
  • Probabilistic Framework: Direct CVAE training can collapse its conditional prior toward a delta-like distribution, yielding negligible diversity when each partial image has one training instance.A fixed latent prior can likewise be ignored, causing the network to estimate Ic directly from Im.
  • Probabilistic Framework: The dual-pipeline framework couples a VAE-based reconstructive path using full-image information with a generative path using visible regions to infer a sampleable conditional distribution.The two paths share representation and generation-network weights.
  • Training Paths: During training, the reconstructive path samples from Ic and reconstructs the original image, while the generative path samples from Im and reconstructs only visible regions.Adversarial networks support both paths and encourage synthesized images to fit the training-set distribution.
  • Training Loss: The objective combines KL consistency, appearance fidelity, and adversarial losses for both reconstructive and generative paths.The KL terms regularize distribution consistency, appearance terms encourage matching fidelity, and adversarial terms enforce training-distribution fit.
  • Short+Long Term Attention: Short+long term attention uses decoder self-attention and encoder-decoder contextual flow, allowing attention to finer encoder features or more generative decoder features.This extends self-attention beyond a decoder layer to feature-feature context between encoder and decoder layers.

4. Experimental Results

PICNet is evaluated across four datasets with quantitative and qualitative comparisons, testing image quality, diversity, training strategies, and attention behavior. It produces multiple plausible completions, with especially strong qualitative results for large holes.

  • Evaluation setup: PICNet was evaluated on Paris, CelebA-HQ, Places2, and ImageNet with regular and irregular holes, sampling 50 outputs per masked image.The top 10 samples were selected using discriminator scores.
  • Quantitative comparisons: Quantitative evaluation used ImageNet’s 20,000 test images with mean ℓ1 loss, PSNR, TV, and IS under a 128 × 128 center mask.The comparison selected the best-balanced sample among the top 10 discriminator-ranked outputs.
  • Qualitative comparisons: On Paris, PICNet produced more natural images than PatchMatch, Context Encoder, and Shift-Net while also generating variations in window counts and door sizes.PatchMatch benefited from repetitive structures, whereas Context Encoder produced blurry textures.
  • Qualitative comparisons: On CelebA-HQ, PICNet generated various plausible results by sampling the conditional prior, unlike Contextual Attention’s single output per masked input.The comparison used large regular holes to highlight output diversity.
  • Qualitative comparisons: On ImageNet, PICNet inferred content effectively across varied objects while being compared with PatchMatch, Context Encoder, Global and Local, and Contextual Attention.PICNet was trained on the original ImageNet training set, whereas some baselines used a 100k-image subset.
  • Ablation study: CVAE and instance-blind variants showed limited variation, while BicycleGAN produced poor results or minimal variation for center masks.The CVAE conditional prior became narrowly centered, and full-image reconstruction in the instance-blind variant encouraged deterministic outputs.
  • Attention analysis: PICNet’s attention uses decoder and encoder features, enabling broader contextual use than directly copying features from one visible position.For faces with large masks, Contextual Attention struggled to borrow hidden facial content, while PICNet’s attention used both feature sources as appropriate.

5. Conclusion

The paper concludes that its dual-pipeline architecture generates diverse, plausible completions for single masked inputs. Across datasets and hole types, the method achieves high-quality outputs, particularly for large holes, and supports additional applications such as face editing.

  • Conclusion: The dual-pipeline architecture generates multiple diverse solutions with plausible content for a single masked input.The conclusion identifies prior-conditional lower-bound coupling as significant for conditional image generation.
  • Conclusion: The enhanced short+long term attention layer improves realism by incorporating broader feature relationships.The conclusion presents this attention mechanism as a separate contribution alongside the dual pipeline.
  • Conclusion: Experiments across varied datasets show that PICNet’s multiple solutions are diverse and high-quality, especially for large holes.Additional results cover Paris, CelebA-HQ, Places2, and ImageNet variation sets.
  • Applications: The paper also presents results for random and irregular holes and an interactive face-editing application.The supplemental figures include object removal and face editing examples.

A.2. Additional Results on Random and Irregular Hole Completion

Additional results show that PICNet produces varied completions for random and irregular facial masks while respecting structural constraints such as facial symmetry. The results also demonstrate face editing and object removal in an interactive setting.

  • Random and irregular holes: Facial symmetry strongly constrains completions when one eye or part of the mouth remains visible.The unmasked facial feature guides the corresponding missing feature, reducing variation relative to masking both sides.
  • Random and irregular holes: When both eyes or the mouth are masked, sampled completions show substantial variation in those facial features.The variation includes different plausible shapes and configurations across samples.
  • Face editing: PICNet supports object removal and face editing through its online interactive demo.White mask regions are normalized to gray masks as input.

A.4. Video for Additional Results

The supplemental material includes two videos showing additional pluralistic completion results, comparisons, free-form masks, and face editing.

  • Video supplements: The first supplemental video shows free-form mask results across various datasets.It extends the presented completion examples beyond fixed center masks.
  • Video supplements: The second video covers center-hole and random-hole completion, training-strategy comparisons, and face editing.It includes multiple examples across these application and evaluation settings.

B. Mathematical Derivation and Analysis

This appendix section elaborates on the difficulties encountered when applying the classical CVAE formulation to pluralistic image completion.

  • The section expands the shorter discussion of classical CVAE difficulties presented in section 3.1.
  • Its focus is the classical CVAE formulation for pluralistic image completion.
  • The discussion provides additional mathematical analysis of the formulation’s difficulties.

B.1.1 Background: Derivation of the Conditional Variational Auto-Encoder (CVAE)

The CVAE models hidden content with a latent variable conditioned on the visible image, using variational inference to optimize likelihood.

  • A latent variable zc stochastically generates the hidden partial image Ic conditioned on the visible partial image Im.
  • The conditional likelihood is estimated with importance sampling, whose Monte Carlo variance can be reduced by an appropriate sampling choice.
  • Taking logarithms and applying Jensen’s inequality produces a variational lower bound that is maximized over θ, φ, and ψ.
  • With only one Ic for each unique Im, the posterior can become effectively dependent only on Im, driving the KL divergence toward zero.
  • Under this zero-KL condition, the posterior can be replaced by the conditional prior without changing the variational lower bound.

B.1.3 Unconstrained Learning of the Conditional Prior

The analysis shows why unconstrained conditional-prior learning collapses toward a narrow solution, while the proposed framework imposes shared latent structure and coupled objectives to preserve diversity.

  • Applying Jensen’s and Hölder’s inequalities analyzes how the variational bound can be maximized.
  • With a unique global maximum, equality is achieved when the conditional prior becomes a Dirac delta at the maximum-likelihood latent point.
  • Without additional constraints, the conditional prior therefore learns a narrow delta-like distribution selecting a maximum latent-likelihood value.
  • This narrow prior may estimate one completion effectively but is poor for sampling diverse solutions; unconditional partial-image priors prevent the degeneracy.
  • A fixed-prior CVAE can be interpreted as selecting among condition-specific VAE models using Im as a switch parameter.
  • With one Ic per Im, the fixed-prior model can ignore sampling noise, producing negligible output variation despite sampling zc from N(0, 1).
  • The framework instead shares a common latent space for unconditional partial images, uses adaptive priors, and couples the conditional generative path to this distribution.
  • Its objective jointly links unconditional and conditional variational lower bounds, with reconstructive and generative paths using distinct likelihood and GAN-based terms.

C. Architectural Details

PICNet adapts residual-based generative architectures for image-conditional completion and adds normalization, attention, and separate inference networks for its two paths.

  • PICNet is inspired by SA-GAN and BigGAN but modifies normalization and network components for image-conditional generation.
  • The architecture uses residual blocks across encoder, discriminator, inference, and decoder networks, with distinct block roles.
  • The design includes a short+long attention layer and architectural specifications covering encoder features, output activations, and channel width.
  • Infer1 models the ground-truth hidden image’s latent distribution, while Infer2 predicts that distribution from visible pixels for generation.

D. Experimental Details

The experiments specify the network configuration, loss-weight choices, adversarial update schedule, and substantial training-time differences across datasets and mask types.

  • Implementation: The implementation uses PyTorch v0.4.0, channel width 4·ch with ch = 32, and attention layers at 32×32 feature resolution.The discriminator uses self-attention, while the generator uses short+long term attention; spectral normalization is applied throughout the network.
  • Loss weights: Increasing the KL-loss weight increased generated diversity but made appearance consistency harder to retain.The reported final weights were αKL = αapp = 20 and αad = 1.
  • Adversarial updates: One discriminator step per generator step produced the best results in experiments varying the update ratio from one to five.With αapp below 1, two or four discriminator steps per generator step could be used, whereas larger αapp values required more generator steps.
  • Training cost: Training centered-hole models took roughly 3 days for Paris and CelebA-HQ and roughly 2 weeks for ImageNet and Places2.Random irregular and un-centered holes required about twice the training time of centered holes.
Loading 1903.04227v2…