PaperlayerView exact revision on arXiv

Source-linked AI summary

Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAE

Jialun Peng, Dong Liu, Songcen Xu, Houqiang Li

arXiv:2103.10022v1cs.CV

TL;DR

Multiple-solution inpainting must represent several plausible completions while maintaining the quality of each result. The paper uses hierarchical VQ-VAE to generate diverse structures autoregressively, then refines each with structure-guided texture synthesis; experiments report improved quality and diversity across three datasets.

  • Problem

    Existing multiple-solution inpainting methods have difficulty maintaining quality, producing distorted structures and/or blurry textures.

  • Method

    A two-stage hierarchical VQ-VAE model samples diverse discrete structures with a conditional autoregressive generator and synthesizes texture for each structure using structural attention.

  • Results

    Experiments on CelebA-HQ, Places2, and ImageNet demonstrate superiority in both inpainting quality and diversity.

  • Takeaways & Limitations

    Hierarchical VQ-VAE structural information benefits multiple-solution image inpainting by supporting diverse structures and structure-consistent textures.

  • Takeaways & Limitations

    Typical learning-based inpainting methods focus on reconstructing ground truth rather than creating plausible alternative results.

Abstract

from arXiv · show

Given an incomplete image without additional constraint, image inpainting natively allows for multiple solutions as long as they appear plausible. Recently, multiplesolution inpainting methods have been proposed and shown the potential of generating diverse results. However, these methods have difficulty in ensuring the quality of each solution, e.g. they produce distorted structure and/or blurry texture. We propose a two-stage model for diverse inpainting, where the first stage generates multiple coarse results each of which has a different structure, and the second stage refines each coarse result separately by augmenting texture. The proposed model is inspired by the hierarchical vector quantized variational auto-encoder (VQ-VAE), whose hierarchical architecture isentangles structural and textural information. In addition, the vector quantization in VQVAE enables autoregressive modeling of the discrete distribution over the structural information. Sampling from the distribution can easily generate diverse and high-quality structures, making up the first stage of our model. In the second stage, we propose a structural attention module inside the texture generation network, where the module utilizes the structural information to capture distant correlations. We further reuse the VQ-VAE to calculate two feature losses, which help improve structure coherence and texture realism, respectively. Experimental results on CelebA-HQ, Places2, and ImageNet datasets show that our method not only enhances the diversity of the inpainting solutions but also improves the visual quality of the generated multiple images. Code and models are available at: https://github.com/USTC-JialunPeng/Diverse-Structure-Inpainting.

1. Introduction

The paper motivates multiple-solution inpainting because incomplete images can admit several plausible completions, then proposes a hierarchical VQ-VAE-based two-stage model that generates diverse structures and refines their textures.

  • Prior Work: Existing inpainting methods commonly use encoder-decoder networks with reconstruction and adversarial losses, contextual attention, or modified convolutions.These approaches aim to improve visual quality and reduce artifacts using available image content.
  • Motivation: Multiple plausible completions can exist for one incomplete image, particularly when the missing region is large and complex.Providing alternatives may let users select results according to their preferences.
  • Design Rationale: The proposed model uses hierarchical VQ-VAE because vector quantization provides discrete latents and its hierarchy can separate structural from textural information.Discrete latents support autoregressive distribution modeling, while the hierarchical layout separates global and local image information.
  • Proposed Method: The first stage samples a conditional autoregressive distribution to produce multiple structural features, while the second stage generates a complete image for each structure.Each generated structural-feature set leads to a separate completion.
  • Contributions: The method combines a hierarchical VQ-VAE, a diverse structure generator, and a texture generator in a two-stage pipeline.The paper identifies these components as its main contributions and reports evaluation on CelebA-HQ, Places2, and ImageNet.

2. Related Work

Prior inpainting methods either copy source patches or predict a single completion, while VAE-based multiple-solution methods struggle to separate structural and textural information.

  • Traditional Methods: Traditional diffusion-based and patch-based methods borrow source-image patches and cannot generate unique content absent from those sources.They may also produce unreasonable results without high-level semantic guidance.
  • Learning-Based Methods: Learning-based methods use deep convolutional networks to semantically predict missing regions, but typically generate only one optimal result per input.Their training focuses on reconstructing ground truth rather than producing multiple plausible completions.
  • Multiple-Solution Methods: VAE-based multiple-solution methods sample continuous Gaussian latent variables to obtain diverse inpainted images.These methods use combinations of KL-divergence and adversarial losses, but do not effectively separate structural and textural information.
  • VQ-VAE: Hierarchical VQ-VAE separates structural and textural information with discrete latent variables that can be modeled autoregressively without posterior collapse.However, class-conditioned PixelCNNs and their lossy textures are not practical for image inpainting.

3. Method

The method uses hierarchical VQ-VAE to separate discrete structural and textural information, then generates diverse structures autoregressively and synthesizes textures conditioned on them. Structural attention and VQ-VAE-based feature losses further promote consistency and image quality.

  • 3.1. Hierarchical VQ-VAE: Hierarchical VQ-VAE encodes images into discrete structural and textural features, which its decoder recombines to reconstruct images.Each codebook contains 512 prototype vectors of dimensionality 64; for 256×256 images, structural and textural features have sizes 32×32 and 64×64, respectively.
  • 3.2. Diverse Structure Generator: The diverse structure generator models the conditional distribution of discrete structural features with an autoregressive network and samples multiple structures during inference.The incomplete image and binary mask condition the distribution pθ(¯sgt|Iin, M), while sequential sampling produces diverse ¯sgen.
  • 3.2. Diverse Structure Generator: Low-resolution structural features help the generator capture global information and avoid GAN-specific mode collapse through maximum-likelihood training over all training samples.The stated objective maximizes the likelihood of all training samples without an additional loss.
  • 3.3. Texture Generator: The texture generator synthesizes a completion from generated structural features, using gated and dilated convolutions plus structural attention for long-range structural correlations.Structural features guide both early generator layers and the attention module; during inference, generated features ¯sgen produce the inpainting result.
  • 3.3. Texture Generator: The structural attention module computes attention directly on structural features to improve consistency between synthesized texture and generated structure.It extracts 3×3 structural patches and computes truncated distance similarity between patches before transferring attention.
  • 3.3. Texture Generator: A pre-trained hierarchical encoder evaluates generated completions with structural and textural feature losses intended to improve structural coherence and texture realism.The encoder maps the completion to structural and textural features, which are compared with the corresponding quantized ground-truth features.

4. Experiments

Experiments on CelebA-HQ, Places2, and ImageNet evaluate visual quality, diversity, and component effects. The method produces more reasonable structures, realistic textures, and competitive diversity than compared multiple-solution methods.

  • Datasets and Protocol: The evaluation uses CelebA-HQ, Places2, and ImageNet with regular or irregular missing regions.Images are processed at 256×256 resolution using the datasets’ original splits, down-sampling, cropping, and augmentation procedures.
  • Qualitative Comparisons: The method produces more reasonable structures and realistic textures than compared methods in qualitative CelebA-HQ and Places2 results.Examples include fine-grained hair and eyebrows, different eye colors, and varying window sizes.
  • Quantitative Comparisons: Our method outperforms PIC and UCTGAN on PSNR and SSIM, and outperforms all other methods on IS, MIS, and FID.The comparison uses all samples from 1,000 CelebA-HQ testing images with 128×128 center holes.
  • Quantitative Comparisons: 0.029 LPIPS: our method achieves diversity comparable to PIC at 0.029 and UCTGAN at 0.030.Scores average consecutive pairs among 50K results sampled from 1,000 incomplete images; higher scores indicate higher diversity.
  • Ablation: Structural Attention: Full attention on structural features synthesizes realistic textures, including symmetric eyes and eyebrows, while improving structure-texture consistency.The quantitative results in Table 2 also demonstrate benefits from the structural attention module.
  • Ablation: Auxiliary Losses: The structural feature loss improves structure coherence, while the textural feature loss improves texture realism.Using both losses generates more natural images; the authors report quantitative benefits in Table 3.

5. Conclusion

The paper presents a hierarchical VQ-VAE-based method that generates diverse structures and then synthesizes consistent textures for each structure. Qualitative and quantitative comparisons report improvements in both image quality and diversity.

  • Conclusion: The method uses an autoregressive distribution to generate diverse structures, then synthesizes image texture for each structure.A structural attention module aligns synthesized texture with generated structure, and two feature losses target structure coherence and texture realism.
  • Conclusion: Extensive qualitative and quantitative comparisons show superiority in both quality and diversity.The paper also reports that structural information extracted by hierarchical VQ-VAE benefits image inpainting.

A. Architecture Hyperparameters and Training Details.

The appendix reports implementation settings and training organization for the hierarchical VQ-VAE, diverse structure generator, and texture generator.

  • Training Details: The model is implemented in TensorFlow v1.12 with batch size 8 and trained using NVIDIA 2080 Ti GPUs.The hierarchical VQ-VAE and texture generator can be trained on one GPU, while the diverse structure generator and texture generator can be trained in parallel after VQ-VAE training.

B. Negative Log Likelihood and Reconstruction Error.

The appendix characterizes the hierarchical VQ-VAE using reconstruction distortion and the diverse structure generator using global-latent coding rate.

  • Evaluation Measures: The hierarchical VQ-VAE minimizes MSE reconstruction error, while the diverse structure generator minimizes global-latent NLL.The reported quantities are MSE distortion and global-latent NLL, interpreted as an estimate of coding rate.

C. Inference Time.

Our model has substantially slower inference than GAN- and VAE-based methods because naive autoregressive sampling dominates runtime.

  • 45 seconds per 256×256 image is measured for our model on a single NVIDIA 1080 Ti GPU.
  • FE runs at 0.2 second per image under the same resolution and GPU setting.
  • Incremental sampling with cached intermediate states can reduce the autoregressive runtime by an order of magnitude.The authors identify naive autoregressive sampling as the major source of computational time and may integrate this technique in the future.

D. More Visual Examples.

Additional visual examples evaluate the method on center-mask and random-mask settings across CelebA-HQ, Places2, and ImageNet.

  • Center-mask models are illustrated on CelebA-HQ, Places2, and ImageNet test or validation sets.The examples appear in Figures 7, 8, and 9.
  • Random-mask models are also evaluated across the CelebA-HQ, Places2, and ImageNet test or validation sets.Additional random-mask results are shown in Figure 10.
  • The training masks comprise 128×128 center holes or randomly positioned rectangles and brush strokes with random sizes.The center masks are applied to 256×256 images, while random masks follow the type used in prior work [41].

E. Discussions on Artifacts.

Residual artifacts mainly arise from low-quality generated structures, while the texture generator also contributes artifacts.

  • Most artifacts are attributed to the low quality of generated structures.
  • A lightweight autoregressive network is used in the structure generator for computational efficiency instead of the much more complex network in [22].
  • The authors anticipate improved results from using the more complex structure-generation network.
  • The texture generator also produces artifacts that may be addressed with feature, multi-scale, and generator-discriminator techniques.The cited examples are feature discriminator [16], multi-scale discriminator [23], and multi-scale generator [15].

F. Discussions on Diversity.

The method’s diversity depends on the learned structural distribution, training data, incomplete-image condition, mask type, and model design. The authors also identify a possible diversity–quality tradeoff.

  • Diversity determinants: Diversity is fully determined by the learned conditional distribution because texture generation has no randomness.Pixel-wise entropy is used to analyze diversity, and higher entropy is associated with higher diversity.
  • Diversity determinants: More diverse training data enrich the learned structural distribution, with less diversity observed for the face dataset than for natural-image datasets.The passage contrasts roughly 10^4 face training images with roughly 10^7 natural-image training images and refers to Figures 7 and 8.
  • Diversity determinants: Available content and the location and size of the missing region substantially influence diversity.Figures 12 and 13 show the effect of available content, while Figure 14 shows the effect of hole location and size.
  • Diversity determinants: Models trained with random masks seem to produce higher diversity than center-mask models for the same incomplete image.The comparison is made between Figure 7 Row 3 and Figure 14 Row 1, and the authors conjecture that more random masks may help increase diversity.
  • Diversity determinants: Diversity may be increased by enlarging conditional-distribution support, using a more sophisticated distribution model, or adding entropy-oriented regularization.The passage gives codebook size and entropy-increasing loss terms as examples.
  • Diversity–quality tradeoff: Pursuing higher diversity may deteriorate quality because intentionally biasing the learned distribution can create a diversity–quality tradeoff.
Loading 2103.10022v1…