PaperlayerView exact revision on arXiv

Source-linked AI summary

MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generation

Chuanxia Zheng, Long Tung Vuong, Jianfei Cai, Dinh Phung

arXiv:2209.09002v1cs.CV

TL;DR

VQ generative models can map similar patches to the same index, producing repeated artifacts, while conventional autoregressive priors are slow. MoVQ modulates quantized vectors spatially, adds multichannel quantization, and uses MaskGIT for prior modeling; experiments report improved image quality and high-fidelity, diverse generation. A remaining limitation is that some generated images have high-frequency appearance without sufficient structure.

  • Problem

    VQ quantization can assign similar patches the same index, causing repeated artifacts in adjacent regions, and autoregressive second-stage generation is time-consuming.

  • Method

    MoVQ adds spatially conditional normalization and multichannel quantization to the VQGAN quantizer, then uses MaskGIT to model priors over compressed discrete representations.

  • Results

    MoVQ significantly outperforms state-of-the-art VQ models on image modeling without increasing computational cost, with better unconditional and class-conditional sample fidelity.

  • Takeaways & Limitations

    Spatial modulation and multichannel representations improve the quality and diversity of VQ-based image reconstruction and generation while retaining simple baseline architectures.

  • Takeaways & Limitations

    Some generated images have a high-frequency appearance while missing structural information, motivating better models for multichannel indexes.

Abstract

from arXiv · show

Although two-stage Vector Quantized (VQ) generative models allow for synthesizing high-fidelity and high-resolution images, their quantization operator encodes similar patches within an image into the same index, resulting in a repeated artifact for similar adjacent regions using existing decoder architectures. To address this issue, we propose to incorporate the spatially conditional normalization to modulate the quantized vectors so as to insert spatially variant information to the embedded index maps, encouraging the decoder to generate more photorealistic images. Moreover, we use multichannel quantization to increase the recombination capability of the discrete codes without increasing the cost of model and codebook. Additionally, to generate discrete tokens at the second stage, we adopt a Masked Generative Image Transformer (MaskGIT) to learn an underlying prior distribution in the compressed latent space, which is much faster than the conventional autoregressive model. Experiments on two benchmark datasets demonstrate that our proposed modulated VQGAN is able to greatly improve the reconstructed image quality as well as provide high-fidelity image generation.

1 Introduction

VQ-based models provide compact discrete representations but can produce repeated artifacts and slow second-stage generation. MoVQ addresses these issues with spatial modulation, multichannel quantization, and MaskGIT-based prior modeling.

  • Motivation: VQ-VAE learns compressed discrete image representations in two stages: quantization first, followed by prior modeling in the discrete latent space.Its likelihood-based training provides stable optimization and preserves spatial structure in higher-resolution latent codes.
  • Motivation: Autoregressive second-stage generation is time-consuming because it predicts discrete latent codes one spatial location at a time.MaskGIT reduces inference steps by predicting multiple tokens according to confidence at each step.
  • MoVQ: VQ methods can repeat artifact patterns because similar patches are mapped to the same quantization index and existing decoders lack spatial variation.MoVQ targets this stage-1 limitation rather than changing the overall two-stage recipe.
  • MoVQ: MoVQ uses spatially conditional normalization to add location-dependent information, multichannel representation to improve code recombination, and MaskGIT for second-stage prior learning.The contributions retain the encoder-decoder layers of VQGAN while targeting reconstruction quality, representation capability, and generation speed.
  • Results: Experiments on two benchmark datasets show high-quality, diverse synthesized samples from the proposed approach.The reported evaluation covers unconditional and class-conditional image generation settings.

2 Related Work

MoVQ extends VQ-based image generation by modulating embedded discrete features and using multichannel quantization. Its two-stage design combines a modified decoder with a bidirectional transformer prior.

  • VQ-based image synthesis: VQGAN uses a two-stage pipeline that encodes images into compact discrete tokens and then models their distribution in latent space.The approach inherits VQ-VAE’s discrete codebook representation and prior-learning structure.
  • Spatially conditional normalization: Spatially conditional normalization in MoVQ uses embedded discrete features as the conditioning input rather than style images, domain images, semantic maps, learned vectors, or heatmaps.This distinguishes the paper’s conditioning signal from variants described in related image-synthesis work.
  • MoVQ: MoVQ introduces spatial modulation of quantized vectors and multichannel quantization within the VQ-based image-generation framework.The paper presents these as its central distinctions from concurrent VQ-based models.
  • MoVQ: The proposed architecture combines a spatially modulated quantizer with a bidirectional transformer that estimates priors over multichannel discrete representations.The left side modifies decoder processing, while the right side models masked latent tokens.

3 Method

MoVQ improves the first-stage VQ image synthesis pipeline by modulating quantized vectors with spatially conditional normalization and using multichannel representations, while retaining flexible second-stage prior modeling with autoregressive or MaskGIT generators.

  • Pipeline: The two-stage pipeline first learns a codebook-based quantizer and decoder, then models the discrete latent distribution for image generation.Images are embedded into compact discrete sequences before a prior network learns their distribution.
  • Modulating Quantized Vectors: Spatially conditional normalization adds spatial variants to discrete feature maps, addressing repeated artifacts caused by similar patches receiving identical quantization indices.The decoder normalizes activations and modulates them using learned scale and bias derived from quantized vectors.
  • Multichannel Representation: Multichannel quantization represents images with parallel index-map channels sharing one codebook, increasing code recombination capacity without proportionally increasing model or codebook cost.For 256×256 images, the default representation uses 16 × 16 × 4 features and reduces each codevector dimension to nq = nz/c.
  • Modeling Prior Distribution: The second stage supports conventional autoregressive modeling, which samples tokens sequentially but slowly, and MaskGIT, which predicts masked tokens in parallel.MaskGIT uses a bidirectional transformer and iteratively reveals high-confidence tokens.
  • Modeling Prior Distribution: MoVQ extends masked token generation to multichannel sequences by predicting c indices per position while independently masking indices across channels and positions.Each multichannel position is assembled into one input token, while the output predicts its component indices.

4 Experiments

Experiments evaluate MoVQ on 256 × 256 images for reconstruction and image generation, including ablations of architecture, codebook size, and latent size. MoVQ improves reconstruction quality and produces photorealistic, diverse generated images while using a compact representation.

  • Image quantization: MoVQ outperforms state-of-the-art methods for image reconstruction at the same 192x compression ratio.The comparison includes ViT-VQGAN and RQ-VAE, despite MoVQ using substantially fewer parameters than ViT-VQGAN.
  • Codebook size: MoVQ’s 1024-codevector, 64-dimensional codebook outperforms methods using larger codebooks, indicating that greedily increasing codebook size is unnecessary.The authors also report improved codebook usage compared with current VQ models.
  • Qualitative reconstruction: MoVQ removes repeated artifacts visible in VQGAN and produces more realistic details than VQGAN and RQ-VAE.The qualitative comparison focuses on similar semantic patches such as grass and trees.
  • Image generation: MoVQ outperforms most VQ-based methods for unconditional generation while using smaller models and faster inference.The comparison uses FFHQ at 256 × 256 resolution; ViT-VQGAN uses a much larger second-stage model and longer training.
  • Image generation: On ImageNet class-conditional generation, MoVQ significantly improves over VQGAN and is competitive with RQ-VAE without a specially designed multichannel transformer.The model generates high-fidelity, diverse samples, including clear and diverse backgrounds in the reported class-conditional examples.
  • Ablations: Multichannel representation improves reconstruction over the VQGAN baseline, while Fourier features provide a significant improvement among tested spatial initializations.The ablations compare decoder architectures, spatial feature choices, codebook sizes, and latent configurations.
  • Ablations: More latent channels improve reconstruction through lower compression, but generation requires a trade-off because the second stage must predict more information.The authors identify combining spatial vectors with channel information in the likelihood model as a direction for future generation improvements.

5 Conclusion

MoVQ improves VQ-based image synthesis by combining spatially conditional normalization and multichannel latent maps with a transformer-based prior. Experiments report better image modeling and higher-fidelity unconditional and class-conditional samples without increased computational cost, while multichannel representations can omit structural information.

  • MoVQ incorporates spatially conditional normalization and multichannel latent maps into a VQGAN-style architecture.The encoder and decoder remain simple as in VQGAN, while the quantizer and prior are improved.
  • MoVQ significantly outperforms state-of-the-art VQ models on image modeling without increasing computational cost.
  • Its unconditional and class-conditional samples achieve higher fidelity than existing methods.
  • The authors plan to apply MoVQ to image inversion, interpolation, and translation, while exploring the semantic meaning of codebook entries.
  • The model sometimes produces high-frequency appearances while missing structural information, partly due to its multichannel representation.The authors state that better generation models for multichannel indexes require further study.

A Discussion on Masked Image Reconstruction

Masked image reconstruction uses a bidirectional transformer to infer randomly hidden tokens from visible ones. The model can produce identity-preserving or diverse reconstructions under severe masking, with sampling choices trading off quality and diversity.

  • A binary mask hides tokens at ratios from 100% to 20%, and a bidirectional transformer predicts invisible tokens from the remaining visible tokens.
  • With 95% of tokens masked, one-step top-1 sampling can generate pluralistic images that retain identity attributes.Only 12 of 256 tokens are visible in each channel in this setting.
  • Stochastic values are gradually reduced during later random samples to balance reconstruction quality and diversity.
  • At a 100% mask ratio, multistep random sampling produces plausible and diverse images.Top-1 predictions yield a neutral face, while random results approach top-1 results as more tokens become visible.

B Additional Image Generation Results

Additional samples show MoVQ producing diverse, high-fidelity images for unconditional FFHQ generation and class-conditional ImageNet generation. The FFHQ samples are competitive with StyleGAN and appear to have cleaner backgrounds, while ImageNet samples use 12-step conditioning.

  • On FFHQ, MoVQ produces diverse, high-fidelity unconditional samples competitive with StyleGAN.The samples are drawn from the first 100 generated images without additional sampling.
  • MoVQ samples on FFHQ appear to have cleaner backgrounds than StyleGAN samples.
  • On ImageNet, MoVQ generates high-fidelity and diverse class-conditional images across various categories.The samples are conditioned on class labels and use 12-step sampling.
  • The ImageNet results include clear, diverse backgrounds, including grass in the Red Panda class.
  • Additional qualitative ImageNet results cover animals, foods, and natural scenes.These figures extend the main class-conditional generation results.
Loading 2209.09002v1…