PaperlayerView exact revision on arXiv

Source-linked AI summary

CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers

Ming Ding, Wendi Zheng, Wenyi Hong, Jie Tang

arXiv:2204.14217v2cs.CVcs.LG

TL;DR

Transformer-based text-to-image models face slow generation and expensive high-resolution processing. CogView2 combines flexible CogLM pretraining with hierarchical transformers and local parallel autoregressive super-resolution. It reduces generation steps substantially, is about 10× faster than CogView, and achieves competitive image generation while supporting text-guided editing.

  • Problem

    Autoregressive text-to-image models are limited by slow token-by-token generation and the high time and space complexity of attention for high-resolution images.

  • Method

    CogView2 pretrains CogLM on masked text and image tokens, then uses hierarchical direct and iterative super-resolution with local attention and local parallel autoregressive generation.

  • Results

    CogView2 reduces generation steps from 3,600 to 6, is about 10× faster than CogView, and achieves the best blurred FIDs among comparable methods.

  • Takeaways & Limitations

    Hierarchical transformers and local parallel autoregression help autoregressive text-to-image models address slow generation and high-resolution complexity.

  • Takeaways & Limitations

    Direct super-resolution focuses more on texture transformation than generation, while third-level super-resolution is left for future work; generated images may also pose misinformation risks.

Abstract

from arXiv · show

The development of the transformer-based text-to-image models are impeded by its slow generation and complexity for high-resolution images. In this work, we put forward a solution based on hierarchical transformers and local parallel auto-regressive generation. We pretrain a 6B-parameter transformer with a simple and flexible self-supervised task, Cross-modal general language model (CogLM), and finetune it for fast super-resolution. The new text-to-image system, CogView2, shows very competitive generation compared to concurrent state-of-the-art DALL-E-2, and naturally supports interactive text-guided editing on images.

1 Introduction

CogView2 addresses slow autoregressive generation, expensive high-resolution training, and unidirectional modeling with CogLM, hierarchical generation, and local parallel autoregression.

  • Autoregressive text-to-image models generate slowly because token-by-token decoding cannot fully exploit GPU parallelism.
  • Transformer attention costs O(n^2) in time and space, creating a trade-off between model capacity and high-resolution image generation.
  • CogLM masks text and image tokens flexibly, supporting text-to-image generation, image infilling, and image captioning within one pretraining framework.
  • CogView2 hierarchically generates 20 × 20 images, directly super-resolves them to 60 × 60 tokens, then iteratively refines the high-resolution result.
  • Local parallel autoregressive generation reduces high-resolution generation steps from 3,600 to 6, while CogView2 is about 10× faster than CogView.
  • CogLM uses text and image tokens in one sequence, with masking changing the attention map rather than the input sequence.

2 Related Work

Related work progressed from domain-specific GANs to large pretrained autoregressive transformers and diffusion models, while non-autoregressive methods target faster decoding.

  • Early GAN-based text-to-image systems produced vivid results on domain-specific datasets but faced challenges on general-domain datasets such as MS COCO.
  • DALL-E, CogView, and related models use VQ-VAE image-token compression and large transformers for autoregressive generation in the general domain.
  • Diffusion models have attracted interest partly because autoregressive text-to-image models generate slowly, with Glide given as an example.
  • Non-autoregressive research explores parallel decoding for autoregressive-like models, including Mask-Predict, GLAT, M6-UFC, and VQ-diffusion.

3 Method

CogView2 uses CogLM to unify autoregressive generation with bidirectional mask prediction, then builds a hierarchical super-resolution pipeline with local attention and parallel refinement. The method combines direct mapping from 20 × 20 to 60 × 60 tokens with iterative LoPAR refinement, while acknowledging that direct mapping mainly transforms texture rather than generating full image distributions.

  • 3.1 The Cross-Modal General Language Model: CogLM unifies autoregressive generation with bidirectional context-aware mask prediction across text and image tokens.Its masking framework supports text-to-image generation, image infilling, and image captioning.
  • 3.1 The Cross-Modal General Language Model: The attention mask lets unmasked tokens provide context while masked-region tokens predict subsequent tokens within their regions.During inference, moving the last context token before each mask region enables infilling; regions may be processed individually to preserve context.
  • 3.2 Pretraining: CogLM is pretrained as a 6-billion-parameter Transformer with a unified Chinese-English-image tokenizer and randomized masking strategies.The model uses 20,000 image tokens, 130,000 text tokens, and masks random 4 × 4 token patches until 75% of tokens are masked for mask prediction.
  • 3.3 Hierarchical Generation: The hierarchical pipeline first generates 20 × 20 images, then maps them to 60 × 60 tokens through direct super-resolution using cross-resolution local attention.The direct module uses a pretrained CogLM encoder-decoder and fine-tunes decoder attention weights to reduce memory consumption.
  • 3.3 Hierarchical Generation: Direct super-resolution mainly predicts marginal token or pixel distributions, so a second module is needed to refine the high-resolution image.The paper characterizes direct mapping as texture transformation rather than fully qualified generation.
  • 3.3 Hierarchical Generation: LoPAR refines high-resolution images by re-masking 75% of tokens while retaining local context, allowing local windows to generate tokens in parallel.The hierarchical process is intended to remove global dependence, and diagonal factorization reduces inconsistency from simultaneously generating adjacent tokens.

4 Plug-in Improved Techniques for Transformers

CogView2 adds cluster sampling, efficient local attention, and textual-attention upweighting to address sampling distortion, computational cost, and text-image relevance.

  • 4.1 Cluster Sampling: Top-k and top-p sampling can incompletely truncate distributions when visually similar image-token embeddings represent related patterns.A 20,000-token vocabulary worsens this issue because related tokens may be filtered separately.
  • 4.1 Cluster Sampling: Cluster sampling groups 20,000 image tokens into 500 VQ-VAE-embedding clusters and samples clusters before sampling individual tokens.Keeping cluster members together alleviates incomplete truncation.
  • 4.2 Local Attention: A customized CUDA kernel supports efficient 2D local attention, which high-level frameworks such as PyTorch cannot implement efficiently.Figure 6 compares CUDA local attention with full attention and PyTorch unfold-based implementation at hidden size 64.
  • 4.3 Upweighting Textual Attention: Adding a constant c to attention scores from tokens to text tokens improves textual relevance with negligible time consumption.The authors report that c < 3 does not influence image quality.

5 Experiments

Experiments evaluate CogView2 on MS-COCO with machine metrics and large-scale human comparisons. CogView2 achieves the best blurred FIDs and the strongest human-evaluation performance, while COCO fine-tuning improves FID but hurts human-rated quality.

  • 5.2 Machine Evaluation: CogView2 achieves the best blurred FIDs among all comparable methods.The machine evaluation uses MS-COCO captions and downsamples CogView2 images to 256×256 for comparison.
  • 5.2 Machine Evaluation: Fine-tuning on MS-COCO reduces FID from 24.0 to 19.2 after 2,500 iterations and 17.5 after 7,500 iterations.The same fine-tuning causes human-evaluated generation quality to deteriorate.
  • 5.2 Machine Evaluation: The non-fine-tuned version generates more accurate images than the COCO-fine-tuned version despite the latter’s similar COCO style.This difference corresponds to the human-evaluation scores in Figure 7.
  • 5.3 Human Evaluation: CogView2 performs best in all aspects of the human evaluation.The evaluation contains 4,600 comparison groups across several public text-to-image systems and recovered ground truth.

6 Discussion

The discussion contrasts autoregressive and diffusion image generation and compares CogView2 with DALL-E-2. CogView2 follows a hierarchical-generation spirit while leaving a third super-resolution level for future work.

  • Autoregressive or Diffusion?: Diffusion models require more FLOPs than autoregressive models under the same architecture but offer greater parallelism and adjustable quality-time trade-offs.Sampling schedules can manually vary diffusion stride to balance quality and time consumption.
  • Comparison between DALL-E-2 and CogView2: CogView2 and DALL-E-2 share hierarchical generation, and CogView2 synthesizes similar scenes despite using about 5% of DALL-E-2’s training data.CogView2 lacks DALL-E-2’s third-level super-resolution and zeroth-level image-prior generation; the third level is left for future work.

7 Conclusion

CogView2 addresses autoregressive text-to-image models’ slow generation and high complexity through hierarchical transformers, while also bridging text-to-image pretraining with bidirectional visual learning. The paper notes both creative benefits and misinformation risks from text-guided image generation.

  • Hierarchical transformers help autoregressive models remedy slow generation and high complexity while bridging text-to-image pretraining with visual representation learning.
  • Text-guided image generation may ease creative work for artists and designers, but may also increase misinformation risks involving web photos.
  • A classifier based on texture features can distinguish real images from CogView2-generated images.

A Details of the Tokenizers

The unified tokenizer, icetk, combines multilingual text tokenization with a large shared vocabulary and controllable token sampling for different generation tasks.

  • icetk is a unified tokenizer whose text component is trained on a mixed English-Chinese corpus using SentencePiece’s unigram algorithm.
  • The icetk dictionary contains 150,000 tokens spanning image, common, English, Chinese, and special-token categories.
  • Sampling masks can restrict generated token categories, such as allowing only English tokens for image captioning.

A.2 Image Tokenizer

CogView2’s image tokenizer uses perceptual reconstruction losses and multiple compression rates, balancing texture detail, accuracy, and flexible image-token resolutions.

  • Perceptual loss improves reconstruction quality over the comparison without it, while VQGAN preserves more detail but can distort important elements such as faces.
  • The image tokenizer is a multi-compression-rate VQVAE using perceptual loss as a key difference from CogView’s tokenizer.
  • In this paper, VQVAE refers only to the first-stage image compression process, not the autoregressive prior.
  • The reconstruction loss combines L1 error, MS-SSIM, and perceptual loss between the input image and recovered output.
  • The tokenizer supports 4^2, 8^2, and 16^2 compression rates, although CogView2 experiments use 8^2 compression.
  • The multi-compression-rate design trains three tokenizers with shared dictionary and low-level-layer parameters, using pooling as the DownSample operation.

B High-resolution Text-guided Infilling

CogView2 performs high-resolution text-guided infilling by enlarging the masked region, completing and super-resolving it, then restoring edge coherence. The procedure supports applications such as portrait editing.

  • Small-patch infilling can prioritize contextual coherence over text because CogLM operates at 20 × 20 granularity.
  • The procedure begins by computing the mask’s square bounding box and enlarging it by 1.4 as the default hyperparameter.
  • For a partial region, CogLM completes and super-resolves a magnified 480 × 480 patch before scaling it back and replacing the masked area.
  • The resulting image is immediately tokenized and detokenized to preserve coherence at the masked-region boundary.
  • Examples demonstrate CogView2’s potential for text-guided infilling applications such as portrait editing.

C Details about Human Evaluation

The human evaluation compares generated images across clarity, texture quality, caption relevance, and overall preference, using both ratings and best-image selection.

  • Evaluators scored each image from 1 to 5 for clarity, texture quality, and caption relevance.
  • Evaluators also assigned an overall score from 1 to 10.
  • The evaluation used 50 randomly selected MS COCO captions.
  • Evaluators selected the best image after scoring the samples.
  • 92 of 100 recruited evaluators completed the tasks, with each scorer evaluating 300 images.Participants were paid 75 yuan; a small subset may have scored randomly, potentially explaining DF-GAN wins in 1.26% of questions.
Loading 2204.14217v2…