Source-linked AI summary
CogView: Mastering Text-to-Image Generation via Transformers
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, Jie Tang
TL;DR
General-domain text-to-image generation requires powerful generation and cross-modal understanding. CogView combines a 4-billion-parameter Transformer with VQ-VAE tokens, finetuning for downstream tasks, and stabilization methods for pretraining. It outperforms DALL-E and prior GAN-based methods in FID on blurred MS COCO, while its authors identify slow autoregressive generation and VQVAE-induced blurriness as limitations.
Problem
General-domain text-to-image generation remains an open problem requiring both powerful generative modeling and cross-modal understanding.
Method
CogView jointly pretrains a 4-billion-parameter Transformer on text and VQ-VAE image tokens, then applies finetuning and precision-stabilization techniques.
Results
CogView outperforms DALL-E and previous GAN-based methods by a large margin in FID on blurred MS COCO.
Takeaways & Limitations
CogView demonstrates scalable cross-modal generative pretraining with finetuning for diverse downstream tasks and methods that stabilize large-Transformer training.
Takeaways & Limitations
Generation is slow because images are produced token-by-token, and lossy VQVAE compression makes generated images blurrier than real images.
Abstract
from arXiv · showhide
Text-to-Image generation in the general domain has long been an open problem, which requires both a powerful generative model and cross-modal understanding. We propose CogView, a 4-billion-parameter Transformer with VQ-VAE tokenizer to advance this problem. We also demonstrate the finetuning strategies for various downstream tasks, e.g. style learning, super-resolution, text-image ranking and fashion design, and methods to stabilize pretraining, e.g. eliminating NaN losses. CogView achieves the state-of-the-art FID on the blurred MS COCO dataset, outperforming previous GAN-based models and a recent similar work DALL-E.
1 Introduction
CogView addresses general-domain text-to-image generation with a large Transformer trained jointly on text and image tokens. It extends beyond zero-shot generation through finetuning, self-reranking, finer-grained evaluation, and training-stabilization techniques.
- Text-to-image generation requires both visual generation and cross-modal understanding to disentangle image features, interpret text, align words with visual content, and model complex compositions.
- CogView pretrains a 4-billion-parameter Transformer on 30 million Chinese text-image pairs using VQ-VAE image tokens.
- CogView outperforms DALL-E and previous GAN-based methods by a large margin in FID on blurred MS COCO and is presented as the first open-source large text-to-image Transformer.
- Finetuning adapts CogView to style learning, super-resolution, image captioning, and text-image reranking beyond zero-shot generation.
- Self-reranking removes DALL-E's need for an additional CLIP model, while Caption Loss evaluates text-image generation more finely than FID and Inception Score.
- PB-relaxation and Sandwich-LN stabilize large-Transformer training, eliminate overflow characterized by NaN losses, and support CogView training with almost FP16.
2 Method
CogView combines a VQ-VAE-style discrete image tokenizer with a 4-billion-parameter autoregressive Transformer trained on concatenated text and image tokens. The method also introduces stabilization techniques for large-Transformer pretraining and evaluates alternative tokenizer-training strategies.
- Theory: CogView optimizes the joint image-text likelihood through an ELBO formulation that reduces to a VQ-VAE interpretation when text is absent.The model uses a variational encoder and a text-conditional prior over image latents.
- Training framework: Training proceeds in two stages: an encoder-decoder minimizes reconstruction loss, then GPT models concatenated text and image-token sequences with two NLL losses.The first stage becomes a discrete auto-encoder, while GPT performs most of the generative modeling.
- Image tokenizer: The image tokenizer encodes images into h × w discrete codebook indices, which the decoder reconstructs into images.Each encoder output vector is quantized to a nearby learnable codebook embedding before decoding.
- Tokenizer comparison: Four image-tokenizer training strategies are compared, and their loss curves are essentially evenly matched when codebook initialization is appropriate.CogView uses the moving-average tokenizer during pretraining.
- Autoregressive Transformer: The backbone is a 48-layer unidirectional Transformer with hidden size 2560, 40 attention heads, 4 billion parameters, and separator tokens marking text-image boundaries.Sequences are clipped or padded to length 1088.
- Stabilization of training: PB-relaxation and Sandwich-LayerNorm stabilize large-Transformer training by preventing attention overflow and limiting layer-by-layer value growth.Together they support CogView training with almost FP16 and also stabilize an 8.3B-parameter CogView-large model.
3 Finetuning
CogView is finetuned for super-resolution, self-reranking, style learning, and fashion design, extending its pretrained text-image capabilities to diverse downstream tasks. These adaptations include generating new high-resolution structures, selecting better-aligned images, transferring styles, and supporting fashion production.
- Overview: CogView improves text-to-image generation through finetuning for super-resolution and self-reranking, with all finetuning tasks completed within one day on a single DGX-2.The paper presents finetuning as a step beyond DALL-E.
- Super-resolution: Lossy compression from 256 × 256 pixels to 32 × 32 tokens makes generated images blurrier, motivating super-resolution that adds meaningful details rather than only sharpening textures.Previous restoration methods target already high-resolution images and do not directly address this setting.
- Super-resolution: CogView first predicts 32 × 32 tokens from 16 × 16 tokens, then expands to 64 × 64 tokens patch-by-patch using a center-continuous sliding window.The center-continuous order preserves the completeness of the central area better than raster scanning.
- Image Captioning and Self-reranking: For self-reranking, Caption Loss measures text-image correspondence, and images with the lowest Caption Loss are selected.Caption Loss is the cross-entropy loss for text tokens conditioned on the image and preceding text tokens.
- Image Captioning and Self-reranking: Self-reranking achieves better FID than CLIP-based selection while using less computational resource because it requires only finetuning.The method reverses text-image generation to evaluate correspondence and is illustrated by diverse generated samples.
- Style Learning and Fashion Design: Style finetuning uses 1,000 images per style across Chinese traditional drawing, oil painting, sketch, and cartoon, while fashion finetuning trains on about 10 million fashion-caption pairs.The fashion model uses 50×50 VQGAN image tokens, decodes to 800 × 800 pixels, and was deployed to Alibaba Rhino fashion production.
4 Experimental Results
CogView is evaluated through machine metrics, self-reranking, and human preference studies, including fashion-design generation. It outperforms GAN-based baselines and benefits from self-reranking and super-resolution.
- Machine Evaluation: CogView is evaluated on blurred MS COCO using FID and related machine metrics, with comparisons to DALL-E and GAN-based methods.The evaluation uses 30,000 translated captions, Gaussian-blurred images, and no super-resolution for the DALL-E comparison.
- Machine Evaluation: Caption Loss provides an absolute text-image score that is more consistent with human preference than FID and IS alone.DM-GAN has the best unblurred FID and IS but ranks last in human preference.
- Machine Evaluation: Self-reranking produces better FID than CLIP and steadily improves FID as the number of candidates increases.CLIP performs better for increasing IS, but the paper argues it is less suitable for text-to-image evaluation.
- Human Evaluation: Human evaluation compares CogView with GAN baselines and recovered ground truth across 2,950 image-comparison groups.Recovered ground truth is obtained by encoding and decoding the ground-truth image, forming a theoretical upper bound for CogView.
- Human Evaluation: CogView is preferred over GAN-based baselines by a large margin, with 37.02% preference versus 59.53% for recovered ground truth.The super-resolution model further improves clarity and is reported to outperform recovered ground truth in Figure 10(b)(c).
5 Conclusion and Discussion
CogView combines VQ-VAE and Transformers for scalable text-to-image generation, supports diverse finetuning applications, and identifies important safety and quality limitations.
- Limitations: CogView’s limitations include slow token-by-token generation and blur introduced by lossy VQ-VAE compression.The paper states that both problems are left for future work.
- Ethics Concerns: CogView is vulnerable to malicious use and fairness problems because of its controllable and strong image-generation capacity.The paper discusses mitigation through a survey and a simple word-replacing method for fairness.
- Conclusion: The paper systematically investigates VQ-VAE–Transformer text-to-image generation, precision stabilization, and finetuning for diverse downstream tasks.The stated aim is to advance controllable image generation and cross-modal knowledge understanding while preventing misinformation use.
A Data Collection and Details about the Tokenizers
The authors build a large multimodal corpus from diverse sources and describe the tokenizer components used to represent text and images.
- Data Collection: The dataset contains about 30 million text-image pairs and reaches 2.5TB before tokenization and about 250GB afterward.About half of the text is English, including Conceptual Captions translated into Chinese by machine translation.
- Data Collection: Data comes from professional image websites, Conceptual Captions, ImageNet, news pictures, Alibaba item-caption pairs, and image search engines.The sources are selected to cover common entities and include both Chinese and English material.
- Tokenizers: The text tokenizer uses SentencePiece, while the image tokenizer uses a four-layer convolutional encoder with 512 hidden units per layer.The first three convolutional layers use receptive field 4 and stride 2; the final layer is a 1 × 1 convolution.
B Sparse Attention
Three-region sparse attention combines global text and pivot-token access with local blockwise windows to reduce computation and memory. It accelerates training without changing loss curves, but was not used for the 4-billion-parameter CogView because of super-resolution compatibility concerns.
- Design: Each token attends to all text tokens, randomly sampled image pivots, and tokens in adjacent blocks.The design provides global information through pivots and local information through blockwise windows.
- Efficiency: On 4096-token sequences, sparse attention is 2.5× faster than vanilla attention and saves 40% GPU memory.The benchmark uses 768 text and pivot tokens plus 768 blockwise-window tokens.
- Efficiency: Whole training is 1.5× faster and saves 20% GPU memory, while matched loss curves indicate no convergence change.The comparison uses identical hyperparameters, data, and random seeds.
- Limitation: Three-region sparse attention was not used to train the 4-billion-parameter CogView because it might be incompatible with super-resolution finetuning.It successfully accelerated CogView-fashion training without side effects.
C Attention Analysis
CogView’s attention captures image positional structure and exhibits broader spatial biases across layers, while attention patterns vary during inference.
- Attention visualizations reveal positional structure as a central property of CogView’s attention distribution.
- Attention heads strongly track image position, including fixed offsets, row regions, checkerboard boundaries, and broad spatial areas.
- Some heads attend selectively to a few tokens, including separator tokens.
C.2 Semantic Segmentation
CogView’s attention heads capture salient objects named in text, indicating implicit semantic segmentation. Attention also shifts from positional information in early layers toward content, becoming sparse in the final layers.
- Semantic segmentation: Attention heads successfully captured salient objects such as apples and purple flowers named in the input text.The experiment marked pixels corresponding to highly attended tokens with red dots.
- Layer-wise attention: Earlier layers focus mainly on positional information, whereas later layers focus more on content.
- Layer-wise attention: After layer 42, attention becomes sparse, with many heads attending to only a few tokens such as separator tokens.
C.4 Value Scales of Attention
Attention-score scales vary substantially across heads while remaining relatively stable within each head. The model addresses resulting bias through PB-relax, which removes attention bias during computation.
- Attention-score scales: Attention-score scales vary dramatically across heads but have low variance within each individual head.The passage links this pattern to non-degenerate attention despite large scores.
- Stabilization: PB-relax removes attention bias during computation to address the large bias that can accompany differing attention scales.
- Attention-score scales: Text-to-text attention scores are smaller than other attention-score scales in the 38-th layer.The figure reports the minimum-to-maximum range with error bars and relates the scales to the data.
D Fairness in CogView: Situation and Solution
CogView generates nearly correct specified races, genders, and occupations but exhibits gender stereotypes when gender is unspecified. The paper proposes prepending sampled demographic adjectives to human-related text, using population proportions.
- Situation: With detailed racial or gender descriptions, CogView generates correctly for almost all samples.
- Situation: Unspecified-gender generation exhibits gender bias because of stereotypes, while specified gender yields almost-correct gender and occupation combinations.
- Solution: The proposed word-replacing solution prepends adjectives such as racial and gender labels to human-related words when those attributes are unspecified.
- Solution: The adjectives are sampled according to real population proportions, and an additional NER model can identify words referring to humans.
- Solution: Because CogView follows specified descriptions, the proposed method is reported to help address fairness in generative models.
E Details about Human Evaluation
The human evaluation compares CogView with several GAN-based methods using randomly selected MS COCO captions and standardized image-quality judgments. It also tests super-resolution through a balanced A-B design and reports evaluation examples and score summaries.
- 50 MS COCO images and captions were randomly selected to compare AttnGAN, DM-GAN, DF-GAN, and CogView; DALL-E was excluded because its model was unreleased.
- Evaluators scored each generated image and a recovered ground-truth image for clarity, texture quality, caption relevance, overall quality, and best-image selection.The three aspect scores used a 1–5 scale, while overall quality used a 1–10 scale.
- 72 evaluators were invited, but 59 were retained after requiring complete answers and agreement between best-image selections and highest overall-quality scores.Retained evaluators were compensated 150 yuan each, with no time limit.
- Super-resolution evaluation: A balanced A-B test compared CogView images generated with and without super-resolution across randomly divided evaluator and caption groups.Each group evaluated the opposite super-resolution condition for the two caption subsets, yielding equal numbers of results.
- Average scores and standard deviations were plotted in Figure 10, with additional evaluation examples and website snapshots provided in Figures 16, 17, and 18.