PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Autoregressive Models for Content-Rich Text-to-Image Generation

Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, Yonghui Wu

arXiv:2206.10789v1cs.CVcs.LG

TL;DR

Text-to-image generation requires models to turn rich language descriptions into detailed images, including complex compositions and world knowledge. Parti uses ViT-VQGAN to tokenize images into discrete sequences and an encoder-decoder Transformer to generate those tokens, scaling the model to 20B parameters. It achieves 7.23 zero-shot and 3.22 finetuned FID on MS-COCO, while the paper also evaluates longer descriptions and documents limitations in precise visual control.

  • Problem

    Text-to-image models need to generate high-fidelity images from rich descriptions involving complex compositions and world knowledge.

  • Method

    Parti uses a two-stage architecture in which ViT-VQGAN tokenizes images and an autoregressive encoder-decoder Transformer generates image tokens from text.

  • Results

    7.23 zero-shot FID and 3.22 finetuned FID are reported on MS-COCO, alongside strong zero-shot and finetuned results on Localized Narratives.

  • Takeaways & Limitations

    Scaling autoregressive models enables Parti to generate diverse, high-quality images and represent visual world knowledge in novel settings and configurations.

  • Takeaways & Limitations

    Parti remains imprecise in visual control, motivating combinations of broad capability, visual quality, and stronger control.

Abstract

from arXiv · show

We present the Pathways Autoregressive Text-to-Image (Parti) model, which generates high-fidelity photorealistic images and supports content-rich synthesis involving complex compositions and world knowledge. Parti treats text-to-image generation as a sequence-to-sequence modeling problem, akin to machine translation, with sequences of image tokens as the target outputs rather than text tokens in another language. This strategy can naturally tap into the rich body of prior work on large language models, which have seen continued advances in capabilities and performance through scaling data and model sizes. Our approach is simple: First, Parti uses a Transformer-based image tokenizer, ViT-VQGAN, to encode images as sequences of discrete tokens. Second, we achieve consistent quality improvements by scaling the encoder-decoder Transformer model up to 20B parameters, with a new state-of-the-art zero-shot FID score of 7.23 and finetuned FID score of 3.22 on MS-COCO. Our detailed analysis on Localized Narratives as well as PartiPrompts (P2), a new holistic benchmark of over 1600 English prompts, demonstrate the effectiveness of Parti across a wide variety of categories and difficulty aspects. We also explore and highlight limitations of our models in order to define and exemplify key areas of focus for further improvements. See https://parti.research.google/ for high-resolution images.

1 Introduction

Parti frames text-to-image generation as sequence-to-sequence modeling, using discrete visual tokens to connect language inputs with high-fidelity image outputs. It scales this autoregressive approach to support photorealistic and content-rich synthesis.

  • Parti targets descriptions involving rich scenes, object combinations, and visual concepts that extend beyond simple everyday image captions.The motivating examples include complex compositions and combinations of objects and concepts.
  • Diffusion models have recently improved text-to-image quality, but autoregressive models remain attractive because of advances in scaling language models and discretizing modalities.Parti builds on this autoregressive direction while generating photorealistic images, paintings, drawings, and other outputs.
  • Parti treats text-to-image generation as sequence-to-sequence modeling, with text tokens encoded as inputs and discrete image tokens predicted autoregressively.Its Transformer encoder-decoder architecture follows the machine-translation analogy while targeting visual tokens.
  • ViT-VQGAN provides the discrete image-token representation used by Parti.The tokenizer produces higher-fidelity reconstructions and better codebook utilization than dVAE-based alternatives.

F. The saying "BE EXCELLENT TO EACH OTHER" ..., (a) brick

The selected examples illustrate Parti’s ability to handle varied prompts involving text, image-within-image specifications, world knowledge, detailed compositions, and concept combinations. They also show diversity across multiple outputs from one prompt.

  • Parti’s scaling results are associated with improved text-image alignment and image quality, including a 7.23 zero-shot FID score.The 20B model is described as supporting content-rich synthesis involving complex compositions and world knowledge.

2 Parti Model

Parti is a two-stage text-to-image system: an image tokenizer converts images into discrete visual tokens, and an encoder-decoder Transformer generates those tokens from text. The design supports scaling and incorporates guidance, reranking, and super-resolution components.

  • Parti first tokenizes images into discrete visual-token sequences, then trains an autoregressive model to generate image tokens from text tokens.The generated tokens are decoded back into pixels at inference time.
  • Discrete visual tokens replace long, low-level pixel sequences with learned codebook entries representing image patches.This makes image representations more compact and richer than direct pixel modeling.
  • ViT-VQGAN is used to reconstruct high-quality images and provide balanced codebook usage for the autoregressive stage.The tokenizer uses normalized and factorized codes to support stability, reconstruction quality, and codebook utilization.
  • A super-resolution module upsamples reconstructed 256 × 256 images to 1024 × 1024 without conditioning on text inputs.It is built on a frozen ViT-VQGAN tokenizer.
  • Encoder-decoder Parti variants outperformed decoder-only variants at 350M to 750M parameters in early experiments.The comparison covered both training loss and text-to-image generation quality.
  • Classifier-free guidance and contrastive reranking are applied to improve alignment and select outputs from sampled image sets.Reranking uses image-text embedding alignment scores, and the authors report complementarity with classifier-free guidance.

3 Scaling

Parti uses progressively more distributed training strategies as model size increases, combining model, pipeline, and data parallelism. The 20B configuration uses staged pipelines and a circular decoder schedule to reduce pipeline stalls.

  • GSPMD distributes Parti training and inference across TPU clusters by automatically partitioning data and computation.The system treats a TPU cluster as a single virtual device using sharding annotations.
  • The 20B model uses separate 16-stage encoder and decoder pipelines plus 64-way data parallelism.Pipeline parallelism is chosen because the model has many layers with moderate per-layer weight sizes.
  • A 4-round circular decoder schedule reduces pipeline bubble ratio caused by inter-stage data dependencies.The schedule executes four layers in each stage in round-robin order.
  • The 3B model uses 4-way in-layer model parallelism together with 128-way data parallelism.Feed-forward hidden dimensions, attention-head dimensions, and internal activations are partitioned.
  • Localized Narratives descriptions are four times longer than MS-COCO captions on average, providing a more detailed evaluation setting.The datasets use images from the COCO portion of Localized Narratives and the MS-COCO 2017 set.

4 Training and Evaluation Datasets

Parti is trained on large image-text datasets and evaluated on MS-COCO and Localized Narratives, with P2 extending evaluation to diverse, content-rich prompts and challenge aspects. P2 separates prompt categories from difficulty dimensions for finer-grained analysis.

  • Training datasets: Parti training combines LAION-400M, filtered FIT400M, and JFT-4B image-text datasets.JFT descriptions use either original labels or machine-generated SimVLM captions.
  • Evaluation datasets: Evaluation uses MS-COCO for standard zero-shot and finetuned comparison, plus Localized Narratives for longer descriptions and out-of-domain generalization.Localized Narratives descriptions are typically longer and differ from training descriptions.
  • PartiPrompts benchmark: P2 provides 1600 diverse English prompts spanning broad categories and challenging aspects to test open-domain text-to-image synthesis.Its categories range from abstract concepts to concrete world knowledge, while challenge aspects range from basic objects to complex descriptions.
  • PartiPrompts benchmark: Each P2 prompt receives one primary category and one primary challenge aspect, enabling analysis of generated content and captured details.The labeling procedure resolves cases with multiple possible labels by applying manual priorities, including favoring world knowledge for proper nouns.
  • Benchmark design: Compared with DrawBench, P2 separates categories from challenge aspects and contains 8× more prompts for finer-grained analysis.DrawBench mixes the two label types in eleven labels across 200 prompts.

5 Experiments

Parti is evaluated with automated metrics and human judgments across MS-COCO, Localized Narratives, and PartiPrompts. Scaling from 3B to 20B parameters improves quality, alignment, and performance on challenging prompts.

  • Automated evaluation: Parti achieves a finetuned MS-COCO FID of 3.22, improving over Make-a-Scene’s previous best autoregressive FID of 7.55.Its zero-shot FID is 7.23, comparable to Imagen, and its in-dataset retrieval baseline obtains 6.82.
  • Automated evaluation: 8.29 finetuned FID on LN-COCO substantially improves over XMC-GAN’s 14.12 and the retrieval baseline’s 16.48.Parti’s zero-shot FID is 15.97, nearly matching XMC-GAN’s finetuned score, while qualitative comparisons favor Parti in realism and image-text fit.
  • Automated evaluation: Parti outperforms other models on automatic image-text alignment and closes much of the gap to captions generated from ground-truth images.The captioner evaluation is limited by the captioner’s ability to discriminate between outputs from different approaches.
  • Human evaluation: 91.7% of human judgments preferred zero-shot Parti over finetuned XMC-GAN for realism, and 90.5% preferred it for image-text match on MS-COCO.Against retrieved images, Parti scored 45.2% versus 54.8% on realism but 55.2% versus 44.8% on image-text match.
  • Model scaling: Scaling from 750M to 3B yields a significant quality jump, while the 20B model further improves challenging prompts such as text rendering.All four model sizes use the same dataset mixture, image tokenizer, and CoCa reranking model.
  • PartiPrompts: On PartiPrompts, the 20B model is preferred over the 3B model for realism by 56.8% and for image-text match by 62.7%.It is especially stronger on world knowledge, writing and symbols, perspective, imagination, abstract concepts, and quantity.

6 Discussion

Parti handles substantial descriptive and compositional complexity, but its reliability declines as prompts become more complex. The discussion identifies recurring failures in counting, spatial relations, attributes, omissions, and scene integration.

  • Strengths: Parti accommodates complex prompts, multiple visual styles, words-on-text, and world knowledge, including co-locating famous landmarks and rendering detailed scenes.The examples include a long description of Van Gogh’s The Starry Night and prompts combining Anubis with the Los Angeles skyline.
  • Interpretation: Cherry-picked examples do not represent a single-shot interaction or the model’s highest-ranked output, limiting how directly they characterize typical performance.The authors note that the most challenging examples were selected and that the model itself could not be publicly released.
  • Growing a Cherry Tree: Combining detailed entities and relations can produce missing components, confused compositions, cartoonish outputs, or incorrect positioning.In the sloth-and-van sequence, rephrasing, positional descriptions, and format changes sometimes improve results but do not reliably preserve all details.
  • Failure modes: Prompt complexity increases the likelihood of errors, including color bleeding, feature blending, omitted or hallucinated details, and displaced object interactions.These failures concern both object attributes and relationships among entities in complex scenes.
  • Failure modes: Counting is reliable up to seven same-type objects under limited conditions, but becomes mostly imprecise beyond seven and nearly fails for multiple entity types.For ten red apples, one batch’s top-eight counts were 8, 10, 9, 9, 9, 8, 11, and 6.
  • Failure modes: Spatial relations are inconsistent, especially left-versus-right arrangements and relations between groups of objects.The model often depicts above-or-below relations correctly, but performance compounds poorly for more complex group relations.
  • Failure modes: Parti often draws items mentioned as absent, such as bananas or orange juice, showing difficulty with negation and absence.Some correct outputs occur at very low rankings, while generator and reranker effects may compound the problem.

7 Related Work

Text-to-image generation has progressed from earlier rule-based and conditional-GAN approaches toward sequence modeling with large image-text datasets and discrete image tokens. Diffusion models form another major approach, while image tokenizers support the autoregressive sequence-to-sequence formulation.

  • Text-to-image generation: Text-to-image generation synthesizes realistic images from natural-language descriptions, building on rule-based, explicit-3D, and conditional-GAN approaches.WordsEye is described as a pioneering rule-based system, followed by deep-learning models generating birds and flowers from language.
  • Sequence modeling: Large-scale image-text training and sequence modeling have enabled major improvements in text-to-image generation.A common two-stage framework first tokenizes images into discrete latent variables, making the task sequence-to-sequence modeling for Transformer language models.
  • Diffusion models: Diffusion models generate images by learning to reverse noise added iteratively under text conditioning, and recent systems achieve impressive results.These systems commonly condition on CLIP or other pretrained text encoders, with cascading used in some models.
  • Image tokenizers: Image tokenizers convert images into discrete latent variables, with VQGAN improving on dVAE through adversarial and perceptual losses.dVAEs can produce blurry natural-image pixels, motivating improved tokenizer objectives.

8 Broader Impacts

The paper discusses creative opportunities and serious risks from large-scale text-to-image models, including bias, misuse, communication problems, and restricted intended uses. These concerns motivate withholding the models, code, and data while pursuing further measurement and mitigation.

  • Creativity and art: Large-scale text-to-image models can support creative work, but artistic merit depends on the model, users, training data, and broader artistic context.Training data may contain cultural biases toward Western imagery and limit radically new artistic styles.
  • Visual (mis)communication: Text-to-image models should not yet be used as communication aids because their efficacy, utility, and cross-cultural accessibility remain insufficiently studied.Text and images convey meaning through distinct forms and limitations.
  • Deepfakes and disinformation: Photorealistic outputs raise deepfake and disinformation concerns, motivating safeguards such as imperceptible watermarks for provenance verification.The passage specifically connects confusion with real photographs to these concerns.
  • Bias and safety: Training on large, noisy image-text datasets can reproduce stereotypes, pornography, violence, and other biases.The authors specifically identify problems in LAION-400M and note additional geographic and gendered biases.
  • Intended uses: Parti is a research prototype not intended for high-risk or sensitive domains or for generating images of people.The authors cite the model’s impacts and limitations and the need for further exploration as the basis for this boundary.
  • Responsible release: The authors therefore withhold the models, code, and data and plan further bias measurement and mitigation through prompt filtering, output filtering, and model recalibration.They also describe probing models to understand biases in large image-text datasets.

9 Conclusion

The conclusion presents Parti as a diverse, high-quality autoregressive text-to-image model with scaling advantages and broad visual world knowledge. It identifies hybrid autoregressive-diffusion systems, stronger evaluation, and Responsible AI work as next steps.

  • Conclusion: Parti produces diverse, high-quality images from textual prompts and exhibits distinct scaling advantages.The conclusion frames these as the work’s central demonstrated properties.
  • Conclusion: The model represents visual world knowledge, including landmarks, specific years, vehicle makes and models, pottery types, and visual styles, in novel configurations.The examples span named entities, temporal details, object categories, and stylistic knowledge.
  • Conclusion: PartiPrompts is intentionally designed to induce diverse model error types and support analysis of the model’s limitations.The benchmark accompanies an extensive discussion of errors and challenges.
  • Future work: Future work could combine scaled autoregressive models with diffusion modules for low-resolution generation followed by iterative refinement and super-resolution.The passage describes this as an opportunity rather than an established result.
  • Future work: Additional work is needed on evaluations and Responsible AI for text-to-image generation models.The authors state that they will conduct more experiments and comparisons.
  • Examples: Selected examples probe Parti’s handling of complex prompts, multiple visual styles, text in images, and world knowledge.These examples are presented across Figures 16–19.

B Image Samples for Cross Reference and Comparison

This section presents image samples generated from prompts drawn from DALL-E, GLIDE, unCLIP, and Imagen for cross-reference and comparison. The authors modify the prompt set for impact concerns and add prompts to complete typography rows.

  • Cross-reference: The section compares Parti samples using the same text prompts as DALL-E, GLIDE, unCLIP, and Imagen.The samples are provided for cross-reference and comparison with related work.
  • Prompt construction: Some prompts are excluded or modified to address broader impact concerns discussed elsewhere in the paper.The passage also notes that new sub-prompts are added for typography rows.

C Qualitative comparison on MS-COCO

The qualitative comparison presents non-cherry-picked Parti samples alongside outputs from other approaches on MS-COCO prompts. It emphasizes zero-shot generalization and image realism while also documenting broad prompt categories and challenge aspects.

  • Qualitative comparison: Parti samples are compared with outputs from DALL-E, Make-A-Scene, GLIDE, unCLIP, and Imagen on MS-COCO prompts.Figure 24 uses sampled rather than cherry-picked images.
  • Qualitative comparison: Parti demonstrates strong generalization without fine-tuning on specific domains such as MS-COCO.The comparison passage reports this as a qualitative result.
  • Qualitative comparison: Parti achieves image realism that is typically very close to that of real images.This claim is made for the qualitative MS-COCO comparison.
  • Prompt coverage: The displayed samples include prompts involving water-made objects, animals in clothing, detailed animal descriptions, landmarks, and complex scenes.Examples include a water television, a hedgehog using a calculator, a beret-wearing shiba inu, and landmark settings.
  • Prompt coverage: The section’s broader prompt set covers complex prompts, multiple visual styles, text-on-image, world knowledge, and additional categories and challenge aspects.The categories and challenge aspects are catalogued in the P2 benchmark tables.

E Human Evaluation Procedure

The human evaluation examines Parti across categories and challenge aspects in the PartiPrompts benchmark, comparing model variants and assessing image realism. Results indicate that scaling from 3B to 20B helps most on difficult prompts, while basic prompts show little benefit or slight degradation.

  • PartiPrompts benchmark: The evaluation includes examples ranging from simple objects and concepts to detailed scenes involving multiple participants, attributes, relationships, and viewpoints.Examples include surreal paintings, food scenes, diagrams, indoor settings, outdoor settings, vehicles, landmarks, animals, and abstract concepts.
  • PartiPrompts benchmark: The P2 benchmark organizes prompts into categories including ARTS, PEOPLE, VEHICLES, PLANTS, ANIMALS, ARTIFACTS, and ABSTRACT.It also includes challenge aspects such as BASIC, COMPLEX, IMAGINATION, and PERSPECTIVE, with examples spanning indoor and outdoor scenes.
  • F.1 Effect of Model Scale: The evaluation compares the 20B and 3B Parti models across PartiPrompts categories and challenge aspects for image realism.The benchmark includes categories such as ABSTRACT, WORLD KNOWLEDGE, VEHICLES, and ARTS, alongside BASIC, COMPLEX, and IMAGINATION challenge dimensions.
  • F.1 Effect of Model Scale: The 20B model is preferred over the 3B model in most categories, especially ABSTRACT, WORLD KNOWLEDGE, VEHICLES, and ARTS.The 3B model is slightly preferred for outdoor scenes.
  • F.1 Effect of Model Scale: The 20B model outperforms the 3B model on every challenge aspect except BASIC prompts.Scaling provides marked gains on COMPLEX and IMAGINATION prompts, whereas BASIC prompts show no benefit and may slightly worsen.

F.2 Comparison Against Retrieval Baseline

Parti 20B is compared with a retrieval baseline using human preferences across PartiPrompts categories and challenge aspects. The model is preferred for image-text match in most categories and for image realism across most categories and all challenge dimensions.

  • Human evaluation: Human evaluations compare Parti 20B with a retrieval baseline across PartiPrompts categories and challenge aspects.The comparison covers both image-text match and image realism.
  • Image-text match: Parti significantly outperforms the retrieval baseline on image-text match in most categories, except ABSTRACT.The authors identify ABSTRACT prompts as difficult for image generation and note that such prompts are unlikely to appear in standard training datasets.
  • Image realism: Parti is preferred by human annotators for image realism across most P2 categories except ABSTRACT and INDOOR SCENES.The retrieval baseline uses real images from a training corpus, making this comparison notable within the reported evaluation.
  • Image realism: Parti is preferred over the retrieval baseline along all image-realism challenge dimensions.The reported outcome supports the model’s ability to synthesize images judged close to photorealistic by human evaluators.

H Pixelation Patterns of ViT-VQGAN

The authors identify pixelation patterns in some ViT-VQGAN outputs when images are viewed closely. They trace the issue to the tokenizer’s output projection and describe a decoder fine-tuning fix that removes the final sigmoid and logit-laplace loss.

  • Observed artifact: Some ViT-VQGAN outputs exhibit pixelation patterns, appearing as saturated pixel values at particular locations when zoomed in.The issue is observed in generated tokenizer outputs rather than described as a general property of all images.
  • Cause: The authors find ill-conditioned weight matrices in the output projection layer before the sigmoid activation as a source of the artifact.This diagnosis connects the visible pixelation to the tokenizer’s output projection.
  • Fix: Removing the final sigmoid activation and logit-laplace loss exposes raw RGB values in the range [0, 1].The change is presented as a fix for the pixelation patterns.
  • Fix: The fix can be hot-swapped into an already trained image tokenizer by fine-tuning its decoder.This avoids requiring a complete tokenizer retraining according to the authors’ description.
Loading 2206.10789v1…