Muse: Text-To-Image Generation via Masked Generative Transformers

Model approach

• Muse uses a pre-trained T5-XXL text encoder to condition a masked image-modeling approach for text-to-image synthesis.The model's image decoder is conditioned on frozen T5-XXL embeddings, while the image-generation models use Transformer architectures.Source: §1 · p. 1

Efficiency

• More than 10x faster inference than Imagen-3B or Parti-3B and 3x faster than Stable Diffusion v1.4 is estimated on comparable TPU-v4 hardware.These comparisons use images of the same size, either 256 × 256 or 512 × 512.Source: §1 · pp. 2–3

Results

• A 3B-parameter Muse model achieves a CLIP score of 0.32 and FID score of 7.88 on zero-shot COCO, while a 632M(base)+268M(super-res) model achieves FID 6.06 on CC3M.Muse also generates images better aligned with prompts 2.7x more often than Stable Diffusion v1.4 in PartiPrompts human evaluation.Source: §1 · p. 3

Semantic understanding

• The model demonstrates understanding of language and multi-object properties including nouns, verbs, adjectives, compositionality, and cardinality.These capabilities are illustrated through generated examples and qualitative evaluations.Source: Figure 1 · p. 2, §1 · p. 3

Editing

• Muse supports zero-shot text-guided inpainting, outpainting, and mask-free editing without fine-tuning.The mask-based training procedure enables these editing capabilities using real input images.Source: Figure 2 · p. 3, §1 · p. 4

Model approach

• Muse represents images with discrete VQGAN tokens and predicts randomly masked tokens using base and super-resolution Transformer models.The base model operates on lower-resolution tokens, after which a super-resolution model predicts higher-resolution masked tokens.Source: §1 · pp. 1–2, Figure 3 · p. 4

2. Model

• Muse conditions image generation on embeddings from a pre-trained T5-XXL language model that represents objects, actions, visual properties, spatial relationships, and cardinality.The T5-XXL encoder is pre-trained and frozen.Source: §2.1 · p. 4

2.2. Semantic Tokenization using VQGAN

• Muse uses VQGAN models to encode images into discrete semantic tokens and decode token sequences back into images.The base model operates on 16 × 16 tokens for 256 × 256 images, while the super-resolution model operates on 64 × 64 tokens for 512 × 512 images.Source: §2.2 · p. 5

2.3. Base Model

• The base model is a masked Transformer that predicts randomly masked image tokens from projected T5 embeddings and unmasked image tokens.Training applies cross-entropy loss to logits for masked tokens, while inference predicts masks iteratively.Source: §2.3 · p. 5

2.4. Super-Resolution Model

• The super-resolution model processes low-resolution tokens with self-attention, combines them with text embeddings, and uses cross-attention to predict masked high-resolution tokens.The resulting high-resolution tokens are decoded through the higher-resolution VQGAN.Source: Figure 4 · p. 5, §2.4 · p. 6
• The architecture cascades a base model generating a 16 × 16 latent map with a super-resolution model that upsamples it to a 64 × 64 latent map.This cascade addresses the tendency of direct 512 × 512 prediction to focus on low-level details over large-scale semantics.Source: §2.4 · pp. 5–6

2.5. Decoder Finetuning

• Muse increases VQGAN decoder capacity with additional residual layers and channels, then fine-tunes only the new decoder layers.The encoder capacity, encoder weights, codebook, base model, and super-resolution model remain fixed.Source: §2.5 · p. 6

2.6. Masking Schedule

• Muse trains with a variable masking rate sampled from a truncated arccos distribution, whose expected masking rate is 0.64.The distribution is biased toward higher masking rates, making prediction harder.Source: §2.6 · p. 6
• Random masking trains models to learn P(x_i|x_Λ) for arbitrary subsets of tokens rather than autoregressive P(x_i|x_<i).This conditioning structure is described as critical for parallel sampling.Source: §2.6 · p. 6

2.7. Classifier Free Guidance

• Classifier-free guidance forms final logits by moving from unconditional logits toward conditional logits using guidance scale t.The model removes text conditioning on 10% of training samples and computes conditional and unconditional logits during inference.Source: §2.7 · p. 6
• Muse increases guidance scale through sampling so early tokens are sampled more freely and later tokens receive stronger prompt influence.The paper describes classifier-free guidance as trading diversity for fidelity.Source: §2.7 · p. 6

2.8. Iterative Parallel Decoding at Inference

• Parallel decoding predicts multiple output tokens in one forward pass, using a cosine schedule to unmask the highest-confidence tokens iteratively.Muse performs inference with 24 steps for 256 base-model tokens and 8 steps for 4096 super-resolution tokens.Source: §2.8 · p. 6
• The super-resolution model requires significantly fewer sampling steps for convergence because it is conditioned on low-resolution tokens.Figure 5 visualizes masked-token evolution for the base and super-resolution models.Source: §2.8 · p. 6, Figure 5 · p. 7
• The comparison with faster diffusion methods remains future work because those methods have not been widely validated in large-scale text-to-image generation.The paper specifically mentions progressive distillation and better ODE solvers.Source: §2.8 · p. 7

3.1. Qualitative Performance

• Muse reflects cardinality, composition, style, text rendering, and full-prompt information in generated images.Examples include contextual variation among repeated objects rather than simply duplicating identical pixels.Source: §3.1 · p. 7, Figure 6 · p. 8
• Muse is qualitatively comparable to Imagen and better than DALL-E2 for many selected prompts.Source: Figure 7 · p. 9, §3.1 · p. 10
• Muse struggles to render long phrases, high object cardinalities, and multiple cardinalities accurately.Cardinality accuracy decreases as the requested cardinality increases, and multi-cardinality prompts often contain at least one incorrect count.Source: §3.1 · p. 10

3.2.1. HUMAN EVALUATION

• Human realism comparisons require caution because mode collapse can outperform prompt-responsive models on realism-only judgments.The authors therefore restrict this question to models with similar alignment and omit realism ratings against Stable Diffusion.Source: §3.2.1 · p. 12

3.2. Quantitative Performance

• Muse is more than 10x faster than Imagen-3B or Parti on comparable TPU-v4 hardware.Discrete tokens and fewer sampling iterations drive the advantage over diffusion, while parallel decoding drives the advantage over Parti.Source: §3.2.1 · p. 12, Table 3 · p. 12

3.3. Image Editing

• Muse supports out-of-the-box zero-shot inpainting, outpainting, and mask-free editing without additional training or fine-tuning.Mask-free editing operates directly on tokenized images without inverting the full generative process.Source: §3.3 · p. 12, §3.3.2 · p. 12

4.2. Image Tokenizers

• Image tokenizers shift computation from pixel space to latents or enable classification-style losses for generative modeling.Source: §4.2 · p. 13

4.3. Large Language Models

• Muse uses a pre-trained T5 language model because its embeddings encode objects, actions, visual properties, spatial relationships, cardinality, and composition.Source: §4.3 · pp. 13, 15

4.5. Image Editing

• Muse’s mask-based token modeling enables zero-shot inpainting, outpainting, and mask-free editing while preserving pose and overall structure during targeted changes.Source: Figure 11 · p. 14

4.4. Text-Image Models

• Paired text-image data supports representation learning and text-conditioned image generation, with classifier-free guidance trading off diversity and quality.Source: §4.4 · p. 15

4.5. Image Editing

• Image editing has been explored with GANs and diffusion models, including methods requiring fine-tuning or inversion.Source: §4.5 · p. 15

• Muse uses a Transformer architecture that performs comparably to diffusion models while being more efficient at inference time.Source: §5 · p. 15
• Muse supports zero-shot inpainting, outpainting, and mask-free image editing.Source: §5 · p. 15
• The authors identify misinformation, harassment, and social and cultural biases as risks of generative models and therefore do not release code or a public demo.Source: §5 · p. 15
• Automatically curated large datasets may contain consent issues and negative social stereotypes, potentially amplifying bias during training.Source: §5 · p. 15
• The authors caution against using text-to-image models without attention to use cases and potential harm, especially for generating people, humans, and faces.Source: §5 · p. 15

• The largest base model configuration contains 3B parameters.Source: §A.1 · p. 21
• Table 4 presents the configuration and training hyperparameters for the base model.Source: Table 4 · p. 21

• Table 5 presents configuration and training hyperparameters for VQGAN.Source: Table 5 · p. 21
• The VQGAN architecture uses residual, downsampling, and upsampling blocks, with a fully convolutional encoder and decoder supporting different image sizes.Source: §A.2 · p. 21
• Fine-tuning the decoder better preserves details such as house numbers, storefront signs, and window bars than the VQGAN reconstruction.Source: Figure 13 · p. 21

• Table 6 presents configuration and training hyperparameters for the Super-Resolution Model.Source: Table 6 · p. 22