Source-linked AI summary
Muse: Text-To-Image Generation via Masked Generative Transformers
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, Dilip Krishnan
TL;DR
Text-to-image generation requires models that combine high fidelity and semantic alignment with practical inference efficiency. Muse addresses this with masked modeling over discrete image tokens conditioned on a pre-trained LLM, achieving strong benchmark performance and enabling zero-shot editing, although alignment degrades for long rendered phrases and high object cardinalities.
Problem
The paper addresses the need for text-to-image models that provide high-quality, semantically aligned images while being more efficient than diffusion and autoregressive approaches.
Method
Muse predicts randomly masked discrete image tokens with Transformer models conditioned on embeddings from a pre-trained T5-XXL language model, using parallel decoding for inference.
Results
Muse achieves strong image-generation quality and alignment while improving efficiency, including FID 6.06 on CC3M, FID 7.88 and CLIP 0.32 on zero-shot COCO, and 2.7x better prompt alignment than Stable Diffusion v1.4.
Takeaways & Limitations
Muse combines non-diffusion, non-autoregressive Transformer generation with efficient inference and out-of-the-box zero-shot image editing.
Takeaways & Limitations
Muse has difficulty rendering long multi-word phrases and accurately representing prompts with high object cardinality, with cardinality accuracy decreasing as requested cardinality increases.
Abstract
from arXiv · showhide
We present Muse, a text-to-image Transformer model that achieves state-of-the-art image generation performance while being significantly more efficient than diffusion or autoregressive models. Muse is trained on a masked modeling task in discrete token space: given the text embedding extracted from a pre-trained large language model (LLM), Muse is trained to predict randomly masked image tokens. Compared to pixel-space diffusion models, such as Imagen and DALL-E 2, Muse is significantly more efficient due to the use of discrete tokens and requiring fewer sampling iterations; compared to autoregressive models, such as Parti, Muse is more efficient due to the use of parallel decoding. The use of a pre-trained LLM enables fine-grained language understanding, translating to high-fidelity image generation and the understanding of visual concepts such as objects, their spatial relationships, pose, cardinality etc. Our 900M parameter model achieves a new SOTA on CC3M, with an FID score of 6.06. The Muse 3B parameter model achieves an FID of 7.88 on zero-shot COCO evaluation, along with a CLIP score of 0.32. Muse also directly enables a number of image editing applications without the need to fine-tune or invert the model: inpainting, outpainting, and mask-free editing. More results are available at https://muse-model.github.io
1. Introduction
Muse is a masked-image Transformer for text-to-image generation that combines discrete image tokens with pre-trained language-model conditioning. It targets efficient generation while supporting strong image quality, semantic understanding, and zero-shot editing.
- Model approach: Muse uses a pre-trained T5-XXL text encoder to condition a masked image-modeling approach for text-to-image synthesis.The model's image decoder is conditioned on frozen T5-XXL embeddings, while the image-generation models use Transformer architectures.
- Efficiency: More than 10x faster inference than Imagen-3B or Parti-3B and 3x faster than Stable Diffusion v1.4 is estimated on comparable TPU-v4 hardware.These comparisons use images of the same size, either 256 × 256 or 512 × 512.
- Results: A 3B-parameter Muse model achieves a CLIP score of 0.32 and FID score of 7.88 on zero-shot COCO, while a 632M(base)+268M(super-res) model achieves FID 6.06 on CC3M.Muse also generates images better aligned with prompts 2.7x more often than Stable Diffusion v1.4 in PartiPrompts human evaluation.
- Semantic understanding: The model demonstrates understanding of language and multi-object properties including nouns, verbs, adjectives, compositionality, and cardinality.These capabilities are illustrated through generated examples and qualitative evaluations.
- Editing: Muse supports zero-shot text-guided inpainting, outpainting, and mask-free editing without fine-tuning.The mask-based training procedure enables these editing capabilities using real input images.
- Model approach: Muse represents images with discrete VQGAN tokens and predicts randomly masked tokens using base and super-resolution Transformer models.The base model operates on lower-resolution tokens, after which a super-resolution model predicts higher-resolution masked tokens.
2. Model
Muse combines semantic VQGAN tokens, T5-XXL text embeddings, and masked Transformer models to generate images through a base model and super-resolution cascade.
- 2. Model: Muse conditions image generation on embeddings from a pre-trained T5-XXL language model that represents objects, actions, visual properties, spatial relationships, and cardinality.The T5-XXL encoder is pre-trained and frozen.
- 2.2. Semantic Tokenization using VQGAN: Muse uses VQGAN models to encode images into discrete semantic tokens and decode token sequences back into images.The base model operates on 16 × 16 tokens for 256 × 256 images, while the super-resolution model operates on 64 × 64 tokens for 512 × 512 images.
- 2.3. Base Model: The base model is a masked Transformer that predicts randomly masked image tokens from projected T5 embeddings and unmasked image tokens.Training applies cross-entropy loss to logits for masked tokens, while inference predicts masks iteratively.
- 2.4. Super-Resolution Model: The super-resolution model processes low-resolution tokens with self-attention, combines them with text embeddings, and uses cross-attention to predict masked high-resolution tokens.The resulting high-resolution tokens are decoded through the higher-resolution VQGAN.
- 2.4. Super-Resolution Model: The architecture cascades a base model generating a 16 × 16 latent map with a super-resolution model that upsamples it to a 64 × 64 latent map.This cascade addresses the tendency of direct 512 × 512 prediction to focus on low-level details over large-scale semantics.
2.5. Decoder Finetuning
The decoder is enhanced for finer visual details while the trained encoder, codebook, and Transformer models remain fixed; variable masking supports arbitrary token conditioning and parallel sampling.
- 2.5. Decoder Finetuning: Muse increases VQGAN decoder capacity with additional residual layers and channels, then fine-tunes only the new decoder layers.The encoder capacity, encoder weights, codebook, base model, and super-resolution model remain fixed.
- 2.6. Masking Schedule: Muse trains with a variable masking rate sampled from a truncated arccos distribution, whose expected masking rate is 0.64.The distribution is biased toward higher masking rates, making prediction harder.
- 2.6. Masking Schedule: Random masking trains models to learn P(x_i|x_Λ) for arbitrary subsets of tokens rather than autoregressive P(x_i|x_<i).This conditioning structure is described as critical for parallel sampling.
2.7. Classifier Free Guidance
Classifier-free guidance combines conditional and unconditional logits to improve generation quality and text-image alignment, while iterative parallel decoding reduces sampling steps.
- 2.7. Classifier Free Guidance: Classifier-free guidance forms final logits by moving from unconditional logits toward conditional logits using guidance scale t.The model removes text conditioning on 10% of training samples and computes conditional and unconditional logits during inference.
- 2.7. Classifier Free Guidance: Muse increases guidance scale through sampling so early tokens are sampled more freely and later tokens receive stronger prompt influence.The paper describes classifier-free guidance as trading diversity for fidelity.
- 2.8. Iterative Parallel Decoding at Inference: Parallel decoding predicts multiple output tokens in one forward pass, using a cosine schedule to unmask the highest-confidence tokens iteratively.Muse performs inference with 24 steps for 256 base-model tokens and 8 steps for 4096 super-resolution tokens.
- 2.8. Iterative Parallel Decoding at Inference: The super-resolution model requires significantly fewer sampling steps for convergence because it is conditioned on low-resolution tokens.Figure 5 visualizes masked-token evolution for the base and super-resolution models.
- 2.8. Iterative Parallel Decoding at Inference: The comparison with faster diffusion methods remains future work because those methods have not been widely validated in large-scale text-to-image generation.The paper specifically mentions progressive distillation and better ODE solvers.
3. Results
Muse demonstrates strong text-to-image quality and prompt understanding while offering efficient inference and zero-shot image editing. Its evaluations also identify failures on long rendered phrases and increasingly complex cardinality prompts.
- 3.1. Qualitative Performance: Muse reflects cardinality, composition, style, text rendering, and full-prompt information in generated images.Examples include contextual variation among repeated objects rather than simply duplicating identical pixels.
- 3.1. Qualitative Performance: Muse is qualitatively comparable to Imagen and better than DALL-E2 for many selected prompts.
- 3.1. Qualitative Performance: Muse struggles to render long phrases, high object cardinalities, and multiple cardinalities accurately.Cardinality accuracy decreases as the requested cardinality increases, and multi-cardinality prompts often contain at least one incorrect count.
- 3.2.1. HUMAN EVALUATION: Human realism comparisons require caution because mode collapse can outperform prompt-responsive models on realism-only judgments.The authors therefore restrict this question to models with similar alignment and omit realism ratings against Stable Diffusion.
- 3.2. Quantitative Performance: Muse is more than 10x faster than Imagen-3B or Parti on comparable TPU-v4 hardware.Discrete tokens and fewer sampling iterations drive the advantage over diffusion, while parallel decoding drives the advantage over Parti.
- 3.3. Image Editing: Muse supports out-of-the-box zero-shot inpainting, outpainting, and mask-free editing without additional training or fine-tuning.Mask-free editing operates directly on tokenized images without inverting the full generative process.
4. Related Work
Muse builds on latent image tokenization, large language model representations, and paired text-image learning to generate and edit images. Its masked-token formulation connects visual-token prediction with flexible editing operations.
- 4.2. Image Tokenizers: Image tokenizers shift computation from pixel space to latents or enable classification-style losses for generative modeling.
- 4.3. Large Language Models: Muse uses a pre-trained T5 language model because its embeddings encode objects, actions, visual properties, spatial relationships, cardinality, and composition.
- 4.5. Image Editing: Muse’s mask-based token modeling enables zero-shot inpainting, outpainting, and mask-free editing while preserving pose and overall structure during targeted changes.
- 4.4. Text-Image Models: Paired text-image data supports representation learning and text-conditioned image generation, with classifier-free guidance trading off diversity and quality.
- 4.5. Image Editing: Image editing has been explored with GANs and diffusion models, including methods requiring fine-tuning or inversion.
5. Discussion and Social Impact
Muse is presented as an efficient, flexible text-to-image model whose pretrained language conditioning supports strong text-image alignment and image editing. The authors also emphasize societal risks, dataset bias, and the need for careful use.
- Muse uses a Transformer architecture that performs comparably to diffusion models while being more efficient at inference time.
- Muse supports zero-shot inpainting, outpainting, and mask-free image editing.
- The authors identify misinformation, harassment, and social and cultural biases as risks of generative models and therefore do not release code or a public demo.
- Automatically curated large datasets may contain consent issues and negative social stereotypes, potentially amplifying bias during training.
- The authors caution against using text-to-image models without attention to use cases and potential harm, especially for generating people, humans, and faces.
A.1. Base Model Configurations
This section identifies the largest Muse base model configuration and its associated training-hyperparameter table.
- The largest base model configuration contains 3B parameters.
- Table 4 presents the configuration and training hyperparameters for the base model.
A.2. VQGAN Configurations
This section covers VQGAN configuration information and illustrates how decoder fine-tuning improves reconstruction of fine visual details.
- Table 5 presents configuration and training hyperparameters for VQGAN.
- The VQGAN architecture uses residual, downsampling, and upsampling blocks, with a fully convolutional encoder and decoder supporting different image sizes.
- Fine-tuning the decoder better preserves details such as house numbers, storefront signs, and window bars than the VQGAN reconstruction.
A.3. Super Resolution Configurations
This section presents the configuration and training-hyperparameter information for the super-resolution model.
- Table 6 presents configuration and training hyperparameters for the Super-Resolution Model.