Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining

• The paper identifies prior AR image-generation approaches as limited in image quality or benchmark scope, architectural scalability, resolution flexibility, and extension to broader tasks.Source: §1 · p. 1
• Its simple decoder-only transformer offers an extensible next-token framework for unifying multimodal understanding and generation.The model is initialized with multimodal generative pretraining and can draw on LLM training and inference techniques.Source: §1 · pp. 1, 3
• FP-SFT progressively moves from low- to high-resolution discrete tokens, while UniRep enables high-quality photorealistic images with flexible aspect ratios.Source: §1 · p. 3
• Omni-SFT treats dialogue, visual understanding, dense labeling, image generation, editing, and spatial-conditional generation as unified discrete modeling tasks through a natural-language interface.Source: §1 · p. 3
• Lumina-mGPT uses scaled multimodal next-token pretraining from Chameleon 7B and 30B, whose learned representation accelerates text-to-image and downstream vision-language learning.Source: §1 · p. 3
• Lumina-mGPT demonstrates decoder-only AR image generation performance on par with modern diffusion models, training a 7B model with 32 A100 GPUs over 7 days.Source: §1 · p. 3
• The authors release the full-pipeline code implementation to support further research.Source: §1 · p. 3

2.1. Revisiting mGPT with Chameleon

• Chameleon-style mGPT turns text and quantized image patches into one discrete token sequence, which a decoder-only transformer models with next-token prediction.Chameleon applies z-loss for training stability; the paper positions Lumina-mGPT as addressing Chameleon's limited image-generation quality, resolution flexibility, and vision-task scope.Source: §2.1 · p. 4

2.2.1. EFFECTIVE INITLIZATION

• Lumina-mGPT initializes from Chameleon models ranging from 7B to 30B parameters and uses 10M high-quality image-text data points.The paper motivates mGPT initialization by its large-scale pretraining and generalizable representations.Source: §2.2.1 · pp. 4–5

2.2.2. SUPERVISED FINETUNING FOR LUMINA-MGPT

• UniRep encodes image height, width, and row boundaries so image shapes can be parsed from flattened tokens and variable resolutions supported.Height/width indicators precede image tokens to determine the generated shape; end-of-line tokens add explicit spatial anchors.Source: §2.2.2 · p. 5
• FP-SFT progressively trains across three image-area stages, from approximately 512^2 to 768^2 to 1024^2, to support high-resolution images with flexible aspect ratios.Lower-resolution stages favor throughput and broad composition learning, while the high-resolution stage focuses on fine detail and is data-efficient given the earlier foundation.Source: §2.2.2 · p. 5
• After FP-SFT, Omni-SFT transfers Lumina-mGPT to a wider range of image understanding and generation tasks beyond text-to-image.The paper presents this as a preliminary exploration toward a visual generalist, with broader task-completion capability reported after Omni-SFT.Source: §2.2.2 · pp. 5–6
• Omni-SFT covers language-guided image editing, dense prediction, spatially conditioned image generation, and text-conditional multiview generation while retaining text modeling and text-to-image data.The multiview data contains 100k samples with 384^2 images rendered from 16 viewpoints.Source: §2.2.2 · p. 5
• All fine-tuning tasks use next-token prediction on dialog responses, with context randomly dropped 10% of the time.Training uses AdamW with weight decay 0.1 and betas (0.9, 0.95), a 2e-5 learning rate, FSDP with gradient checkpointing, and pre-tokenized, length-clustered batches; z-loss is also applied for stability.Source: §2.2.2 · pp. 5–6

3.1. Photorealistic Text-to-Image Generation

• Compared with LlamaGen and Parti, Lumina-mGPT has better visual quality and aesthetics, supports end-to-end high-resolution and arbitrary-aspect-ratio generation, and FP-SFT progressively improves details as resolution increases.LlamaGen supports only 512 × 512, while Parti uses a super-resolution upsampler for 1024 × 1024; FP-SFT stages show decreasing VQ-VAE artifacts and emerging fine-grained details.Source: §3.1 · pp. 6–7, Figure 4 · p. 8
• A gap remains between Lumina-mGPT and diffusion models SD3 (Esser et al., 2024) and DALL-E3.The paper identifies this as room for further progress.Source: §3.1 · pp. 6–7
• Lumina-mGPT improves significantly over Chameleon (Team, 2024a), surpasses same-dataset Lumina-Next (Zhuo et al., 2024), outperforms SDv1.5 and SDv2.1 (Rombach et al., 2022), and is competitive with SDXL (Podell et al., 2023).The comparison is on text-to-image benchmarks; the supplied passages provide no benchmark cell values.Source: §3.1 · p. 6, Table 2 · p. 7
• Text and image generation favor different decoding hyperparameters, motivating status-aware setting switches; lower temperatures and top-k often oversmooth images, while higher values add detail but may introduce artifacts.The paper also explores classifier-free guidance scale and defers configuration details to Appendix F.Source: §3.1 · p. 7

3.2. Omnipotent Task Unification with Lumina-mGPT

• After Omni-SFT, Lumina-mGPT extends beyond text-only dialog and image understanding to multi-turn image editing, dense labeling, spatial-conditional synthesis, and multiview generation.The paper presents this unification as enabled by its high-quality image-generation capability.Source: §3.2 · p. 7
• Lumina-mGPT handles text-only math, coding, and commonsense tasks, plus captioning, visual question answering, and multi-turn multi-image dialogue with flexible aspect ratios; visual-understanding benchmarks show significant gains over original Chameleon.The text-only capability is attributed to Chameleon pretraining and text-only data intended to mitigate catastrophic forgetting.Source: §3.2 · p. 7, Table 3 · p. 7
• Using natural language, Lumina-mGPT performs segmentation, pose and depth estimation, surface-normal estimation, and referring-object detection, and generates images conditioned on depth, segmentation, normal, or pose maps, including consistent multiview sequences.The paper describes multiview generation as a preliminary step toward video generation by rendering the images into a coherent video.Source: §3.2 · pp. 7–8

3.3. Attention Visualization

• During text-to-image generation, attention declines for more distant tokens, while <start-of-image> and <end-of-line> receive exceptionally high attention.The pattern aligns with local image correlations and RoPE’s long-term decay; the indicator tokens are proposed to carry text semantics and 2-D positional information, respectively.Source: Figure 5 · p. 8, §3.3 · p. 8

• The authors present Lumina-mGPT as a decoder-only multimodal autoregressive model that uses Uni-Rep, FP-SFT, and Omni-SFT to support flexible photorealistic text-to-image generation and unified vision-language tasks.Source: §4 · p. 8
• Lumina-mGPT performs on par with modern diffusion models and surpasses LlamaGen and Parti in visual quality and detail.Source: §4 · p. 8
• The model demonstrates multimodal versatility across tasks from text-to-image synthesis to visual question answering.Source: §4 · p. 8
• The authors release Lumina-mGPT's code and checkpoints to foster further research and collaboration.Source: §4 · p. 8

A. Additional Qualitative Results

• Figure 6 presents selected photorealistic images generated by Lumina-mGPT at various resolutions.The figure caption also notes two examples at the bottom, but the supplied text does not specify their content.Source: Figure 6 · p. 12
• Figures 7–11 present Lumina-mGPT as a generalist for various multimodal tasks.Source: Figure 7 · p. 13, Figure 8 · p. 14, Figure 9 · p. 15

Multimodal Large Language Models

• Existing multimodal LLMs mainly support visual understanding, while image-generation extensions rely on external generators and can produce inconsistent, inferior results.Lumina-mGPT aims to combine understanding and generation through native multimodal capabilities, drawing inspiration from Chameleon (Team, 2024a).Source: B. Related Work · p. 18

Text-to-Image Generation

• Diffusion models in pixel and latent space are the de-facto text-to-image approach, but DiTs still use separate text encoders such as CLIP or T5, creating a modality gap.The paper links that gap to inaccurate generation and difficulty developing a unified multimodal generative foundation model.Source: B. Related Work · p. 18
• For reconstruction, images are VQ-VAE-encoded, passed to Lumina-mGPT with an image-editing prompt instructing “no edit,” and decoded from the generated latents.Figure 12 is captioned as comparing reconstruction quality across methods; the supplied passages provide no comparison values.Source: B. Related Work · p. 18, Figure 12 · p. 18
• Earlier autoregressive methods used VQ-VAE tokenization followed by transformer modeling; Parti scaled this approach, and LlamaGen improved tokenization and incorporated LLM techniques.Lumina-mGPT differs by pretrained multimodally on unified text-image sequences and then supervised-finetuned on high-quality text-to-image pairs for flexible, high-aesthetic generation.Source: B. Related Work · p. 18

C. Discussions on Reconstruction Quality

• VQ-VAE information loss degrades reconstructions of high-frequency details, potentially setting an upper bound on image-generation quality for models trained only on its latents.The passage identifies edges, hair, and text as examples of details especially affected by reconstruction degradation.Source: C. Discussions on Reconstruction Quality · p. 18

Verbose Encoder-Decoder Architeture

• Existing approaches’ separate text encoding and image-token modeling complicate autoregressive decoding, limit scalability, and hinder generalization to additional modalities and tasks.The section contrasts this verbose design with the trend toward scaling LLMs (Touvron et al., 2023; Bai et al., 2023; Bi et al., 2024).Source: D. Limitations of Existing Approaches · p. 19

Limited Decoding Resolution and Flexibility

• Current AR approaches crop images to 512 × 512 and encode them as fixed-length VQVAE token sequences, sacrificing image quality and generation flexibility (Yu et al., 2022; Team, 2024a; Sun et al., 2024).The passage contrasts this constraint with diffusion models’ ability to generate photorealistic images at arbitrary resolutions and skewed aspect ratios (Chen et al., 2024a;b; Esser et al., 2024; Team, 2024b; Li et al., 2024; Zhuo et al., 2024; Lu et al., 2024b).Source: D. Limitations of Existing Approaches · p. 19

Poor Task Extensability

• The section flags poor task extensibility despite autoregressive modeling’s capacity to unify tasks and modalities using discrete tokens (Wang et al., 2022; Lu et al., 2022; 2024a).Source: D. Limitations of Existing Approaches · p. 19

Failure Cases

• Lumina-mGPT produces visual artifacts, including implausible human or animal limbs, and trails SD3 (Esser et al., 2024), Kolors (Team, 2024b), and HunyuanDiT (Li et al., 2024) in prompt following.Dense labeling and controllable generation remain preliminary under a limited training budget, with inaccurate predictions or semantically inconsistent images that fail to follow image conditions; the authors expect more data and compute to address these failures.Source: E. Limitations of Lumina-mGPT · p. 19

Generation Speed

• High-resolution autoregressive generation can take minutes because next-token prediction requires numerous network evaluations, making it significantly slower than diffusion models with advanced samplers.The authors suggest integrating vLLM (Kwon et al., 2023) and FlashAttention (Dao et al., 2022; Dao, 2024) to speed inference.Source: E. Limitations of Lumina-mGPT · pp. 19–20

VQ-VAE Reconstruction Quality

• VQ-VAE compression creates an information bottleneck, so its reconstruction quality limits generation quality, particularly for high-frequency details in text and human faces.The Chameleon VQ-VAE sometimes struggles with these details; the authors suggest that improvements such as FSQ (Mentzer et al., 2023) may enhance generation quality.Source: E. Limitations of Lumina-mGPT · p. 20

Classifier-Free Guidance

• Increasing CFG initially improves generation quality and stability, but further increases deteriorate quality.Lumina-mGPT applies CFG (Ho & Salimans, 2022) to autoregressive logits; training randomly drops context before <start-of-image> with 10% probability, and KV caching can accelerate both conditional and context-independent logits.Source: F. Inference Configuration of… · p. 20

Influence of Temperature

• With T varied from 0.7 to 1.0, lower temperature diminishes visual details and over-smooths objects, while higher temperature adds content but risks artifacts.The standard decoding setting is T=1.0, top-k=2000, and CFG=4.0.Source: F. Inference Configuration of… · p. 20, Figure 15 · p. 20

Influence of Top-k

• As top-k rises from 50 to 8192, image detail and texture become more diverse and aesthetically appealing, but artifact risk increases; low top-k yields simpler, over-smoothed content.The standard decoding setting uses top-k=2000; 8192 equals Lumina-mGPT’s VQ-VAE codebook size.Source: Figure 15 · p. 20, F. Inference Configuration of… · p. 21

On the Similarity between Diffusion- and AR-based Generation

• With the same training data and budget and comparable model sizes, Lumina-mGPT achieves photorealistic text-to-image performance and visual aesthetics on par with Lumina-Next-SFT.Both approaches produce images with similar aesthetic style and fine-grained details.Source: G. Comparison with Diffusion-based… · p. 21

On the Differences between Diffusion- and AR-based Generation

• Lumina-mGPT produces more diverse images across random seeds, while Lumina-Next-SFT repeats layouts and textures; excessive diversity makes Lumina-mGPT less stable and more artifact-prone.The authors partly attribute Lumina-mGPT’s diversity to its high temperature and top-k settings.Source: G. Comparison with Diffusion-based… · p. 21
• Lumina-mGPT renders text significantly better than Lumina-Next-SFT, which struggles to generate complete characters, but performs worse on emoji and Chinese prompt understanding.The authors attribute its text-rendering strength to mGPT’s seamless text-image representation learned from massive interleaved data, and its multilingual weakness to limited multilingual pretraining text; Lumina-Next-SFT’s text encoder is stronger on multilingual capabilities than Chameleon.Source: G. Comparison with Diffusion-based… · pp. 21–22, Figure 17 · p. 21, Figure 18 · p. 21
• Lumina-mGPT unifies vision and language tasks by representing modalities as token sequences and using natural language with next-token prediction, whereas diffusion models often need task-specific designs and extra training for unseen tasks.The paper presents adding more multilingual data to the base mGPT model as a way it could benefit downstream tasks.Source: G. Comparison with Diffusion-based… · p. 22