PaperlayerView exact revision on arXiv

Source-linked AI summary

Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining

Dongyang Liu, Shitian Zhao, Le Zhuo, Weifeng Lin, Yi Xin, Xinyue Li, Qi Qin, Yu Qiao, Hongsheng Li, Peng Gao

arXiv:2408.02657v3cs.CV

TL;DR

Lumina-mGPT addresses limitations in multimodal autoregressive image generation, including quality, resolution flexibility, and restricted task coverage. It combines decoder-only multimodal pretraining with Uni-Rep, FP-SFT, and Omni-SFT, achieving photorealistic image generation on par with modern diffusion models while supporting a broad range of multimodal tasks. The paper presents this combination as a route toward a unified multimodal generalist.

  • Problem

    Earlier multimodal autoregressive models had limited image quality or resolution flexibility and often lacked broader vision-centric capabilities.

  • Method

    Lumina-mGPT uses a decoder-only transformer initialized with multimodal generative pretraining and fine-tuned with Uni-Rep, FP-SFT, and Omni-SFT.

  • Results

    Lumina-mGPT achieves photorealistic image-generation performance on par with modern diffusion models while handling a wide range of vision and language tasks.

  • Takeaways & Limitations

    The model demonstrates that a decoder-only multimodal autoregressive framework can combine flexible image generation with unified multimodal task handling.

  • Takeaways & Limitations

    The qualitative comparison indicates room for further progress relative to diffusion-based counterparts.

Abstract

from arXiv · show

We present Lumina-mGPT, a family of multimodal autoregressive models capable of various vision and language tasks, particularly excelling in generating flexible photorealistic images from text descriptions. By initializing from multimodal Generative PreTraining (mGPT), we demonstrate that decoder-only Autoregressive (AR) model can achieve image generation performance comparable to modern diffusion models with high efficiency through Flexible Progressive Supervised Fine-tuning (FP-SFT). Equipped with our proposed Unambiguous image Representation (UniRep), Lumina-mGPT can flexibly generate high-quality images of varying aspect ratios. Building on the strong image generation capabilities, we further explore Ominiponent Supervised Fine-tuning (Omni-SFT), an initial attempt to elevate Lumina-mGPT into a unified multi-modal generalist. The resulting model demonstrates versatile multimodal capabilities, including visual generation tasks like text-to-image/multiview generation and controllable generation, visual recognition tasks like segmentation and depth estimation, and vision-language tasks like multi-turn visual question answering, showing the rosy potential of the technical direction. Codes and checkpoints are available at https://github.com/Alpha-VLLM/Lumina-mGPT.

1. Introduction

Lumina-mGPT addresses limitations of earlier multimodal autoregressive models with a decoder-only architecture initialized by multimodal generative pretraining. Its fine-tuning strategies target flexible photorealistic image generation and unified multimodal task handling.

  • Motivation: Earlier autoregressive models often had unsatisfactory image generation, complex architectures, or restricted decoding capabilities.These limitations constrained image quality, scalability, generalization, or flexibility.
  • Approach: Lumina-mGPT uses a decoder-only transformer initialized from multimodal Generative PreTraining and progressively fine-tuned on flexible, high-quality, high-resolution discrete image tokens.The approach combines mGPT initialization with progressive supervised fine-tuning.
  • Approach: Flexible Progressive Supervised Finetuning transitions from low-resolution to high-resolution discrete tokens, while Uni-Rep supports photorealistic images with flexible aspect ratios.The weak-to-strong strategy is designed to improve high-quality text-to-image generation.
  • Unified tasks: Omnipotent Supervised Finetuning treats multimodal tasks as unified discrete modeling through a natural-language interface.Tasks include dialogue, visual understanding, dense labeling, text-to-image and multiview generation, editing, and spatial-conditional generation.
  • Results: Multimodal generative pretraining provides a representation that accelerates learning for text-to-image generation and downstream vision-language tasks.The initialization is taken from pretrained Chameleon 7B and 30B models.
  • Results: Lumina-mGPT demonstrates image generation performance on par with modern diffusion models and requires 32 A100 GPUs over 7 days to train a 7B model.The contribution summary presents this as an efficient decoder-only autoregressive result.

2. Methodolgy

The method unifies text and images as discrete token sequences in a decoder-only transformer, then uses Uni-Rep, FP-SFT, and Omni-SFT to support flexible image generation and multimodal tasks. Uni-Rep removes ambiguity in variable-resolution image representations, while progressive fine-tuning expands resolution and task coverage.

  • Multimodal pretraining: mGPT uses a decoder-only transformer pretrained on extensive multimodal token sequences and models multimodal sequences with next-token prediction.Text and images are tokenized into a unified discrete sequence.
  • Uni-Rep: Variable-resolution images can share the same number of flattened tokens, making their original shapes ambiguous and complicating image perception and generation.For example, 512 × 512, 256 × 1024, and 1024 × 256 images can map to the same token count.
  • Uni-Rep: Uni-Rep adds height and width indicators after the image-start token and end-of-line tokens after each image row, allowing the original image shape to be parsed from the sequence.The indicators pre-determine image shape during generation, while end-of-line tokens provide explicit spatial anchors.
  • FP-SFT: FP-SFT progressively trains across stages whose approximate image areas are 512^2, 768^2, and 1024^2, using candidate resolutions with different aspect ratios.The high-resolution stage benefits from earlier high-throughput pretraining and low-resolution fine-tuning, improving data efficiency.
  • Omni-SFT: Omni-SFT transfers the image-generation model to image editing, dense prediction, spatial conditioning, multiview generation, and other tasks under a unified next-token objective.The training data includes editing, surface normal and depth estimation, pose estimation, segmentation, grounding, and multiview generation.
  • Omni-SFT: After Omni-SFT, Lumina-mGPT exhibits general capability across tasks beyond text-to-image generation, including sequences of image frames for preliminary video generation.The authors describe this as evidence of potential for a multimodal generalist.

3. Experiments

Lumina-mGPT combines strong photorealistic image generation with broad multimodal task coverage. Experiments show improved visual quality, flexible resolutions and aspect ratios, progressive detail gains, and attention patterns consistent with image structure.

  • Photorealistic Text-to-Image Generation: Lumina-mGPT achieves better visual quality and aesthetics than LlamaGen and Parti while supporting end-to-end generation at high resolutions and arbitrary aspect ratios.LlamaGen supports only 512 × 512, while Parti uses an additional super-resolution upsampler for 1024 × 1024 images.
  • Photorealistic Text-to-Image Generation: Lumina-mGPT significantly improves over Chameleon, surpasses Lumina-Next, and consistently outperforms SDv1.5 and SDv2.1, reaching competitive quality to SDXL.A remaining gap exists relative to diffusion models such as SD3 and DALL-E3.
  • Photorealistic Text-to-Image Generation: FP-SFT progressively reduces VQ-VAE artifacts and reveals increasingly diverse fine-grained details as image resolution rises.The visualizations support progressive high-quality image generation from the mGPT initialization.
  • Omnipotent Task Unification with Lumina-mGPT: After Omni-SFT, Lumina-mGPT supports text-only dialog, image understanding, editing, dense labeling, spatially conditioned synthesis, and multiview generation.The reported examples include captioning, visual question answering, segmentation, pose estimation, depth estimation, surface-normal estimation, and referring-object detection.
  • Omnipotent Task Unification with Lumina-mGPT: Lumina-mGPT generates images under depth, segmentation, normal-map, and human-pose conditions, and produces consistent multiview image sequences.Rendering the multiview images into a coherent video is described as a preliminary step toward video generation.
  • Omnipotent Task Unification with Lumina-mGPT: Attention concentrates on local tokens and assigns exceptionally high scores to indicator tokens during image generation.The <start-of-image> token is associated with semantic information, while <end-of-line> tokens introduce 2-D positional information into 1-D RoPE embeddings.

4. Conclusion

Lumina-mGPT presents a decoder-only multimodal autoregressive approach for flexible photorealistic image generation and unified vision-language tasks. Its reported performance is on par with modern diffusion models while exceeding existing autoregressive models in visual quality and detail.

  • Conclusion: Lumina-mGPT combines Uni-Rep, FP-SFT, and Omni-SFT to generate flexible photorealistic images and unify vision and language tasks.The model supports image generation alongside tasks such as visual question answering.
  • Conclusion: Lumina-mGPT achieves performance on par with modern diffusion models and surpasses LlamaGen and Parti in visual quality and detail.The paper also reports versatility across text-to-image synthesis and visual question answering.

A. Additional Qualitative Results

Additional qualitative results depict Lumina-mGPT across general multimodal tasks, reconstruction, image editing, spatial conditioning, and multiview generation. The examples illustrate the breadth of its unified task interface.

  • General Multimodal Tasks: The qualitative results present Lumina-mGPT as a generalist across various multimodal tasks.Several figures are dedicated to visualizing this broad task coverage.
  • Controllable and Multiview Generation: The examples include segmentation-, surface-normal-, pose-conditioned, and multiview image generation.The prompts demonstrate image generation from structured visual conditions and multiview instructions.
  • Related Multimodal Settings: Lumina-mGPT is illustrated in relation to multimodal large language models and text-to-image generation.The supplied qualitative materials also reference reconstruction quality and VQ-VAE-based image processing.

C. Discussions on Reconstruction Quality

The discussion identifies reconstruction fidelity, architectural complexity, and decoding flexibility as important constraints for multimodal autoregressive image generation. It frames Lumina-mGPT against these limitations without eliminating the underlying reconstruction-quality boundary.

  • Reconstruction Quality: VQ-VAE compression loses information and degrades reconstruction quality, especially for high-frequency edges, hair, and text.Because Lumina-mGPT trains on VQ-VAE latents rather than original images, reconstruction quality can bound generation quality.
  • Architecture: Encoder-decoder designs complicate autoregressive decoding, limit image-generation scalability, and hinder extension to additional modalities and tasks.The discussion contrasts these designs with simpler decoder-only architectures.
  • Resolution and Flexibility: Existing autoregressive methods rely on cropped low-resolution 512 × 512 images, limiting resolution and aspect-ratio flexibility compared with advanced diffusion models.Natural images occur at varied resolutions and aspect ratios, creating a decoding-flexibility challenge.
  • Task Extensibility: Autoregressive modeling offers unified generative modeling across tasks and modalities, but prior systems show poor task extensibility.The supplied discussion positions task unification as an unrealized strength of autoregressive modeling.
  • Failure Cases: Figure 13 presents failure cases of the current Lumina-mGPT.The supplied figure caption does not specify the failure modes.

E. Limitations of Lumina-mGPT

Lumina-mGPT remains constrained by visual artifacts, slower high-resolution inference, and image-tokenizer reconstruction quality. These limitations are linked to limited training resources, autoregressive decoding, and the Chameleon VQ-VAE.

  • Lumina-mGPT sometimes generates noticeable visual artifacts, including unreasonable limbs in people and animals.Its prompt-following is also inferior to several large-scale text-to-image systems, while dense labeling and controllable generation remain preliminary.
  • High-resolution generation can take minutes because autoregressive decoding requires numerous network evaluations for long image-token sequences.This is significantly slower than diffusion models using advanced samplers.
  • Inference speed may improve through techniques such as vLLM and FlashAttention.
  • The Chameleon VQ-VAE can struggle to reconstruct high-frequency details, especially text and human faces, limiting generation quality.The authors suggest improved tokenizers such as FSQ as a potential route to better quality.

F. Inference Configuration of Lumina-mGPT

Lumina-mGPT’s decoding configuration strongly affects image quality, texture, style, and diversity. Classifier-free guidance initially improves quality and stability, while temperature and top-k regulate detail and diversity with trade-offs at extreme settings.

  • Classifier-Free Guidance: Classifier-free guidance initially improves generation quality and stability, but excessive guidance deteriorates quality.Lumina-mGPT applies CFG to autoregressive logits using conditioned and context-independent predictions.
  • Random Seeds: Different random seeds produce comparable aesthetics with greater diversity for Lumina-mGPT than Lumina-Next-SFT.
  • Influence of Top-k: Lower top-k produces simpler, over-smoothed content, while higher top-k yields more diverse image detail and texture.The explored top-k range is 50 to 8192, with 8192 equal to the VQ-VAE codebook size.

G. Comparison with Diffusion-based Appraoches

Under matched prompts and training conditions, Lumina-mGPT achieves photorealistic aesthetics comparable to diffusion-based approaches while producing more diverse outputs. It also renders text better, but multilingual and emoji understanding remain weaker, and its unified token-sequence design supports multiple modalities and tasks.

  • Similarity between Diffusion- and AR-based Generation: With the same prompts, training data, budget, and comparable model sizes, AR and diffusion approaches achieve similar photorealistic aesthetics and fine-grained details.The comparison challenges the view that diffusion models are necessarily more effective for generative modeling.
  • Differences between Diffusion- and AR-based Generation: Lumina-mGPT produces more diverse outputs across random seeds, but excessive diversity makes it less stable and more prone to visual artifacts.
  • Text Rendering: Lumina-mGPT renders text substantially better than Lumina-Next-SFT, which struggles to generate complete characters.The authors associate this with multimodal pretraining on massive interleaved text-image data.
  • Multilingual Understanding: Lumina-mGPT performs worse than Lumina-Next-SFT on emoji and Chinese prompts because its pretraining used less multilingual text data.
  • Unified Multimodal Framework: Lumina-mGPT represents inputs from all modalities as multimodal token sequences within a unified framework for vision and language tasks.
Loading 2408.02657v3…