Source-linked AI summary
Show-o: One Single Transformer to Unify Multimodal Understanding and Generation
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, Mike Zheng Shou
TL;DR
Existing multimodal systems commonly separate understanding and generation, while integrating discrete text modeling with visual generation remains challenging. Show-o uses one transformer that combines autoregressive text modeling with discrete diffusion for image tokens, achieving comparable or better performance than individual models across benchmarks and supporting diverse vision-language applications.
Problem
Existing attempts often use separate models for multimodal understanding and image generation, and integrating autoregressive text with diffusion-based visual modeling is non-trivial.
Method
Show-o uses one pretrained-LLM-based transformer with autoregressive text modeling, discrete diffusion for image tokens, tokenizers, and unified prompting.
Results
Show-o achieves comparable or better performance than individual models with equivalent or larger parameter counts across multimodal benchmarks.
Takeaways & Limitations
The unified model supports visual question answering, text-to-image generation, text-guided inpainting and extrapolation without fine-tuning, and mixed-modality generation.
Takeaways & Limitations
The paper discusses Show-o's failure modes in Appendix K.
Abstract
from arXiv · showhide
We present a unified transformer, i.e., Show-o, that unifies multimodal understanding and generation. Unlike fully autoregressive models, Show-o unifies autoregressive and (discrete) diffusion modeling to adaptively handle inputs and outputs of various and mixed modalities. The unified model flexibly supports a wide range of vision-language tasks including visual question-answering, text-to-image generation, text-guided inpainting/extrapolation, and mixed-modality generation. Across various benchmarks, it demonstrates comparable or superior performance to existing individual models with an equivalent or larger number of parameters tailored for understanding or generation. This significantly highlights its potential as a next-generation foundation model. Code and models are released at https://github.com/showlab/Show-o.
1 INTRODUCTION
Show-o addresses the separation between multimodal understanding and generation by combining autoregressive text modeling with discrete diffusion for image tokens in one transformer. It supports diverse vision-language tasks and achieves comparable or better benchmark performance than individual expert models.
- Motivation: Existing unified systems often keep separate understanding and generation models, while autoregressively generating high-resolution images requires many sampling steps.Diffusion models provide an alternative visual-generation approach using full attention.
- Contribution: Show-o unifies multimodal understanding and generation using one single transformer.
- Architecture: Show-o combines autoregressive modeling for text with discrete diffusion modeling for image tokens.It uses discrete tokenizers and a unified prompting strategy to format diverse inputs as sequences.
- Results: Show-o achieves comparable or better performance than individual baseline models with equivalent or larger parameter counts across multimodal benchmarks.
- Applications: Show-o supports text-guided inpainting and extrapolation without fine-tuning, alongside mixed-modality generation, video understanding, and video generation.
- Analysis: The paper studies how dataset scale, image resolution, and discrete versus continuous representations affect multimodal understanding.
2 RELATED WORK
Related work develops multimodal understanding, visual generation, and unified models through distinct architectural choices. These approaches include autoregressive image modeling, continuous diffusion, and systems that interleave or combine modalities.
- Multimodal understanding: Multimodal large language models such as LLaVA, MiniGPT-4, and InstructBLIP established strong multimodal understanding capabilities.
- Autoregressive models: Autoregressive approaches model image or video dependencies directly, extending transformer language-modeling techniques to visual generation.
- Diffusion models: Diffusion-based methods typically denoise continuous latent representations by predicting added Gaussian noise for text-to-image or text-to-video generation.
- Unified models: Unified multimodal models explore interleaving continuous representations with text tokens or using token-based mixed-modal autoregressive modeling.
- Show-o: Figure 2 presents Show-o as a unified system that tokenizes multimodal inputs and supports understanding, generation, and mixed-modality tasks.
3 METHODOLOGY
Show-o unifies autoregressive text modeling and discrete diffusion-style image-token modeling within one transformer. Unified prompting and omni-attention adapt the model to multimodal understanding, generation, and mixed-modality sequences.
- 3.1 TOKENIZATION: Show-o uses a unified discrete-token space containing both text and image tokens, with the pretrained language tokenizer retained for text.Images are encoded into discrete tokens using a lookup-free quantizer with a codebook of size K = 8,192 and a 16×16 token representation for 256×256 inputs.
- 3.1 TOKENIZATION: The model supports alternative continuous image representations from pretrained MAGVIT-v2 and CLIP-ViT encoders for multimodal understanding.The default configuration uses discrete image tokens for both understanding and generation.
- 3.1 TOKENIZATION: Unified prompting formats multimodal inputs as sequential data using task markers and special tokens for text and image boundaries.[MMU] and [T2I] identify tasks, while [SOT]/[EOT] and [SOI]/[EOI] mark text and image spans.
- 3.2 ARCHITECTURE: Omni-attention mixes causal and full attention according to sequence format: text tokens use causal attention, while image tokens use full attention.This lets text attend to preceding image tokens for understanding and image tokens attend to preceding text for generation; text-only inputs reduce to causal attention.
- 3.2 ARCHITECTURE: Show-o combines next-token prediction for text with mask-token prediction for image tokens to unify autoregressive and discrete diffusion modeling.Image tokens are randomly masked and reconstructed from all text and unmasked image tokens; the loss applies only to masked tokens.
- 3.2 ARCHITECTURE: The overall training objective is L = LMTP + αLNTP, and inference autoregressively generates textual answers while iteratively replacing image masks within T steps.The model retains a pretrained LLM architecture with QK-Norm prepended to attention layers and adds 8,192 learnable image-token embeddings.
4 EXPERIMENTS
Experiments evaluate Show-o across multimodal understanding, visual generation, inpainting, extrapolation, mixed-modality generation, video tasks, and ablations. Results show competitive benchmark performance and broad qualitative support, while scaling data and resolution improves understanding.
- Experimental setup: Show-o is evaluated on six multimodal-understanding benchmarks, MSCOCO zero-shot FID, and GenEval text-to-image generation.The understanding benchmarks are POPE, MME, Flickr30k, VQAv2, GQA, and MMMU.
- Multimodal understanding: Show-o’s understanding performance is comparable to the dedicated LLaVA-v1.5-Phi-1.5 baseline across all reported evaluation metrics.The baseline uses the same Phi-1.5 foundation model.
- Visual generation: 9.24 FID: Show-o outperforms GLIDE and DALL·E 2 on zero-shot MSCOCO generation despite using 1.3B parameters and 35M training examples.The comparison concerns generation fidelity on MSCOCO 30K.
- Visual generation: Around 0.24 overall improvement: Show-o outperforms similarly sized LDM on all six GenEval metrics and is comparable to the larger SD3.Show-o also exceeds DALL·E 2, SDXL, and other unified models in the reported comparisons.
- Mixed-modality and video tasks: Show-o supports text-guided inpainting and extrapolation without fine-tuning, mixed text-keyframe generation, and temporally consistent instructional video keyframes.The paper also demonstrates video understanding and generation after adapting video tokenization and fine-tuning.
- Ablation studies: Increasing dataset scale and image resolution consistently improves multimodal understanding by supporting image-text alignment and richer discrete image-token representations.The ablation studies focus on learning discrete image-token embeddings from scratch.
- Ablation studies: More sampling steps improve prompt adherence and fidelity, while classifier-free guidance makes generated colors and contents more diverse and text-consistent.These effects are illustrated qualitatively in the appendix.
5 CONCLUSION
Show-o unifies multimodal understanding and generation by combining autoregressive text modeling with discrete diffusion for image tokens. Its discrete diffusion formulation simplifies image-token corruption and reconstruction into a MaskGIT-style masked-token prediction objective.
- 5 CONCLUSION: Show-o unifies multimodal understanding and generation in one transformer using autoregressive and discrete diffusion modeling.Text is modeled autoregressively, while image tokens are modeled with discrete diffusion.
- 5 CONCLUSION: Discrete diffusion corrupts image tokens through a stochastic transition matrix over categorical tokens and a [MASK] state.The matrix defines transitions among K image-token categories and the additional [MASK] state.
- 5 CONCLUSION: The absorbing-uniform process assigns image tokens probabilities of masking, uniform diffusion, or remaining unchanged during corruption.Masked tokens remain in the [MASK] state in subsequent corruption steps.
- 5 CONCLUSION: The variational diffusion objective is expressed through an evidence lower bound and an alternative lower-bound form involving pθ(x0|xt).The supplied derivation includes the ELBO, constants, and an equivalent lower-bound expression.
- 5 CONCLUSION: Restricting corruption to unchanged or [MASK] states reduces the discrete diffusion objective to the MaskGIT cross-entropy loss for reconstructing masked image regions.The model learns to recover masked regions of x0 from noised xt.
C TRAINING PIPELINE
Show-o is trained in three stages to preserve language reasoning, learn image dependencies and image-text alignment, and refine multimodal capabilities with higher-quality data.
- C TRAINING PIPELINE: Show-o requires large-scale pre-training because its image-token embeddings are newly initialized and it removes the text encoder.The training pipeline addresses alignment between text and image content within one transformer.
- C TRAINING PIPELINE: Stage one uses RefinedWeb for language modeling, ImageNet-1K for class-conditional image generation, and image-text pairs for image captioning.This stage primarily establishes image-token embedding and pixel-dependency learning.
- C TRAINING PIPELINE: Stage two trains text-to-image generation on image-text data to align images and text for captioning and generation.It builds on the pre-trained weights from the first stage.
- C TRAINING PIPELINE: Stage three fine-tunes Show-o with filtered high-quality image-text pairs and instructional data for understanding and mixed-modality generation.This stage further refines the pre-trained model.
D INFERENCE DETAILS
Show-o uses autoregressive sampling for text in multimodal understanding and iterative masked-token prediction for visual generation. Generated image tokens are repeatedly re-masked by confidence until decoding produces the final image.
- D INFERENCE DETAILS: In multimodal understanding, Show-o autoregressively samples text tokens conditioned on images and questions.Sampling selects predicted tokens with higher confidence.
- D INFERENCE DETAILS: For visual generation, Show-o starts with text tokens and [MASK] tokens, then predicts logits for the masked image tokens.The image-token logits are computed at each diffusion time step.
- D INFERENCE DETAILS: Each masked position is sampled from the predicted codebook distribution, and its score is used as token confidence.Unmasked tokens receive confidence 1.0.
- D INFERENCE DETAILS: A mask schedule determines how many image tokens are re-masked, with m = ⌈γ(t/T)M⌉.Predicted tokens below the confidence threshold are replaced with [MASK] tokens for the next round.
- D INFERENCE DETAILS: The remaining image tokens and re-masked positions are fed back into Show-o until time step T, after which the image tokenizer decodes the finalized tokens.This creates an iterative refinement process for visual generation.
E DATASET DETAILS
Show-o training combines text-only data, class-labeled image data, and image-text pairs to support language reasoning, image generation, and multimodal alignment.
- E DATASET DETAILS: Text-only training uses RefinedWeb, comprising approximately 1 billion instances and 2.8 terabytes of curated text data.The dataset contains 968 million individual web pages.
- E DATASET DETAILS: Image training uses 1.28M ImageNet-1K images with class names to teach image-token dependencies and class-conditional generation.Class names are used as textual inputs.
F IMPLEMENTATION DETAILS
Show-o is trained through staged multimodal pretraining, text-to-image training, higher-resolution continuation, and final instruction tuning. The accompanying examples span diverse generated visual content, including futuristic vehicles, animals, stained-glass scenes, surreal compositions, and detailed faces.
- Training pipeline: Show-o is jointly trained for language modeling, image captioning, and class-conditional image generation before text-to-image training.The initial stage runs over 500K steps, followed by 1,000K steps using around 35M image-text pairs.
- Training pipeline: The model is further trained on 2.0B image-text pairs, increased to 512 × 512 resolution, and then instruction-tuned on around 1M internal image-text pairs.The supplied passages describe continued pretraining and final tuning with a configuration adhering to LLaVA-v1.5.
- Generated examples: Generated examples cover futuristic cars, cartoon animals, stained-glass birds, surreal beach scenes, and hyper-realistic human faces.The examples illustrate varied visual styles and subjects rather than a single generation domain.
H MORE EXAMPLES OF VIDEO
The paper provides additional examples demonstrating Show-o’s video understanding and generation capabilities. Figure 10 samples only selected key frames for illustration.
- Video examples: Additional examples of video understanding and generation are presented in Figure 10.The passage identifies the figure as a source of further qualitative examples.
- Video examples: The figure is intended to illustrate video results through a subset of frames.This frames the examples as qualitative visual evidence rather than a complete temporal presentation.
I ABLATION STUDIES
The ablations examine vision encoders, image representations, unified pretraining, sampling steps, classifier-free guidance, and mask-free editing. They show that representation and pretraining choices affect understanding, while sampling and guidance affect generation quality and efficiency.
- Representation ablations: Discrete image tokens perform much worse than continuous representations on most multimodal-understanding benchmarks.The authors attribute this gap to insufficient alignment data for discrete tokens, whereas continuous features already occupy a well-shaped embedding space.
- Pretraining ablations: Unified pretraining improves MAGVIT-v2 continuous-representation understanding but slightly degrades most CLIP ViT-based results.The paper hypothesizes that token-based pretraining and CLIP-based tuning operate in nearly orthogonal dimensions.
- Pretraining ablations: Unified pretraining significantly improves MAGVIT-v2 discrete-token multimodal understanding.The reported gain is attributed to better cross-modal alignment from large-scale multimodal data.
- Sampling ablations: At 512 × 512 resolution, increasing sampling steps from 5 to 25 and then 50 improves prompt adherence, detail, and realism.Five steps produce roughly related images, 25 steps improve prompt fidelity, and 50 steps produce more detailed and realistic images.
- Guidance ablations: Increasing classifier-free guidance scale w to 3 and 5 makes generated colors and contents more diverse and more consistent with the prompt.Without guidance, generated objects lack detail.
- Editing: Iterative random masking of discrete image tokens enables mask-free local edits and global style transformations.Examples include changing a red apple to green, replacing it with a mouse, and transforming an image into cartoon or oil-paint styles.
K FAILURE CASES
Show-o has failure cases in both multimodal understanding and generation, particularly for text recognition, object counting, and correct object attributes. The authors attribute these failures mainly to limited, automatically captioned training data.
- Failure cases: Show-o sometimes misrecognizes text, miscounts object instances, and generates incorrect belongings for objects.The passage gives failures involving the phrase “closing down,” the term “mardefly,” and skis assigned to each instance.
- Failure cases: The authors attribute these failure modes mainly to insufficient scenario-specific data from public image-text datasets and automatically generated captions.They identify richer targeted data as a prospective way to address the failures.