Source-linked AI summary
Emu3: Next-Token Prediction is All You Need
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, Yingli Zhao, Yulong Ao, Xuebin Min, Tao Li, Boya Wu, Bo Zhao, Bowen Zhang, Liangdong Wang, Guang Liu, Zheqi He, Xi Yang, Jingjing Liu, Yonghua Lin, Tiejun Huang, Zhongyuan Wang
TL;DR
Multimodal modeling has been dominated by diffusion and compositional systems, leaving the effectiveness of next-token prediction across tasks uncertain. Emu3 discretizes images, text, and videos and trains one transformer solely with next-token prediction. It achieves state-of-the-art generation and perception results, including competitive image and video generation, while the paper presents the paradigm as a promising route beyond language.
Problem
Next-token prediction’s effectiveness for achieving competitive performance across multimodal tasks remains uncertain, while diffusion and compositional approaches dominate generation and perception.
Method
Emu3 tokenizes images, text, and videos into a discrete space and jointly trains a single transformer from scratch on multimodal sequences using next-token prediction.
Results
Emu3 achieves state-of-the-art performance across multimodal generation and perception, surpasses SDXL and LLaVA-1.6, and generates high-fidelity videos by predicting video tokens causally.
Takeaways & Limitations
The results provide strong evidence that next-token prediction is a powerful multimodal paradigm with potential to scale beyond language and simplify model designs.
Abstract
from arXiv · showhide
While next-token prediction is considered a promising path towards artificial general intelligence, it has struggled to excel in multimodal tasks, which are still dominated by diffusion models (e.g., Stable Diffusion) and compositional approaches (e.g., CLIP combined with LLMs). In this paper, we introduce Emu3, a new suite of state-of-the-art multimodal models trained solely with next-token prediction. By tokenizing images, text, and videos into a discrete space, we train a single transformer from scratch on a mixture of multimodal sequences. Emu3 outperforms several well-established task-specific models in both generation and perception tasks, surpassing flagship models such as SDXL and LLaVA-1.6, while eliminating the need for diffusion or compositional architectures. Emu3 is also capable of generating high-fidelity video via predicting the next token in a video sequence. We simplify complex multimodal model designs by converging on a singular focus: tokens, unlocking great potential for scaling both during training and inference. Our results demonstrate that next-token prediction is a promising path towards building general multimodal intelligence beyond language. We open-source key techniques and models to support further research in this direction.
1 Introduction
Emu3 tests whether next-token prediction can support competitive multimodal generation and perception without diffusion or compositional architectures. It tokenizes multimodal content and trains a single transformer, achieving strong results across image, language-vision, and video tasks.
- Motivation: Next-token prediction’s effectiveness for multimodal tasks remains uncertain, while generation and perception are dominated by diffusion and compositional methods.Earlier unified attempts either connected language models to diffusion models or failed to match task-specific performance.
- Method: Emu3 tokenizes images, text, and videos into discrete tokens and jointly trains a single transformer from scratch on multimodal sequences.This design eliminates the need for diffusion or compositional architectures.
- Results: Emu3 achieves state-of-the-art performance across generation and perception, outperforming SDXL and competing with LLaVA-1.6 on public benchmarks.Image-generation evaluations include MSCOCO-30K, GenEval, T2I-CompBench, and DPG-Bench; perception evaluations include SEED-Bench, RealWorldQA, and OCRBench.
- Results: Emu3 generates videos causally by predicting successive video tokens, supports continuation from video context, and competes with video diffusion models on VBench.The paper reports high-fidelity text-conditioned video generation and simulation of aspects of physical-world environments.
- Resources and alignment: The authors open-source key techniques and models, including a vision tokenizer for converting images and videos into discrete tokens.They also apply direct preference optimization to autoregressive vision generation to align outputs with human preferences.
- Conclusion: The results support next-token prediction as a promising multimodal paradigm that can scale beyond language while simplifying model design.The paper frames token-focused modeling as having potential for scaling during both training and inference.
2 Approach
Emu3’s approach builds a multimodal token pipeline spanning data curation, vision tokenization, unified sequence formatting, transformer training, and post-training. The system combines filtered image-video data with next-token learning, quality fine-tuning, preference optimization, and vision-language tuning.
- 2.1 Data: Emu3 is trained from scratch on language, image, and video data, including Chinese and English language data.The image corpus combines open-source web, AI-generated, and in-house data.
- 2.1 Data: Image data is filtered by resolution and aesthetic quality, then enriched with dense synthetic captions produced through an Emu2-based captioner and GPT-4V.Images below 512 × 512 pixels are discarded, and the aesthetic filter excludes scores below 5.5.
- 2.1 Data: Video preprocessing splits clips into scenes, removes excessive text, filters motion using optical flow, and discards clips with low aesthetic quality.Frames are sampled at 2 FPS and resized for efficiency; clips are rejected when the lowest of three aesthetic scores is below 5.
- 2.1 Data: A video captioner trained from the image captioner uses GPT-4V-labeled frames to describe video content and motion.Eight frames are sampled for each clip during initial labeling, with some annotations manually revised.
- 2.2 Vision tokenizer: The vision tokenizer encodes a 4 × 512 × 512 video clip or 512 × 512 image into 4096 discrete tokens from a 32,768-entry codebook.It provides 4× temporal and 8×8 spatial compression and adds temporal residual layers with 3D convolutions.
- 2.3 Architecture: Emu3 retains an LLM-style architecture, expanding its embedding layer to accommodate discrete vision tokens while using RMSNorm, GQA, SwiGLU, and RoPE.The model configurations are summarized in Table 3.
- 2.4 Pre-training: Images and videos are represented as document-like sequences combining caption text, metadata, special boundary tokens, and vision tokens.Metadata records image resolution or video resolution, frame rate, and duration; some examples place captions after vision tokens for understanding tasks.
- 2.4 Pre-training: Because vision is discretized, training uses standard next-token cross-entropy, with vision-token losses weighted by 0.5 to prevent them from dominating.Pre-training uses extensive context lengths and tensor, context, and data parallelism for video-scale sequences.
3 Main Results
Emu3 delivers competitive image and video generation using autoregressive next-token prediction, including strong benchmark and human-evaluation results. It also supports video extension and vision-language evaluation without relying on diffusion or compositional architectures.
- Text-to-Image Generation: Emu3 outperforms autoregressive diffusion methods on image-text alignment and is comparable to state-of-the-art diffusion models across four text-to-image benchmarks.The benchmarks are MSCOCO-30K, GenEval, T2I-CompBench, and DPG-Bench.
- Text-to-Image Generation: Emu3 surpasses Chameleon, Show-O, Transfusion, and SDXL on rewritten GenEval prompts while matching DALL-E 3.The reported comparison uses GPT-4V to rewrite short prompts before evaluation.
- Text-to-Image Generation: 81.6 overall on DPG-Bench is achieved by Emu3-DPO, surpassing SDXL and PixArt-alpha and remaining comparable to DALL-E 3.The authors note that DPO slightly lowers automated scores relative to Emu3, possibly because preference data emphasizes overall aesthetics.
- Text-to-Image Generation: Emu3 outperforms SDXL and is on par with DALL-E 3 and MJ-v5.2 in human-evaluated overall text-to-image performance.The evaluation uses 100 diverse prompts, three independent voters per prompt, and scores visual quality and prompt following.
- Video Generation: Emu3 natively generates 5-second videos at 24 FPS and can extend them autoregressively.Qualitative examples present frames extracted from the first three seconds.
- Video Generation: Emu3 achieves highly competitive overall VBench results, outperforming most open-source text-to-video models but trailing Kling and Gen-3.VBench evaluates video quality and semantic capabilities across 16 dimensions.
- Video Generation: Emu3 extends videos by tokenizing two seconds of context and predicting the subsequent two seconds as discrete vision tokens.The predicted tokens are detokenized into future video content.
- Vision-Language Understanding: Emu3 is evaluated as an encoder-free method across public vision-language benchmarks and surpasses counterparts across several benchmarks.The comparison includes encoder-based methods using pretrained CLIP vision encoders and encoder-free methods.
4 Related Work
Prior multimodal systems commonly combine pretrained vision encoders with language models, use diffusion for generation, or pursue unified autoregressive objectives. Emu3 extends encoder-free, token-based unification through decoder-only next-token prediction.
- Vision-Language Understanding: CLIP learns generalizable vision representations through contrastive learning on massive image-text pairs and supports zero-shot image classification.Flamingo connects pretrained language models and vision encoders for few-shot multimodal understanding.
- Vision-Language Understanding: LLaVA and related vision-language models improve through curated datasets and training strategies, while Fuyu and EVE feed image patches into language models without vision encoders.The passage states that Fuyu and EVE still face challenges competing with state-of-the-art vision-language models.
- Vision Generation: Diffusion models dominate high-resolution vision generation, while autoregressive alternatives generate images by predicting the next token in a sequence.The passage identifies Stable Diffusion as a major driver of research in this direction.
- Unified Understanding and Generation: Unified vision understanding and generation has included multimodal-element prediction, mixed image-text token modeling, and other generative objectives.Emu and Emu2 regress visual embeddings or classify textual tokens, while CM3Leon and Chameleon train token-based autoregressive models.
5 Conclusion
The paper presents Emu3 as a unified next-token-prediction approach for multimodal generation and perception. Its reported results support using discrete tokens and a single transformer across image, text, video, and understanding tasks.
- Conclusion: Emu3 tokenizes images, text, and videos into a discrete space and trains a single transformer from scratch.This design eliminates reliance on diffusion and compositional methods.
- Conclusion: Emu3 surpasses established task-specific models such as SDXL and LLaVA-1.6 across multimodal generation and perception.The conclusion includes challenging video generation among the diverse evaluated tasks.
- Conclusion: The authors conclude that next-token prediction is a viable paradigm for multimodal models beyond language.They frame this result as progress toward general multimodal intelligence and artificial general intelligence.
Contributors and Acknowledgements
The supplied passage identifies Zhongyuan Wang, Yonghua Lin, and Tiejun Huang as contributors.
- Contributors: Zhongyuan Wang is listed among the paper’s contributors.
- Contributors: Yonghua Lin is listed among the paper’s contributors.
- Contributors: Tiejun Huang is listed among the paper’s contributors.
A.1 Video Dataset
The remaining video clips are characterized by their duration and flow-score distributions, shown in Figures 9 and 10.
- Figure 9 shows the duration distribution of the remaining video clips.
- Figure 10 shows the flow score distribution of the filtered video clips.
B.1 Image Generation
The image-generation evaluation covers multiple text-to-image benchmarks and uses specified decoding settings and output resolutions. Emu3-DPO achieves a reported DPG-Bench score exceeding SDXL and PixArt-alpha and comparable to DALL-E 3.
- Image generation uses Top-k 16,384, Top-p 1.0, 512 x 512 outputs for Emu3, and 720 x 720 outputs for Emu3-DPO.
- The evaluation includes MSCOCO 30K, GenEval, T2I CompBench, and DPG-Bench for assessing text-to-image generation.
- GenEval reports performance across Single Object, Two Objects, Counting, Colors, Position, and Color Attribute dimensions.
- Specialized video stabilization and super-resolution models are applied to generated videos to improve temporal consistency and visual quality.
C Qualitative Examples for Multimodal Understanding
The qualitative examples illustrate Emu3’s multimodal understanding through image description, formula transcription, and visual question answering.
- Emu3 identifies the pictured landmark as the Brooklyn Bridge and describes its connection between Brooklyn and Manhattan.
- The examples include user requests for formula transcription and answering a food-chain question with an option letter.
- Emu3 describes the image as a Fall 2017 issue of the magazine Parktakes.
- The magazine-cover description includes kayakers, safety equipment, park activities, seasonal festivals, and registration information.