Source-linked AI summary
Emu3.5: Native Multimodal Models are World Learners
Yufeng Cui, Honghao Chen, Haoge Deng, Xu Huang, Xinghang Li, Jirong Liu, Yang Liu, Zhuoyan Luo, Jinsheng Wang, Wenxuan Wang, Yueze Wang, Chengyuan Wang, Fan Zhang, Yingli Zhao, Ting Pan, Xianduo Li, Zecheng Hao, Wenxuan Ma, Zhuo Chen, Yulong Ao, Tiejun Huang, Zhongyuan Wang, Xinlong Wang
TL;DR
Long-horizon multimodal learning requires models that can reason over temporally extended vision-language experiences rather than relying on text or short clips alone. Emu3.5 addresses this with unified multimodal training and reinforcement learning, while DiDA accelerates visual decoding; the resulting model supports broad generation, reasoning, and world-modeling capabilities and is openly released.
Problem
Long-horizon vision-language learning and reasoning remain open challenges beyond the limited world view of text-only models and short video clips.
Method
Emu3.5 is trained end-to-end with unified next-token prediction on over 10 trillion interleaved multimodal tokens and post-trained with large-scale multimodal reinforcement learning.
Results
Emu3.5 demonstrates strong native multimodal generation, including long-horizon interleaved outputs, any-to-image generation, text-to-image generation, and world exploration and embodied interaction.
Takeaways & Limitations
Emu3.5 establishes a foundation for long-horizon vision-language generation and reasoning, with capabilities spanning perception, generation, and generalizable world modeling.
Takeaways & Limitations
The evaluation of visual narratives and visual guidance remains limited, motivating systematic quantitative and human benchmarks.
Abstract
from arXiv · showhide
We introduce Emu3.5, a large-scale multimodal world model that natively predicts the next state across vision and language. Emu3.5 is pre-trained end-to-end with a unified next-token prediction objective on a corpus of vision-language interleaved data containing over 10 trillion tokens, primarily derived from sequential frames and transcripts of internet videos. The model naturally accepts interleaved vision-language inputs and generates interleaved vision-language outputs. Emu3.5 is further post-trained with large-scale reinforcement learning to enhance multimodal reasoning and generation. To improve inference efficiency, we propose Discrete Diffusion Adaptation (DiDA), which converts token-by-token decoding into bidirectional parallel prediction, accelerating per-image inference by about 20x without sacrificing performance. Emu3.5 exhibits strong native multimodal capabilities, including long-horizon vision-language generation, any-to-image (X2I) generation, and complex text-rich image generation. It also exhibits generalizable world-modeling abilities, enabling spatiotemporally consistent world exploration and open-world embodied manipulation across diverse scenarios and tasks. For comparison, Emu3.5 achieves performance comparable to Gemini 2.5 Flash Image (Nano Banana) on image generation and editing tasks and demonstrates superior results on a suite of interleaved generation tasks. We open-source Emu3.5 at https://github.com/baaivision/Emu3.5 to support community research.
1 Introduction
Emu3.5 addresses the challenge of learning and reasoning over long-horizon interleaved vision-language data by combining unified multimodal training, reinforcement learning, and efficient inference. It demonstrates broad generation, reasoning, and world-modeling capabilities across temporal and embodied tasks.
- Motivation: Long-horizon vision-language sequences remain difficult because prior efforts focused mainly on short-form or small-scale data.The open challenges span learning from long videos, general-purpose multimodal interaction, and efficient prediction of tens of thousands of visual tokens.
- Approach: Emu3.5 is trained end-to-end with unified next-token prediction on over 10 trillion interleaved vision-language tokens, primarily from video frames and transcripts.Large-scale reinforcement learning further supports long-horizon multimodal generation.
- Capabilities: Emu3.5 generates temporally consistent interleaved visual and textual sequences for visual narratives, visual guidance, world exploration, and embodied manipulation.The reported capabilities span both real and imagined environments and include controllable interaction and dynamic scene simulation.
- Evaluation: Emu3.5 achieves performance comparable to Gemini 2.5 Flash Image on any-to-image generation and surpasses it on text rendering.It supports multiple input images and outputs up to 2K resolution.
- Observations: Unified post-training allows capabilities from text-to-image generation and any-to-image editing to transfer to visual narrative and visual guidance tasks.The authors identify this shared multimodal interface as evidence of the paradigm’s versatility and flexibility.
- Release: The authors open-source Emu3.5, including the model and development components, to support community research and development.They position the release as a foundation for future large-scale world-model research.
- Approach: DiDA converts token-by-token decoding into bidirectional parallel prediction, accelerating per-image inference by approximately 20× without sacrificing performance.This enables hybrid generation with sequential text and parallel visual outputs.
2 Emu3.5
Emu3.5 uses a unified transformer-based multimodal pipeline for training and combines sequential text generation with parallel visual generation at inference. Its tokenizer and decoders are designed to represent and reconstruct visual content efficiently while supporting image and video generation.
- Architecture and inference: Emu3.5 uses unified next-token prediction with a decoder-only transformer across multimodal pre-training, supervised fine-tuning, reinforcement learning, and inference.Inference uses DiDA for hybrid generation, combining sequential textual and parallel visual generation.
- Training pipeline: The two-stage pre-training pipeline uses approximately 13 trillion tokens from sequential video frames and transcripts, with later training improving resolution diversity, data quality, and annotation richness.The setup supports interleaved vision-language inputs and outputs within one generative framework.
- Architecture: The model follows a standard transformer architecture with design modifications intended to balance scalability and multimodal adaptability.The described configuration includes 64 transformer layers, grouped-query attention, RMSNorm, and QK-Norm.
- Tokenizer: The visual tokenizer uses IBQ with a downsampling factor of f = 16, 256-dimensional codebook tokens, and a 131,072-entry codebook.Feature distillation from SigLIP is added to enrich the semantic information of discrete image tokens.
- Image decoder: The vanilla image tokenizer improves reconstruction quality while using one-fourth as many tokens as Emu3 for the same image.A diffusion-based decoder optionally doubles output resolution and improves local, text-region, and facial details.
- Video decoder: A diffusion-based video decoder generates continuous videos from generated keyframe tokens, using quantized embeddings for visual detail and optional inter-frame text for semantic guidance.The decoder is built on a DiT architecture and includes a mask indicating which frame tokens are provided.
3 Pre-training
Emu3.5’s pre-training corpus is built around large-scale, long-horizon video-text sequences, combining sequential visual frames with temporally aligned transcripts and extensive filtering. This data supports unified multimodal learning, with validation losses declining across in-domain and out-of-distribution tasks.
- Data Composition: Over 13 trillion multimodal tokens combine interleaved vision-language data, vision-text pairs, any-to-image data, and text-only data.The interleaved subset is primarily derived from sequential video frames and temporally aligned audio transcripts.
- Data Collection: Approximately 63 million videos averaging 6.5 minutes provide roughly 790 years of footage across education, science, entertainment, sports, gaming, travel, and animation.
- Preprocessing: Keyframes are extracted through scene segmentation and timestamped sampling, while Whisper-based ASR produces grammatically coherent, temporally aligned transcripts.The processed frames and transcripts are ordered by timestamps into naturally interleaved video-text sequences.
- Data Statistics: Silent videos create many examples with no ASR text tokens, requiring balancing during subsequent data processing.
- Data Filtering: The corpus uses two-stage filtering, combining coarse cleaning and balancing with advanced quality evaluation and redundancy reduction.Filtering includes duration, resolution, talking-head, language, silence, frame-quality, cross-frame redundancy, and transcript-quality checks.
- Training Outcomes: Validation loss steadily decreases across nine held-out sets, including in-domain, out-of-distribution, and downstream-task data.The downstream validation sets cover visual narrative, visual guidance, and world exploration without pre-training overlap.
4 Post-training
Post-training expands Emu3.5 across general multimodal, image-generation, interleaved-generation, exploration, and manipulation tasks while using unified reinforcement learning to balance heterogeneous objectives. The section also introduces DiDA to reduce autoregressive image-generation latency through parallel visual-token refinement.
- General Tasks: Emu3.5 supports text-to-image generation, language understanding and generation, and vision-language question answering.
- Any-to-Image: Any-to-Image treats text-driven generation, local editing, subject-driven generation, and in-context editing or generation as instances of general-purpose image editing.The paradigm requires multimodal instruction following, consistency, stylistic control, world knowledge, and physical-law understanding.
- Visual Narrative: Visual Narrative generates consecutive storylines with interleaved narrative text and images that remain temporally consistent in character and style.
- Visual Guidance: Visual Guidance jointly composes visual and textual elements into coherent, step-by-step representations of multi-stage procedures.It aligns visual cues with linguistic expressions in scenarios such as cooking, handcrafting, and mechanical assembly.
- World Exploration: World Exploration supports user-interactive single-step updates and free exploration with autonomous, temporally coherent visual sequences and synchronized narrations.
- Embodied Manipulation: Embodied Manipulation decomposes long-horizon physical tasks into semantically distinct subtasks represented by language instructions and visual keyframes.The formulation targets physical understanding, subtask planning, and feasible intermediate motions.
- Reinforcement Learning: A multi-dimensional reward system improves multiple tasks without concentrating optimization on a single reward.The mixed-task reward rises from around 4.5 to over 7.1 during reinforcement-learning training.
- Efficient Inference: DiDA converts visual-token generation from sequential decoding into progressive parallel refinement while leaving text generation unchanged.It initializes the full image token sequence and iteratively denoises it, enabling faster inference without sacrificing output quality.
5 Tokenizer Training
Tokenizer and decoder training combines broad image and video corpora with quality filtering and staged optimization. The pipeline trains a tokenizer with balanced reconstruction, perceptual, adversarial, semantic, and quantization objectives, alongside image and video decoders for varied visual outputs.
- Image Corpus: Image data covers general, aesthetic, and specialized domains to exploit the visual tokenizer’s representational capacity.Sources include ImageNet, OpenImage, CC3M, CC12M, in-house media, and high-quality open-source images.
- Image Filtering: Images below 512 × 512 pixels or failing quality, aesthetic, or watermark checks are filtered out.
- Video Corpus: The video decoder uses Koala, Sekai, Agibot, and in-house stock videos to cover real-world scenes, exploration, manipulation, and visually appealing clips.
- Video Filtering: Video filtering segments scenes and removes clips with minimal or excessively large motion, with additional suitability scoring for Koala data.
- Tokenizer: The tokenizer uses weighted reconstruction, quantization, perceptual, adversarial, entropy, and semantic-distillation losses.It is trained with Adam, β1 = 0.5, β2 = 0.9, a 1e-4 learning rate, 15K warmup steps, and 500K total iterations.
- Image Decoder: The image decoder includes an optional flow-matching diffusion decoder initialized from Stable-Diffusion 3.5 medium and trained across resolutions and aspect ratios.
- Video Decoder: The video decoder is initialized from Wan2.2 5B and trained progressively from short clips toward longer clips.
6 Experiment
Emu3.5 is evaluated across text-to-image, any-to-image, and image-editing benchmarks, with strong results in instruction following, text rendering, editing, and subject-driven generation.
- Text-to-Image: Emu3.5 achieves the best average score on TIIF-Bench mini and superior performance on OneIG-Bench English, ranking second on its Chinese track.These benchmarks evaluate complex instruction following and fine-grained text-to-image generation.
- Text-to-Image: Emu3.5 outperforms open-source and closed-source models on LeX-Bench, with particularly significant improvements in the hard category.LeX-Bench evaluates English text accuracy across 1,310 prompts covering diverse fonts and styles.
- Text-to-Image: Emu3.5 generates images up to 2048 pixels with improved fine-grained detail, aesthetic quality, prompt following, aspect-ratio flexibility, and artistic-style diversity.Figure 12 presents qualitative results supporting these capabilities.
- Any-to-Image: 4.41 is Emu3.5’s ImgEdit score, surpassing representative baselines including Gemini 2.5 Flash Image and Qwen-Image-Edit-2509.Scores are generated by GPT-4.1 for instruction adherence, editing quality, and detail preservation.
- Any-to-Image: 7.59 is Emu3.5’s best overall score on GEdit-Bench, outperforming Gemini 2.5 Flash Image and Qwen-Image-Edit-2509.GEdit-Bench reports Semantic Consistency, Perceptual Quality, and Overall Score.
- Any-to-Image: Emu3.5 obtains the highest overall average on OmniContext, with stronger results for object than character subject-driven generation.OmniContext covers single, multiple, and scene-level subject-driven generation.
6.3 Visual Narrative
Emu3.5 generates coherent visual narratives from text or interleaved image-text inputs, achieving comparable performance to Gemini 2.5 Flash Image in automated preference evaluation.
- Visual Narrative: Emu3.5 achieves comparable performance to Gemini 2.5 Flash Image on automated visual-narrative preference evaluation.ChatGPT evaluates visual, textual, and cross-modal dimensions.
- Visual Narrative: Emu3.5 supports pure-text and interleaved image-text inputs for flexible, context-sensitive narrative creation.Qualitative results report story coherence, image-text alignment, and visual quality.
- Visual Narrative: The narratives incorporate historical facts, scientific concepts, and cultural context, enhancing their depth, educational value, and engagement.This reflects integration of creativity with domain-specific world knowledge.
- Visual Narrative: The capabilities support potential applications in educational visual materials, interactive storytelling, and creative content production.These applications are presented as promising uses rather than demonstrated deployments.
6.4 Visual Guidance
Emu3.5 produces step-by-step visual guidance by interleaving images and text, with evaluations showing strong context interpretation, actionable instruction, and multimodal consistency.
- Visual Guidance: The visual-guidance evaluation measures step relevance and completeness, instructional clarity, text–image alignment, procedural coherence, visual informativeness, task completion, and image quality.ChatGPT serves as the impartial judge in comparison with Gemini 2.5 Flash Image.
- Visual Guidance: Emu3.5 achieves consistently higher win rates than Gemini 2.5 Flash Image on tasks requiring visual-context interpretation and actionable instruction delivery.Evaluation covers textual logical coherence, visual consistency, and cross-modal relevance.
- Visual Guidance: The results indicate strong generalization and robustness across diverse visual-instruction scenarios.The authors connect this conclusion to clear, coherent, and informative multimodal guidance.
- Visual Guidance: Emu3.5 generates instructional sequences that interleave images and text while maintaining visual continuity and natural-language explanations.It can construct coherent workflows from a reference image or textual instruction.
6.5 World Exploration
Emu3.5 is evaluated for interactive and free exploration across in-domain and out-of-domain scenarios, showing stable, controllable, and spatially coherent long-range trajectories.
- World Exploration: The exploration evaluation covers pure-text and multimodal prompts across User-Interactive and Free-Exploration paradigms, including in-domain and out-of-domain scenes.This setup targets comprehensive coverage of exploration scenarios.
- World Exploration: Emu3.5 preserves stable scene layouts and natural camera transitions over extended exploration trajectories.The qualitative evaluation focuses on pure-text input under User-Interactive Mode.
- World Exploration: Emu3.5 follows user instructions step by step while maintaining spatial reasoning, context retention, scene structure, and temporal continuity.The model adaptively evolves scenes during exploration.
6.6 Embodied Manipulation
Emu3.5 is evaluated for embodied manipulation across varied robotic embodiments, tasks, and altered environments. Qualitative results indicate strong generalization and consistency across viewpoints, skills, and embodiments.
- Evaluation scope: The quantitative evaluation covers 5 robotic embodiments and over 50 tasks spanning 3 to 13 steps.Generalization is tested by perturbing initial arm positions and changing lighting, object appearance, scene layout, camera viewpoint, and background.
- Evaluation scope: The evaluation also includes 10 real-world images as part of its assessment.
- Qualitative results: Emu3.5 demonstrates strong performance and generalization using an interleaved subtask-keyframe format.
- Qualitative results: The framework supports multiple viewpoints, skills, and embodiments while achieving superior background consistency, execution integrity, and physical law adherence.
6.7 Tokenizer Reconstruction
The tokenizer is evaluated for textual, facial, and general reconstruction quality, while DiDA is evaluated for faster image inference. Results report strong specialized representations and up to 20× acceleration with comparable performance to autoregressive decoding.
- Tokenizer evaluation: Tokbench evaluates textual and facial representation, alongside a curated 60k set for general reconstruction assessment.
- Tokenizer evaluation: 51.11 T-ACCm and 70.52 T-NEDm demonstrate superior textual and facial representations, with competitive general-domain reconstruction performance.
- Decoder comparison: Diffusion-based decoding generates 2× resolution for restoration in the qualitative decoder comparison.
- Inference acceleration: 20× faster inference is achieved by DiDA while maintaining comparable performance to the AR baseline on text-to-image and image-editing tasks.The evaluated tasks include GenEval, DPG-Bench, and GEdit-Bench.
- Inference acceleration: DiDA generates a 4,096-token image in only 10s with FlagScale, reaching inference speed on par with a fast continuous-diffusion counterpart.
7 Conclusion, Limitations and Future Work
Emu3.5 is presented as a large-scale native vision-language world model for long-horizon generation, reasoning, and embodied interaction. The authors release the model and development components while identifying tokenizer efficiency, inference latency, evaluation, prompting, and embodied-agent development as future directions.
- Conclusion: Emu3.5 natively predicts the next state across interleaved vision and language using over 10 trillion multimodal tokens and large-scale multimodal reinforcement learning.
- Conclusion: The model establishes a foundation for long-horizon vision-language generation and reasoning, with reported capabilities in X2I, text-to-image, interleaved tasks, world exploration, and embodied interaction.
- Release: The full release includes the model, data pipeline, tokenizer, native multimodal pre-training, unified post-training, and DiDA for efficient inference.
- Future work: Future work targets tokenizer compression and reconstruction fidelity, further inference acceleration, systematic evaluation, multimodal prompting, and generalizable embodied agents.
- Limitations: Emu3.5 still requires 1024 tokens to encode a 512×512 image, motivating improved tokenizer compactness.
8 Authors and Contributions
This section lists the paper’s core contributors and credits their contributions by author name. The supplied passages contain contributor lists but no descriptions of individual contribution roles.
- Authors and contributions: The section identifies core contributors, with the ordering noted as based on Chinese surname stroke count.
- Authors: The supplied author lists include Fan Zhang, Zhuoyan Luo, Xu Huang, Yueze Wang, Wenxuan Wang, and Yufeng Cui.
- Authors: Additional listed contributors include Chengyuan Wang, Xinghang Li, Honghao Chen, Yang Liu, Jinsheng Wang, Jirong Liu, Yingli Zhao, Zecheng Hao, and Wenxuan Ma.
- Authors: The lists also name Haoge Deng, Ting Pan, Xianduo Li, Zhuo Chen, Yulong Ao, Zhongyuan Wang, and Tiejun Huang.