Source-linked AI summary
Seedream 4.0: Toward Next-generation Multimodal Image Generation
Team Seedream, :, Yunpeng Chen, Yu Gao, Lixue Gong, Meng Guo, Qiushan Guo, Zhiyao Guo, Xiaoxia Hou, Weilin Huang, Yixuan Huang, Xiaowen Jian, Huafeng Kuang, Zhichao Lai, Fanshi Li, Liang Li, Xiaochen Lian, Chao Liao, Liyang Liu, Wei Liu, Yanzuo Lu, Zhengxiong Luo, Tongtong Ou, Guang Shi, Yichun Shi, Shiqi Sun, Yu Tian, Zhi Tian, Peng Wang, Rui Wang, Xun Wang, Ye Wang, Guofeng Wu, Jie Wu, Wenxu Wu, Yonghui Wu, Xin Xia, Xuefeng Xiao, Shuang Xu, Xin Yan, Ceyuan Yang, Jianchao Yang, Zhonghua Zhai, Chenlin Zhang, Heng Zhang, Qi Zhang, Xinyu Zhang, Yuwei Zhang, Shijia Zhao, Wenliang Zhao, Wenjia Zhu
TL;DR
Seedream 4.0 addresses the need for scalable, high-quality, controllable multimodal image generation. It combines an efficient DiT–VAE architecture with joint multimodal post-training and acceleration methods, and reports strong performance across T2I and image editing while supporting diverse inputs and outputs.
Problem
Current generative models face a scalability bottleneck as demands increase for image quality, controllability, and multimodal T2I and editing capabilities.
Method
Seedream 4.0 combines an efficient DiT and high-compression VAE with joint post-training for T2I and editing, plus inference acceleration methods.
Results
Seedream 4.0 reports more than 10× acceleration over Seedream 3.0 with significantly better performance and strong multimodal generation across T2I and image editing.
Takeaways & Limitations
The system supports precise editing, reference-based generation, multi-image composition, and multi-image output for creative and professional content.
Takeaways & Limitations
The training-data pipeline identified underrepresentation of fine-grained, knowledge-centric concepts such as instructional content and mathematical expressions.
Abstract
from arXiv · showhide
We introduce Seedream 4.0, an efficient and high-performance multimodal image generation system that unifies text-to-image (T2I) synthesis, image editing, and multi-image composition within a single framework. We develop a highly efficient diffusion transformer with a powerful VAE which also can reduce the number of image tokens considerably. This allows for efficient training of our model, and enables it to fast generate native high-resolution images (e.g., 1K-4K). Seedream 4.0 is pretrained on billions of text-image pairs spanning diverse taxonomies and knowledge-centric concepts. Comprehensive data collection across hundreds of vertical scenarios, coupled with optimized strategies, ensures stable and large-scale training, with strong generalization. By incorporating a carefully fine-tuned VLM model, we perform multi-modal post-training for training both T2I and image editing tasks jointly. For inference acceleration, we integrate adversarial distillation, distribution matching, and quantization, as well as speculative decoding. It achieves an inference time of up to 1.8 seconds for generating a 2K image (without a LLM/VLM as PE model). Comprehensive evaluations reveal that Seedream 4.0 can achieve state-of-the-art results on both T2I and multimodal image editing. In particular, it demonstrates exceptional multimodal capabilities in complex tasks, including precise image editing and in-context reasoning, and also allows for multi-image reference, and can generate multiple output images. This extends traditional T2I systems into an more interactive and multidimensional creative tool, pushing the boundary of generative AI for both creativity and professional applications. We further scale our model and data as Seedream 4.5. Seedream 4.0 and Seedream 4.5 are accessible on Volcano Engine https://www.volcengine.com/experience/ark?launch=seedream.
1 Introduction
Seedream 4.0 is introduced as an efficient, scalable multimodal image-generation model addressing rising demands for image quality, controllability, and multimodal capabilities. Its architecture and joint post-training support high-resolution generation, editing, professional content creation, and faster inference.
- Seedream 4.0 targets the scalability bottleneck faced by models serving higher image quality, controllability, and multimodal tasks such as T2I synthesis and image editing.
- An efficient DiT backbone and high-compression VAE reduce training and inference computation and image-token counts, enabling native 1K–4K training and generation.
- More than 10× acceleration compared to Seedream 3.0 is reported alongside significantly better performance, with the comparison measured by compute FLOPs.
- Joint multimodal post-training covers T2I and image editing, supporting single- or multi-image inputs and outputs.
- Seedream 4.0 supports professional and knowledge-based content, including charts, formulas, and design materials, while enabling high-resolution image generation and editing for improved interaction and production efficiency.
2 Data, Model Training and Acceleration
The paper combines revised data construction, multi-stage training, optimized distributed infrastructure, multimodal post-training, and several inference-acceleration methods. These choices address data imbalance and support scalable training, multimodal generation, and competitive image synthesis and editing.
- 2.1 Model Pre-training: The prior top-down resampling strategy favored natural images and underrepresented fine-grained knowledge-centric concepts such as instructional content and mathematical expressions.
- 2.1 Model Pre-training: Knowledge-data processing combines filtered and difficulty-rated PDF figures with synthetic formula images generated from OCR and available LaTeX sources.
- 2.1 Model Pre-training: Additional data modules improve caption quality, deduplication, visual descriptions, and image–text alignment through specialized classifiers, embeddings, and retrieval components.
- 2.2 Model Training: Multi-stage training begins at average resolution 512^2 and fine-tunes across 1024^2 to 4096^2, while optimized infrastructure improves memory use, kernel efficiency, workload balance, and fault tolerance.
- 2.2 Model Post-training: Joint CT, SFT, and RLHF post-training covers T2I, single-image editing, and multi-image reference and output, with CT improving instruction following and SFT improving reference–edit consistency.
- 2.2 Model Post-training: A VLM prompt-engineering model processes text and one or multiple reference images for task routing, prompt rewriting, and DiT conditioning.
- 2.3 Model Acceleration: Inference acceleration combines adaptive trajectory matching, hardware-aware 4/8-bit quantization and sparsity, and speculative decoding designed for stochastic token sampling.
- 2.4 Evaluation: Seedream 4.0 leads in both the Artificial Analysis Arena Text-to-Image and Image Editing tracks.
3 Model Performance
Seedream 4.0 achieves strong performance across text-to-image, single-image editing, and multi-image editing evaluations. It combines broad task capability with balanced instruction following, consistency, and structural integrity.
- Seedream 4.0 ranks first in both the single-image editing and text-to-image tracks.
- Text-to-Image: Across T2I evaluations, Seedream 4.0 improves over its predecessor across all measured dimensions and substantially outperforms competing models in visual aesthetics.Evaluated dimensions include prompt alignment, structural stability, visual aesthetics, dense text rendering, and content understanding.
- Text-to-Image: Seedream 4.0 is described as having strong visual impact through dynamic motion, natural lighting, and coherent color composition.
- Single-Image Editing: Seedream 4.0 delivers more balanced single-image editing performance than GPT-Image-1 and Gemini-2.5 across instruction following, consistency, structural integrity, and text editing.GPT-Image-1 leads instruction following but has weak consistency, while Gemini-2.5 emphasizes preservation but has weaker instruction following and Chinese text editing.
- Multi-Image Editing: Seedream 4.0 performs at the highest levels across all GSB dimensions in multi-image editing, outperforming GPT-Image-1 and Gemini-2.5 by almost 20% in GSB.The dimensions are instruction alignment, consistency, and structure.
- Multi-Image Editing: With more than ten reference images, Seedream 4.0 maintains more stable and coherent structures than the other compared models.
3.2 Automatic Evaluation with DreamEval
DreamEval provides large-scale, fine-grained multimodal evaluation across generation scenarios and difficulty levels. Seedream 4.0 performs strongly on easier tasks but becomes more variable and weaker on hard multimodal reasoning, especially in single-image editing.
- Benchmark design: DreamEval evaluates multimodal generation across four scenarios and 128 subtasks using 1,600 prompts.Its scoring uses fine-grained visual question answering for more interpretable and deterministic evaluation.
- Benchmark design: DreamEval separates basic generation, advanced abilities, and higher-order understanding and reasoning through tiered difficulty levels.
- Instruction following: Seedream 4.0 and GPT-4o lead instruction adherence, while Seedream 4.0’s best-of-4 results exceed its average performance.The result suggests sampling can provide users with better outputs despite greater variability.
- Instruction following: Seedream 4.0 performs well at Easy and Medium difficulty but drops at Hard difficulty, particularly for single-image editing.The authors identify multimodal understanding and reasoning as areas requiring improvement.
3.3 Inspire Creativity via Seedream 4.0
Seedream 4.0 extends image generation into interactive creative workflows spanning precise editing, references, reasoning, multimodal composition, coherent outputs, text rendering, and adaptive high-resolution generation.
- Precise editing: Seedream 4.0 performs precise prompt-based edits while largely preserving surrounding visual content.Examples include background replacement, object changes, and photorealistic portrait retouching.
- Reference generation: Reference-based generation supports transformations across 2D and 3D domains while balancing preservation and creativity.The relevant preserved attribute may be identity, intellectual property, artistic style, or an abstract concept.
- Reasoning generation: Seedream 4.0 uses implicit contextual cues and reference images to infer plausible outcomes in reasoning-based generation.This extends generation beyond strict instruction following to contextual interpretation.
- Multi-image composition: Multiple reference images enable composition of characters or objects and abstract style transfer by extracting salient visual features.The capability supports applications beyond virtual try-on and image collage.
- Multi-image output: Seedream 4.0 generates coherent image sequences with consistent characters and visual styles for storyboarding, comics, IP products, and emoji.Its global planning and in-context consistency support sequential and set-based outputs.
- Professional creation: Enhanced text rendering supports dense layouts, formulas, chemical equations, statistical charts, interfaces, posters, and schematics.These capabilities target structured and knowledge-intensive visual generation.
- Resolution and composition: Adaptive aspect ratio selection adjusts the canvas to semantic requirements or reference-object shapes, while supporting resolutions up to 4K.The authors report more aesthetically pleasing compositions and quality suitable for commercial applications.
4 Seedream 4.5
Scaling model size and training data produces Seedream 4.5, which improves Seedream 4.0 across text-to-image and image-editing dimensions.
- Seedream 4.5: Seedream 4.5 is a strengthened version produced by scaling both model size and training data.
- Performance improvements: Seedream 4.5 improves text–image alignment, structural fidelity, editing consistency, and fine-grained text rendering over Seedream 4.0.Figure 16 illustrates the reported comparisons.
5 Conclusion
Seedream 4.0 combines an efficient diffusion transformer and high-compression VAE with joint multimodal post-training. It supports diverse creative and professional applications at high resolution, while Seedream 4.5 extends the framework through scaling.
- Conclusion: Seedream 4.0 achieves more than ten times the acceleration of Seedream 3.0 while delivering superior performance across evaluated aspects.The framework combines an efficient scalable diffusion transformer with a high-compression VAE.
- Conclusion: Joint post-training on text-to-image and image-editing tasks supports diverse input and output modalities.The reported capabilities include precise editing, reference generation, multi-image composition, and multi-image output.
- Conclusion: Seedream 4.0 supports ultrafast high-resolution generation and complicated professional content such as knowledge-centric visualizations.The conclusion highlights applications that previous models found difficult to perform.
A Contributions and Acknowledgments
The contributors are listed alphabetically by last name.
- All contributors of Seedream are listed in alphabetical order by their last names.
A.1 Core Contributors
The core contributor list includes researchers whose surnames span Chen through Heng.
- The core contributor roster includes Yunpeng Chen, Yu Gao, Lixue Gong, and Meng Guo.
- The roster continues through contributors including Qiushan Guo, Zhiyao Guo, Xiaoxia Hou, and Weilin Huang.
- Additional listed contributors include Yixuan Huang, Xiaowen Jian, Huafeng Kuang, and Zhichao Lai.
A.2 Contributors
The contributor list includes researchers whose surnames span Chen through Wen.
- The contributor roster includes Haoshen Chen, Kaixi Chen, Tiantian Cheng, and Fei Ding.
- The list continues with contributors including Xiaojing Dong, Xin Dong, Yiming Fan, and Yongde Ge.
- Additional contributors include Shucheng Guo, Bibo He, Jiaao He, and Zhuo Jiang.
- The roster includes contributors such as Lurui Jin, Hongwei Kou, Bo Li, and Changchun Li.