Source-linked AI summary
Seedream 2.0: A Native Chinese-English Bilingual Image Generation Foundation Model
Lixue Gong, Xiaoxia Hou, Fanshi Li, Liang Li, Xiaochen Lian, Fei Liu, Liyang Liu, Wei Liu, Wei Lu, Yichun Shi, Shiqi Sun, Yu Tian, Zhi Tian, Peng Wang, Xun Wang, Ye Wang, Guofeng Wu, Jie Wu, Xin Xia, Xuefeng Xiao, Linjie Yang, Zhonghua Zhai, Xinyu Zhang, Qi Zhang, Yuwei Zhang, Shijia Zhao, Jianchao Yang, Weilin Huang
TL;DR
Current image-generation systems have limitations in model balance, multilingual text rendering, and understanding of Chinese cultural nuances. Seedream 2.0 addresses these gaps with a native Chinese-English bilingual model combining a bilingual LLM text encoder, character-level rendering, resolution generalization, and multi-phase post-training. It demonstrates superior performance across multiple capabilities and ranks strongly in bilingual human evaluations.
Problem
Existing image-generation systems show model bias, limited multilingual text rendering, and insufficient understanding of culturally nuanced Chinese prompts.
Method
Seedream 2.0 combines a bilingual LLM text encoder, Glyph-Aligned ByT5 for character-level rendering, Scaled ROPE for untrained resolutions, and multi-phase SFT and RLHF optimization.
Results
Seedream 2.0 demonstrates superior performance across prompt-following, aesthetics, text rendering, and structural correctness, with the highest total score in Chinese and English public-reviewer evaluations.
Takeaways & Limitations
Seedream 2.0 provides a bilingual image-generation foundation model that combines culturally relevant generation, text rendering, and broad performance across evaluation dimensions.
Abstract
from arXiv · showhide
Rapid advancement of diffusion models has catalyzed remarkable progress in the field of image generation. However, prevalent models such as Flux, SD3.5 and Midjourney, still grapple with issues like model bias, limited text rendering capabilities, and insufficient understanding of Chinese cultural nuances. To address these limitations, we present Seedream 2.0, a native Chinese-English bilingual image generation foundation model that excels across diverse dimensions, which adeptly manages text prompt in both Chinese and English, supporting bilingual image generation and text rendering. We develop a powerful data system that facilitates knowledge integration, and a caption system that balances the accuracy and richness for image description. Particularly, Seedream is integrated with a self-developed bilingual large language model as a text encoder, allowing it to learn native knowledge directly from massive data. This enable it to generate high-fidelity images with accurate cultural nuances and aesthetic expressions described in either Chinese or English. Beside, Glyph-Aligned ByT5 is applied for flexible character-level text rendering, while a Scaled ROPE generalizes well to untrained resolutions. Multi-phase post-training optimizations, including SFT and RLHF iterations, further improve the overall capability. Through extensive experimentation, we demonstrate that Seedream 2.0 achieves state-of-the-art performance across multiple aspects, including prompt-following, aesthetics, text rendering, and structural correctness. Furthermore, Seedream 2.0 has been optimized through multiple RLHF iterations to closely align its output with human preferences, as revealed by its outstanding ELO score. In addition, it can be readily adapted to an instruction-based image editing model, such as SeedEdit, with strong editing capability that balances instruction-following and image consistency.
1 Introduction
Seedream 2.0 addresses model bias, limited multilingual text rendering, and weak understanding of Chinese cultural characteristics with a native Chinese-English bilingual image-generation system. Its data, bilingual text-encoding, rendering, positional-encoding, and post-training designs target broad capability, text fidelity, cultural nuance, and human preference alignment.
- Challenges: Existing image-generation models face trade-offs across aesthetics, prompt-following, and structural correctness, alongside limited multilingual text rendering and insufficient Chinese cultural understanding.These limitations matter especially for design scenarios involving graphic and poster design and for local designers working with culturally specific content.
- Approach: Seedream 2.0 handles Chinese and English prompts while supporting bilingual image generation and text rendering.Its architecture combines a continuously extensible data system, a caption system balancing accuracy and richness, and a self-developed LLM text encoder.
- Approach: A self-developed bilingual LLM text encoder supports learning native knowledge directly from Chinese and English data for culturally nuanced image generation.Multiple calibration rounds enhance bilingual alignment and support long-text understanding and complicated instruction following.
- Optimization: Multi-level optimization across data construction, pre-training, and post-training places the model at the forefront in prompt-following, aesthetics, text rendering, and structural correctness.Post-training includes multiple SFT phases and RLHF iterations, with RLHF described as aligning outputs with human preferences and yielding a strong ELO score.
- Approach: Glyph-Aligned ByT5 provides character-level text rendering, particularly for long textual content containing complicated Chinese characters.The rendering design describes font, color, size, position, and related features through a re-caption model rather than preset text-box layouts.
2 Data Pre-Processing
Seedream 2.0 builds a high-quality, large-scale, diverse pre-training dataset through distribution maintenance, knowledge injection, targeted supplementation, progressive cleaning, active learning, and bilingual captioning. A dedicated text-rendering pipeline further produces refined image-caption pairs from visual text data.
- Data Pipeline: The pre-training data pipeline combines data composition, cleaning, active learning, captioning, and text-rendering preparation.These processes are designed to produce a high-quality, large-scale, diverse dataset.
- Data Composition: Distribution maintenance reduces overrepresented sources and samples hierarchical semantic clusters while preserving useful data relationships.Sampling spans broad categories such as visual designs and finer categories such as posters and book covers.
- Data Composition: Knowledge injection uses a taxonomy and multimodal retrieval engine to add Chinese-specific characters, flora, cuisine, architecture, scenes, and folk culture.The Chinese-context data is intended to improve performance in Chinese-specific scenarios.
- Active Learning: Targeted supplementary data adds difficult action-oriented and counterfactual examples identified by an active learning engine.The active learning system progressively refines image classifiers to support high-quality dataset construction.
- Data Cleaning: Progressive cleaning removes samples failing general quality, structure, or OCR checks, then applies aesthetic scoring, embedding-based deduplication, clustering, and recaptioning.Remaining data receives captions or recaptions stratified by data quality and enriched for higher-level data.
- Captioning: Bilingual captioning creates short and long generic descriptions plus artistic, textual, and surreal specialized captions.Short captions capture core content, while long captions describe more image aspects with appropriate inferences and imaginations.
- Text Rendering Data: The text-rendering dataset uses filtering, OCR extraction, watermark cropping, text-box quality control, recaptioning, and description refinement.The resulting image-caption pairs are used for visual text-rendering tasks.
3 Model Pre-Training
Seedream 2.0 encodes images into patch tokens and combines them with text tokens in multimodal transformer blocks. Its pre-training design addresses bilingual understanding, Chinese-native knowledge, character-level rendering, and resolution generalization.
- Architecture: The training and inference pipeline encodes input images with a VAE, patchifies the latent representation, and concatenates image tokens with encoded text tokens.The combined tokens are processed by transformer blocks.
- Architecture: DiT blocks jointly process image and text tokens with self-attention, modality-specific MLPs, adaptive layer normalization, and QK-Norm.The design follows principles from MMDiT while retaining separate processing for the two modalities.
- Architecture: Scaled RoPE configures resolution-dependent scale factors so central patches share similar position IDs across resolutions.This enables generalization to untrained aspect ratios and resolutions.
- Text Encoder: A decoder-only bilingual LLM is fine-tuned with text-image pairs to provide stronger bilingual understanding and image-text alignment than conventional encoders.The training addresses the feature-distribution mismatch that can destabilize diffusion models using LLM text encoders.
- Text Encoder: The bilingual text encoder supports long-text understanding, complicated instruction following, and direct learning of native knowledge from Chinese and English data.The paper links this capability to culturally nuanced and aesthetically expressed image generation.
- Text Rendering: Glyph-aligned ByT5 provides character-level embeddings whose features are aligned with the LLM text-encoder space before entering a DiT block.Combining LLM and ByT5 features addresses repeated characters and disordered layouts associated with ByT5-only encoding of long text.
- Text Rendering: Rendered-text font, color, size, position, and related features are directly described by a recaption model and encoded through the LLM text encoder.This contrasts with approaches that condition diffusion models on preset text-box layouts.
4 Model Post-Training
Seedream 2.0 uses sequential post-training phases to improve aesthetics, overall multi-dimensional performance, prompt adaptation, and output resolution. Human-feedback alignment optimizes several reward dimensions through iterative reward-model refinement.
- Refinement: A refiner model increases output resolution and fixes some minor structural errors after the base model generates an image.This is the final stage in the sequential post-training process.
- Aesthetic Optimization: Continue Training and SFT improve aesthetic appeal while preserving fundamental prompt-following and structural accuracy.Continue Training transitions to smaller, higher-quality data, while SFT uses carefully collected artistic images and captions.
- Aesthetic Optimization: VMix teaches fine-grained aesthetic characteristics during denoising using labels for dimensions such as color, lighting, and texture.The approach supplements direct continued training on high-quality data.
- Human Feedback Alignment: RLHF uses preference data, reward models, and feedback learning to improve image-text alignment, aesthetics, structural correctness, and text rendering.The procedure is tailored for diffusion models.
- Human Feedback Alignment: Three reward models separately assess image-text alignment, aesthetics, and text rendering, with the rendering model selectively activated for text-related prompts.Selective use of the text-rendering reward model improves character-level text generation precision.
- Human Feedback Alignment: Feedback learning directly optimizes output scores from multiple reward models and iteratively updates both the diffusion model and bad-case-aware reward models.The process alternates diffusion-model optimization, preference annotation, and reward-model training.
- Prompt Engineering: Prompt Engineering uses a fine-tuned bilingual LLM to rephrase simple user prompts into descriptions better matched to the model’s high-quality caption distribution.The framework is trained on paired initial and rephrased prompts.
5 Align to Instruction-Based Image Editing
The paper adapts the diffusion model into SeedEdit, an instruction-based image editor designed to preserve input fidelity while following edits. Subsequent enhancements target facial identity preservation in real images through expert data, face-aware loss, and data optimization.
- 5 Align to Instruction-Based Image Editing: Text-to-image diffusion models can be adapted into instruction-based image editors because their text-conditioned capabilities also support image comprehension.This adaptation is presented as a way to reveal further user-facing potential.
- 5.1 Preliminaries: SeedEdit combines broad rigid and non-rigid edit data, a causal diffusion framework, and iterative optimization of image and textual features.Its data generation strategy is similar to InstructPix2Pix or Jedi, while the diffusion model serves as an image encoder.
- 5.1 Preliminaries: SeedEdit produces edited images with high aesthetic and compositional fidelity to the original input.The reported editing quality surpasses other state-of-the-art academic and product benchmarks for synthesized and real images.
- 5.2 Enhanced Human ID Preservation: SeedEditV1.0 showed limited facial-identity retention in real images when faces were small or strongly affected by text-conditioned bias.A Taj Mahal prompt is given as an example of the person’s appearance shifting toward an Indian face.
- 5.2 Enhanced Human ID Preservation: The revision adds datasets of real identity-preserving images from face-expert workflows and varied environments and camera settings.This multi-expert data fusion addresses unrealistic identity variations in generated data.
- 5.2 Enhanced Human ID Preservation: Face-aware loss combines diffusion loss with AdaFace-based similarity measurement to improve preservation of facial features.The added perception loss is applied to image pairs that preserve human face identities well.
- 5.2 Enhanced Human ID Preservation: Robust filtering and broader sampling strategies produce a 160-image validation set spanning real and generated images and varied editing operations.Figure 15 evaluates expert data and face-aware loss, with each component reported to improve the outcomes.
6 Model Acceleration
This section describes inference acceleration and deployment optimizations, combining guidance-scale embedding, step distillation, operator fusion, and adaptive quantization.
- Inference acceleration: Guidance-scale embedding addresses CFG inefficiency while preserving guidance-scale parameterization during diffusion inference.CFG ordinarily requires two model inferences per timestep.
- Inference acceleration: Step distillation builds on TSCD to compress diffusion models for more efficient generation.The framework targets diffusion model compression while retaining high-fidelity generation.
- Inference acceleration: Adaptive balancing of MSE for proximal predictions and adversarial loss for divergent targets improves stability and efficiency.This loss design is part of the step-distillation framework.
- Implementation optimization: Operator fusion and fine-tuning improve intensive operators’ performance by 5% to 20%.The implementation also supports Attention and GEMM quantization with an adaptive hybrid quantization approach.
7 Model Performance
Seedream 2.0 is evaluated against leading text-to-image models using human and automated assessments across alignment, structure, aesthetics, and text rendering. It achieves leading or highly competitive results across these dimensions, with especially strong Chinese text rendering.
- Overall comparison: Seedream attains the highest score on most evaluated perspectives in both Chinese and English comparisons and is the most widely preferred model.The comparison uses human and machine evaluations against commercial and Chinese-language competitors.
- Benchmark and evaluation: Bench-240 contains 240 bilingual prompts designed to assess image content, relationships, and relevance.Each prompt is provided in both Chinese and English.
- Human evaluation: Seedream achieves the preeminent total public-reviewer score in both Chinese and English evaluations.In English evaluation, it ranks first structurally and second in image-text alignment and aesthetics.
- Automated evaluation: EvalMuse gives Seedream the highest composite score and top position across most crucial metrics, including difficult counting and activity dimensions.VQAScore ranks Seedream second only to Ideogram and ahead of the other models.
- Image quality: Seedream achieves the highest HPSv2 score and trails Midjourney v6.1 closely on MPS while outperforming the other competitors.Its performance remains stable across preference dimensions compared with larger oscillations from competing models.
- Text rendering: Seedream achieves 78% Chinese text accuracy and an 82% hit rate, with the best availability in both Chinese and English rendering.It also handles culturally meaningful forms such as traditional couplets and ancient Chinese poetry.
- Cross-dimensional results: The model demonstrates superiority in image-text alignment, structural coherence, aesthetic appeal, and text-rendering accuracy.These findings are reported across the comparison figures.
8 Conclusion
The conclusion presents Seedream 2.0 as a bilingual text-to-image diffusion model targeting bias, text-rendering, and cultural-understanding limitations. Its architecture and post-training support strong prompt adherence, aesthetics, structural correctness, human-preference alignment, and Chinese-focused generation.
- Conclusion: Seedream 2.0 addresses model bias, insufficient text rendering, and weak understanding of culturally nuanced prompts.The model is designed for bilingual Chinese-English image generation.
- Conclusion: A bilingual LLM text encoder enables native Chinese and English knowledge integration for culturally relevant image generation.The conclusion links this design to high-fidelity generation of culturally relevant content.
- Conclusion: Glyph-Aligned ByT5 supports character-level text rendering, while Scaled ROPE supports generalization to untrained resolutions.Together, these components broaden the model’s rendering and resolution capabilities.
- Conclusion: Multi-phase SFT and RLHF iterations improve prompt adherence, aesthetic quality, structural correctness, and human-preference alignment.The reported evidence includes exceptional ELO scores.
- Conclusion: Seedream shows strong Chinese text rendering and culturally specific scene generation, with reported acclaim on Doubao and Dreamina.Qualitative comparisons highlight alignment, structure, aesthetics, text rendering, poster design, and Chinese aesthetics.
A Contributions and Acknowledgments
The acknowledgments section states that all Seedream contributors are listed alphabetically by their last names.
- Acknowledgments: All Seedream contributors are listed in alphabetical order by their last names.
Core Contributors
The listed core contributors include Gong, Lixue Hou, Xiaoxia Li, Fanshi Li, Liang Lian, Xiaochen Liu, Fei Liu, and others.
- The contributor list names Gong, Lixue Hou, Xiaoxia Li, Fanshi Li, Liang Lian, Xiaochen Liu, and Fei Liu among the core contributors.
Project Leader
The listed project leader is Huang, Weilin Yang, Jianchao.
- The project-leader listing identifies Huang, Weilin Yang, and Jianchao.