Seedream 2.0: A Native Chinese-English Bilingual Image Generation Foundation Model

• Seedream 2.0 combines a continuously extensible data architecture and balanced captioning with a calibrated, self-developed bilingual decoder-only LLM text encoder.Scaled ROPE is proposed to generalize image generation to untrained resolutions.Source: §1 · p. 4
• Seedream 2.0 is presented as a leading model across prompt-following, aesthetics, text rendering, and structural correctness.Source: §1 · p. 4
• Its custom character-level text encoder supports flexible text rendering, with particular strength in long content and complicated Chinese characters.Source: §1 · p. 4
• The multilingual LLM encoder enables learning from high-quality Chinese data and handling prompts with Chinese stylistic nuances and specialized professional vocabulary.The introduction also claims exceptional performance in Chinese text rendering.Source: §1 · p. 4
• Multiple RLHF iterations across post-training modules are reported to align outputs with human preferences, reflected in an advantage in ELO scoring.Source: §1 · p. 4
• As of early December 2024, Seedream 2.0 had been incorporated into Doubao and Dreamina.Source: §1 · p. 4

2.1 Data Composition

• The data pipeline balances source and semantic distributions, injects Chinese-specific knowledge, and supplements challenging action-oriented and counterfactual examples.It uses source downsampling and hierarchical cluster sampling; a taxonomy and multimodal retrieval engine augment Chinese-context data, while active learning integrates targeted supplementary examples.Source: §2.1 · pp. 5–6

2.2 Data Cleaning Process

• Data cleaning progresses from image-quality, structure, and OCR checks to aesthetic scoring, deduplication, hierarchical semantic clustering, and stratified captioning or re-captioning.Samples failing initial quality standards are removed; semantic category tags support subsequent distribution adjustment, and higher-level data receive richer captions from different perspectives.Source: §2.2 · p. 6

2.3 Active Learning Engine

• An iterative active-learning procedure progressively refines image classifiers to improve dataset quality for training.The engine also categorizes and integrates challenging supplementary examples into the final training set.Source: §2.3 · p. 6, §2.1 · p. 6

2.4 Image Captioning

• Image captioning generates meaningful, contextually accurate descriptions in generic and specialized forms.Generic captions are produced in Chinese and English at short and long lengths: short captions capture core content, while long captions describe more aspects, including appropriate inferences and imaginings.Source: §2.4 · p. 7, §2.4.1 · p. 7

2.4.1 Generic Captions

• Generic captions pair concise accounts of an image’s core content with longer descriptions that capture more aspects and may include appropriate inferences or imaginings.Both lengths are formulated in Chinese and English.Source: §2.4.1 · p. 7

2.4.2 Specialized Captions

• Specialized captions target artistic features, image text, or surreal and fantastical content.Artistic captions describe style, color, composition, and light interaction; textual captions focus on image text, while surreal captions offer a more imaginative perspective.Source: §2.4.2 · pp. 7–8

2.5 Text Rendering Data

• The text-rendering dataset filters in-house images for rich visual text, extracts and cleans text regions with OCR, then refines recaptioned descriptions into image-caption pairs for text-rendering tasks.Processing includes cropping watermarks, removing low-quality text boxes, and refining the generated descriptions.Source: §2.5 · p. 8

3.1 Diffusion Transformer

• Images are VAE-encoded into latents and patchified into image tokens, which are concatenated with text tokens and processed by DiT blocks with shared self-attention and modality-specific MLPs.Text tokens receive learned positional embeddings; image tokens use 2D Rotary Positional Embedding.Source: §3.1 · p. 9
• Scaling RoPE assigns resolution-dependent 2D image positions so central patches share similar IDs across resolutions, supporting inference at untrained aspect ratios and resolutions.Source: §3.1 · p. 9

3.2 Text Encoder

• A decoder-only LLM text encoder is fine-tuned on text-image pairs to address embedding mismatch and enable bilingual understanding of long prompts and complex instructions.It learns native Chinese-English knowledge from massive data, supporting culturally nuanced and aesthetic image generation in both languages.Source: §3.2 · pp. 9–10

3.3 Character-level Text Encoder

• For rendered text, LLM and glyph-aligned ByT5 features are projected and concatenated as textual-only conditions, addressing ByT5-only repetition and layout issues while retaining the standard text-to-image pipeline.ByT5 supplies character-level glyph features aligned with the text prompt.Source: §3.3 · p. 10
• A re-caption model encodes font, color, size, and position as LLM-described rendering features, enabling end-to-end learning without preset text-box layouts.The approach learns rendering features directly from training data.Source: §3.3 · p. 10

4.1 Continuing Training (CT) / 4.1.1 Data

• Continuing training shifts to smaller, higher-quality data to improve aesthetics while maintaining prompt-following and structural accuracy.Its data combines automatically IQA-filtered pretraining images with millions of manually curated aesthetic images from domains such as art, photography, and design, sampled jointly to limit overfitting.Source: §4.1 · pp. 10, 12, §4.1.1 · p. 12

4.1.2 Training Strategy

• VMix [34] conditions continuing training on color, lighting, texture, and composition tags to teach fine-grained aesthetic characteristics during denoising.The authors report that this further enhances generated-image aesthetic appeal.Source: §4.1.2 · p. 12

4.2.1 Data / 4.2.2 Training Strategy

• SFT uses carefully curated artistic images, beauty-focused captions, and style and aesthetic labels, while negative generated samples and data resampling target realism and preserve text-image alignment.The authors note that high-artistic-standard SFT data can degrade image-text alignment, motivating resampling to improve aesthetics without losing alignment capacity.Source: §4.2.1 · p. 12, §4.2.2 · p. 12

4.3 Human Feedback Alignment (RLHF)

• RLHF improves image-text alignment, aesthetics, structural correctness, and text rendering, with reward curves rising consistently during alignment.The authors describe the human-feedback stage as pivotal to improving overall diffusion-model performance.Source: §4.3 · p. 12, Figure 12 · p. 13

4.3.1 Preference Data / 4.3.2 Reward Model

• RLHF uses diverse prompts, cross-model preference annotations, and Chinese-English CLIP-based reward models, including a text-rendering RM activated for relevant prompts.The annotation scheme combines image-text matching, text rendering, and aesthetics; reward models use ranking loss and include alignment, aesthetic, and text-rendering variants.Source: §4.3.1 · p. 13, §4.3.2 · pp. 13–14

4.3.3 Feedback Learning

• Feedback learning directly optimizes multiple reward scores and jointly fine-tunes the diffusion transformer and integrated LLM, with iterative bad-case-aware reward-model updates.Training stability is supported by learning-rate and denoising-step choices and exponential moving average; the method is compared with REFL [36], DPO [33], and DDPO [1].Source: §4.3.3 · p. 14

4.4 Prompt Engineering (PE)

• The prompt-engineering model yields a 30% enhancement in aesthetic quality, a 5% improvement in image-text alignment, and a substantial increase in generated-image diversity.Source: §4.4 · p. 14

4.4.1 Fine-tune LLM / 4.4.2 PE RLHF

• Prompt Engineering fine-tunes a bilingual LLM on paired original and rephrased prompts, then applies SimPO preference optimization using image-pair quality judgments.Training pairs are built both by refining user prompts until they produce high-quality images and by degrading detailed captions into simpler initial prompts.Source: §4.4.1 · p. 14, §4.4.2 · p. 15

5 Align to Instruction-Based Image Editing

• A text-to-image model’s inherited text-conditioned capabilities can support image comprehension and adaptation into an instruction-based editing model.Source: §5 · p. 16

5.1 Preliminaries

• SeedEdit uses data generation for varied rigid and non-rigid edits, and a diffusion-model image encoder to align generation and understanding features in one latent domain.The data-generation strategy is similar to InstructPix2Pix ([2]) or Jedi ([40]); the encoder differs from CLIP ([5]) and DINO ([23]) encoders used in approaches such as IP-Adaptor ([38]).Source: §5.1 · p. 16
• SeedEdit delivers superior editing quality for synthesized and real images while retaining high aesthetic and compositional fidelity to the original input.The report identifies this method as SeedEditV1.0 and says subsequent improvements are detailed in the technical report.Source: §5.1 · p. 16
• An iterative optimization strategy further integrates image and textual features to generate new images.Source: §5.1 · p. 16

5.2 Enhanced Human ID Preservation

• SeedEditV1.0 showed limited preservation of human facial identity in real images, especially when faces are small or strong text-conditioned bias alters appearance.The passage illustrates the issue with a person placed in front of “The Taj Mahal.”Source: §5.2 · pp. 16–17
• Multi-expert data fusion adds real-ID images from internal face-expert workflows and varied environments, conditionally merging them by prompt prefixes to preserve original data quality and distributions.Source: §5.2 · p. 17
• Combining diffusion loss with an AdaFace-based face-similarity loss improves the model’s ability to maintain facial features in image pairs that preserve identity well [12].Source: §5.2 · p. 17
• Data optimization uses robust filters and diverse sampling; in a 160-image validation set, expert data and face-aware loss each improve outcomes, as illustrated in Figure 15.The validation set includes real and generated images covering various editing operations.Source: §5.2 · p. 17, Figure 15 · p. 17

6.1 CFG and Step Distillation

• A guidance-scale embedding avoids CFG’s two inferences per timestep, while TSCD progressively distills trajectories from local segments to global consistency.Hyper-SD’s TSCD begins with 16 timestep segments and reduces them through 8, 4, 2, and 1 to mitigate error accumulation.Source: §6.1 · p. 17
• TSCD improves stability and efficiency while enhancing diffusion-model compression and preserving high-fidelity generation.It adaptively balances MSE for proximal predictions with adversarial loss for divergent targets.Source: §6.1 · p. 18

6.2 Quantization

• Operators gained 5% to 20% performance improvements through fusion and fine-tuning, alongside adaptive hybrid low-bit quantization for Attention and GEMM.The approach smooths intra-layer outliers, searches layer, bit-width, and granularity choices using sensitivity metrics, and fine-tunes quantization scales in a few hours to mitigate losses on sensitive layers. GPU kernels are optimized for low-bit mixed-granularity quantization.Source: §6.2 · p. 18

7 Model Performance

• Seedream excels on Chinese and English prompts, scoring highest on most evaluation perspectives and outperforming competitors in Chinese cultural nuances and related content.The comparison includes commercial and open models, with human and machine evaluations.Source: §7 · p. 18

7.1.1 Benchmark

• Bench-240 contains 240 prompts, each in Chinese and English, combining representative public-benchmark prompts with manual curation; it assesses image content and image quality, while prompt distribution is calibrated to user-preference surveys.Source: §7.1.1 · p. 19

7.1.2 Human Evaluation Results

• Seedream leads public preference in Chinese and English, ranks first in English structural evaluation and second in alignment and aesthetics, with no obvious rubric weakness.Experts score images on a 1–5 Likert scale, while public preference reflects more than 500,000 pairwise comparisons; each model averages more than 30,000 comparisons.Source: Figure 17 · p. 19, §7.1.2 · pp. 19–20

7.2 Automatic Evaluation

• Automatic evaluation covers text-image alignment and image quality, but reports only English-prompt results because external evaluation methods mainly support English input.Source: §7.2 · p. 20

7.2.1 Text-Image Alignment

• VQAScore ranks Seedream second to Ideogram, while EvalMuse gives it the highest composite score and top placement on most key metrics, including difficult counting and activity dimensions.EvalMuse uses FGA-BLIP2 to assess fine-grained alignment; VQAScore uses Clip-Flant5-xxl to estimate prompt-image correspondence, and the automated findings closely mirror manual evaluations.Source: §7.2.1 · pp. 20–21, Table 1 · p. 21

7.2.2 Image Quality

• Seedream scores highest on HPSv2, trails Midjourney v6.1 on MPS while substantially outperforming other competitors, and remains stable across internal preference metrics.The internal metrics are Internal-Align for text-image alignment and Internal-Aes for aesthetics.Source: §7.2.2 · p. 21

Text Rendering

• The specialized benchmark includes 180 Chinese prompts and an equal number of English prompts, spanning logos, posters, displays, print, handwriting, and text on unconventional substrates.Examples of unconventional substrates include French fries arranged as text and cloud-like inscriptions.Source: §7.3 · p. 21
• Availability rate measures the share of images judged acceptable when text is almost correct, considering text integration and overall aesthetics; text accuracy and hit rate are objective metrics.The accompanying definitions refer to edit distance between rendered and target text and to the number of correctly rendered characters.Source: §7.3 · p. 22
• 78% Chinese text accuracy and 82% hit rate; Seedream leads availability in Chinese and English, with highest or near-highest accuracy and hit rates, versus MiracleVision 5.0’s 65% Chinese accuracy.Chinese characters pose greater challenges because of their complex structures and larger character set. MiracleVision 5.0’s text layout often disconnects from the image background, reducing availability; Seedream also renders culturally meaningful forms such as couplets and ancient poetry.Source: Figure 19 · p. 22, §7.3 · p. 22

7.4 Chinese Characteristics

• The evaluation uses 350 prompts covering traditional clothing, food, artistic techniques, architecture, and other customs.The benchmark addresses the range of cultural features needed to represent Chinese characteristics.Source: §7.4 · pp. 22–23
• Professional designers score response rate and Chinese aesthetic from 1 to 5, with 5 indicating a perfect meeting of the criterion.Response rate measures whether target elements are correctly addressed; Chinese aesthetic measures whether the expression meets aesthetic tendencies in China.Source: §7.4 · p. 23
• In Figure 20’s Chinese Characteristics Evaluation, the authors’ model outperforms the other models, particularly in response rate, and the passage describes its advantage over them as clear.Source: §7.4 · p. 23, Figure 20 · p. 23
• Our model significantly outperforms competitors in all Chinese-characteristic dimensions, especially food, festival, craftsmanship, and architecture; examples distinguish Hot Dry noodles from Sliced noodles and Mongolian from Tibetan robes.Source: §7.4 · p. 23
• Examples contrast Hot Dry noodles with Sliced noodles and Mongolian with Tibetan robes, distinctions other models struggle to depict.The paper points to Figure 22 for these comparisons.Source: §7.4 · p. 23, Figure 22 · p. 24

7.5 Visualization

• Across Figures 22–28, the authors report that their approach demonstrates superiority in image-text alignment, structural coherence, aesthetic appeal, and text-rendering accuracy.Source: §7.5 · p. 24

8 Conclusion

• Seedream 2.0 demonstrates strong prompt adherence, aesthetic quality, structural correctness, and human-preference alignment, including exceptional ELO scores.The paper also reports widespread acclaim on Doubao (豆包) and Dreamina (即梦).Source: §8 · p. 24
• A self-developed bilingual LLM encoder, Glyph-Aligned ByT5, Scaled ROPE, and multi-phase SFT and RLHF iterations support native knowledge, character-level rendering, resolution generalization, and optimization.The bilingual encoder enables high-fidelity generation of culturally relevant content in Chinese and English.Source: §8 · p. 24
• Seedream 2.0 is particularly effective at Chinese text rendering and culturally specific scene generation.The paper reports acclaim for these capabilities on Doubao (豆包) and Dreamina (即梦).Source: §8 · p. 24

Project Leader

• The Project Leader entry lists Huang, Weilin Yang, Jianchao.Source: Project Leader · p. 33