Source-linked AI summary
ERNIE-Image Technical Report
Jiaxiang Liu, Zhida Feng, Pengyu Zou, Zhenyu Qian, Tianrui Zhu, Jun Xia, Yuehu Dong, Yanzheng Lin, Honglin Xiong, Anqi Chen, Yunpeng Ding, Jinghui Duan, Lin Gao, Chao Han, Tiechao He, Jiakang Hu, Ranjun Hua, Xueming Jiang, Qingli Kong, Yuting Lei, Tianyu Li, Yunlin Liu, Changling Liu, Yaxin Liu, Yi Liu, Xuguang Liu, Xiaolong Ma, Yan Pan, Yiran Ren, Nan Sheng, Yu Sun, Siyang Sun, Yixiang Tu, Yang Wan, Huanai Wang, Siqi Wang, Yang Wu, Youzhi Yang, Xiaowen Yang, Jianwen Yang, Yehua Yang, Quanwen Zhang, Xinmin Zhang, Haoxin Zhang, Xiang Zhang, Jun Zhang, Qian Zhang, Qiao Zhao, Qi Zhou
TL;DR
Open-source text-to-image models face limitations in complex instruction following, text rendering, and aesthetic quality while closed-source systems constrain research and deployment. ERNIE-Image addresses this gap with an 8B DiT model, data-centric training, practical enhancement and distillation tools, and a dedicated aesthetic model. Experiments report leading open-source performance approaching top-tier commercial models, with the models and aesthetic resources released publicly.
Problem
Open-source text-to-image models struggle with complex instruction following, text rendering, and aesthetic quality, while many leading systems are closed-source and harder to research or deploy privately.
Source: §1 · p. 5Method
ERNIE-Image combines an 8B single-stream DiT architecture with bottom-up pre-training data construction, top-down post-training, stabilized DPO, MT-DMD distillation, a Prompt Enhancer, and ERNIE-Image-Aes.
Source: §1 · p. 5Results
ERNIE-Image achieves leading performance among open-source models and approaches top-tier commercial models in instruction following, text rendering, and aesthetic quality.
Source: §1 · pp. 5–6, §5 · p. 22Takeaways & Limitations
The released models, Prompt Enhancer, aesthetic resources, and human-annotated benchmark support practical use and further open research.
Source: §1 · p. 6Naive DPO can cause reward hacking and representation collapse in DiT models, motivating Anchor Losses as regularization.
Source: §2.2.2 · p. 7
Abstract
from arXiv · showhide
We introduce ERNIE-Image, an open-source text-to-image generation model built upon an 8B single-stream DiT architecture. ERNIE-Image aims to bridge the gap between current open-source models and leading closed-source systems through more effective mining of large-scale pre-training data and improved supervision quality throughout training. During pre-training, we adopt a bottom-up data construction pipeline that combines fine-grained image categorization, rich caption annotation, aesthetic assessment, and hierarchical sampling. This strategy reduces data noise while preserving long-tail concepts and detailed real-world knowledge, providing a stronger foundation for complex generation tasks. In the post-training stage, we use a top-down data construction pipeline for high-demand scenarios, diversify prompt annotations to better match real user inputs, and apply a stabilized DPO strategy to align the model with human aesthetic preferences. We further train ERNIE-Image-Turbo for efficient 8-NFE generation and propose MT-DMD to mitigate capability drift during distillation. To make the model easier to use in practical scenarios, we equip it with a lightweight Prompt Enhancer that expands concise user intents into structured visual descriptions. In addition, we develop ERNIE-Image-Aes, an industrial-grade aesthetic model, together with ERNIE-Image-Aes-1K, a human-annotated benchmark for realistic aesthetic evaluation. Extensive qualitative and quantitative experiments show that ERNIE-Image achieves leading performance among open-source models and approaches top-tier commercial models in instruction following, text rendering, and aesthetic quality. We release the trained models and aesthetic resources to facilitate further academic research and technical progress in the AIGC community.
1 Introduction
ERNIE-Image is an open-source 8B single-stream DiT text-to-image model designed to improve instruction following, text rendering, and aesthetic quality through data-centric training and practical tools. Experiments report leading open-source performance approaching top-tier commercial models, alongside released models and aesthetic resources.
Motivation and model
- ERNIE-Image targets complex instruction following, text rendering, and aesthetic image generation, addressing persistent limitations in open-source text-to-image models.The model is built as an 8B latent diffusion model with a single-stream DiT architecture and a smaller text encoder intended to support long, complex instructions.Source: §1 · p. 5
Data and training
- A bottom-up pre-training pipeline combines fine-grained categorization, rich captioning, aesthetic assessment, and hierarchical sampling to reduce noise while retaining long-tail concepts.The approach mines fine-grained information from a large, low-quality image pool rather than relying on deliberately synthesized domain-specific data.Source: §1 · p. 5
- Post-training uses curated high-demand data, diversified SFT annotations, stabilized DPO, and MT-DMD-supported distillation for efficient 8-NFE generation.The pipeline is designed to improve robustness to real user inputs, align aesthetic preferences, and limit capability drift during Turbo distillation.Source: §1 · p. 5
Practical usability
- The Prompt Enhancer expands concise intents into structured visual descriptions covering attributes, spatial relations, composition, text, and style.This tool is intended to make the model’s complex instruction-following capability easier to use in practical scenarios.Source: §1 · p. 5
Capabilities
- ERNIE-Image provides strong instruction following and text rendering, while ERNIE-Image-Aes supports realistic aesthetic assessment and high-quality image generation.The reported applications include commercial posters, advertising design, realistic photography, and simulations of films, lighting, and color tones.Source: §1 · pp. 5–6
Results and release
- Qualitative and quantitative comparisons show performance approaching top-tier closed-source commercial models, and the authors release model weights, Prompt Enhancer, aesthetic models, and ERIA-1K.The releases are intended to support further academic research and technical development in the community.Source: §1 · p. 6
2 Model Training
ERNIE-Image combines structured data construction, progressive training, preference optimization, and multi-teacher distillation to improve controllable, high-quality image generation. The training pipeline balances semantic coverage and sample quality, targets real-world prompting, stabilizes DPO, and routes supervision across specialized teachers.
2.1 Pre-training
- Fine-grained categorization, caption enrichment, aesthetic scoring, and hierarchical sampling construct a pre-training corpus with broader semantic coverage and improved quality control.The pipeline preserves long-tail concepts, emphasizes text in images, and samples both categories and instances using quality signals.Source: §2.1 · p. 6
- A 256 × 256 to 512 × 512 to 1024 × 1024 curriculum with diverse aspect ratios improves early optimization while preserving flexible high-resolution compositions.Source: §2.1 · p. 6
2.2.1 Supervised Fine-Tuning (SFT)
- Top-down SFT curates high-quality data for priority domains and diversifies captions into varied user-facing prompts to improve domain control and robustness to heterogeneous inputs.The targeted corpus covers categories including posters, game screenshots, portraits, objects, and anime-style content.Source: §2.2.1 · p. 7
2.2.2 Direct Preference Optimization (DPO)
- DPO aligns the model with human aesthetic preferences by maximizing preference margins between winning and losing samples using flow-matching velocity reconstruction errors.The objective uses policy and reference differential errors, with β as a scaling parameter and σ as the sigmoid function.Source: §2.2.2 · p. 7
- Anchor losses stabilize DPO by countering reward hacking, in which unbounded rejected-sample errors can cause representation collapse.The regularization grounds the model’s fundamental generative capability.Source: §2.2.2 · p. 7
2.2.3 Multi-Teacher Distribution Matching Distillation (MT-DMD)
- MT-DMD addresses capability drift by dynamically routing supervision from specialized teachers according to noisy latents, noise scale, semantic conditions, and optimization objectives.The routing can combine global stylistic guidance with localized typography or anatomy guidance and hand off from composition experts to rendering specialists across diffusion timesteps.Source: §2.2.3 · p. 8
3 Aesthetic Scoring
ERNIE-Image develops a purpose-built aesthetic annotation and evaluation stack spanning Swiss-system labeling, the realistic ERIA-1K benchmark, and ERNIE-Image-Aes. The resulting scorer substantially outperforms existing models on ERIA-1K while showing more balanced behavior across image categories.
3.1 Swiss-Tournament Based Annotation Method
- Swiss-system tournaments pair images by current standing, concentrating comparisons among similarly rated images with fewer comparisons than ELO requires.Pairwise judgments are adopted to reduce score drift and improve label stability relative to absolute ratings.Source: §3.1 · p. 9
- Calibration screening and separate annotator tournaments improve annotation quality by preventing individual sensitivity and cross-annotator standards from interfering within one tournament.Annotations from different annotators are aggregated at the dataset level.Source: §3.1 · p. 9
3.2 ERIA-1K Benchmark
- ERIA-1K addresses curated-benchmark bias with 1,000 human-annotated images spanning six categories calibrated toward realistic image distributions.Its design targets everyday snapshots, black-and-white images, AI-generated content, and other underrepresented domains.Source: §3.2 · p. 10
- ERIA-1K uses pairwise Swiss-system annotation and tier labels from 1 to 10, with calibrated professional annotators supporting held-out evaluation of generalization and practical robustness.The benchmark is intended to evaluate aesthetic scoring beyond narrow curated image distributions.Source: §3.2 · p. 10
3.3 ERNIE-Image-Aes Evaluation
- The aesthetic scorer is initialized from ArtiMuse and fine-tuned on diverse human annotations spanning both aesthetic-quality and image-content distributions.The authors report that narrower training distributions fail to generalize to out-of-distribution image categories.Source: §3.3 · p. 11
- 0.7445 SRCC and 0.7598 PLCC: ERNIE-Image-Aes outperforms UniPercept at 0.4553 and 0.4748 on ERIA-1K.ERIA-1K is used as a held-out, deployment-oriented benchmark for aesthetic scoring models.Source: §3.3 · p. 11, Table 2 · p. 11
- Existing predictors show category-specific biases: LAION-Aesthetic over-scores anime and casual snapshots, while ArtiMuse and UniPercept favor black-and-white photography.These observations motivate a scorer tailored to the paper’s broader data distribution and quality objectives.Source: §3.1 · pp. 8–9, §3.3 · p. 11
- ERNIE-Image-Aes distributes high scores across professional photography, high-quality anime, and film-style images while suppressing low-quality snapshots.This balanced behavior is reported as consistent with its stronger SRCC and PLCC results.Source: §3.3 · p. 11
4 Performance
ERNIE-Image shows strong performance across prompt enhancement, human preference, benchmark, and qualitative evaluations. Its lightweight Prompt Enhancer improves structured generation, while the 8B model performs competitively against larger open-source and commercial systems.
4.1 Prompt Enhancement Comparison
- Prompt enhancement improves structural completeness, text fidelity, and task-specific realism, with larger language models helping on reasoning-intensive and knowledge-rich tasks.The 3B enhancer improves whiteboard derivations, RPG interfaces, and webpage legibility; a larger enhancer adds denser knowledge and stronger structural constraints.Source: Figure 6 · p. 13, §4.1 · pp. 13–14
4.2 Human Evaluation
- ERNIE-Image ranks second overall and first among open-source models in human preference evaluation, while Nano Banana 2.0 leads all categories.The in-house evaluation covers photography, advertising, graphic design, image quality, aesthetics, text rendering, and instruction following.Source: Table 3 · p. 14, §4.2 · p. 14
4.3 Quantitative Results
- 0.89 is ERNIE-Image’s highest overall GenEval score without prompt enhancement, with particularly strong spatial positioning and object-attribute coordination.GenEval measures single- and two-object generation, counting, and attribute binding.Source: §4.3 · p. 14, Table 4 · p. 15
- 0.973 overall accuracy on LongText-Bench demonstrates competitive English and Chinese text rendering, while OneIG-Bench results remain state of the art across English and Chinese subsets.The results indicate balanced semantic reasoning, style consistency, and character-level spatial coordination.Source: §4.3 · pp. 14–15, Table 5 · p. 15, Table 6 · p. 16
4.4 Qualitative Analysis
- ERNIE-Image produces realistic portraits, follows complex multi-character instructions accurately, and renders Chinese text more reliably than several competing models.Comparisons identify finer facial detail, correct sticker text and poses, and fewer Chinese-character artifacts than competing systems.Source: §4.4 · p. 22
5 Conclusion
The ERNIE-Image series combines an open-source text-to-image model, Prompt Enhancer, and aesthetic model to improve instruction following, text rendering, and aesthetic quality. The authors report leading open-source performance and release the trained resources for research and community use.
5 Conclusion
- ERNIE-Image achieves leading open-source performance across benchmarks and human evaluations while targeting instruction following, text rendering, and aesthetic quality.The conclusion attributes these gains to detailed early training instructions, improved aesthetic modeling, and broader data mining.Source: §5 · p. 22
- The authors release all trained models to promote open academic research and broader access to AIGC technology.Source: §5 · p. 22
6 Authors
The paper assigns contributors to data, pre-training, post-training, evaluation, deployment, and aesthetic research, alongside a broader alphabetical contributor list.
6 Authors
- Named responsibilities cover data, pre-training, post-training, evaluation and analysis, deployment, and aesthetic research.Additional contributors are listed alphabetically by last name.Source: §6 · p. 22
- The broader contributor list contains additional team members listed in alphabetical order by surname.Source: §6 · pp. 22–23