Source-linked AI summary
ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Experts
Zhida Feng, Zhenyu Zhang, Xintong Yu, Yewei Fang, Lanxin Li, Xuyi Chen, Yuxiang Lu, Jiaxiang Liu, Weichong Yin, Shikun Feng, Yu Sun, Li Chen, Hao Tian, Hua Wu, Haifeng Wang
TL;DR
Existing text-to-image diffusion models can generate photorealistic images but still face limitations in image fidelity and text relevancy. ERNIE-ViLG 2.0 adds fine-grained textual and visual scene knowledge and stage-specific denoising experts, achieving state-of-the-art MS-COCO performance and stronger human-evaluated alignment and fidelity.
Problem
Existing diffusion models still have limitations in image fidelity and text relevancy, including missed key elements, attribute confusion, and shared parameters across different denoising stages.
Method
ERNIE-ViLG 2.0 extracts textual and visual scene elements for alignment learning and assigns different denoising experts to different denoising stages.
Results
6.75 zero-shot FID-30k on MS-COCO marks a new state-of-the-art, while human evaluation finds higher image-text alignment and image fidelity than recent methods.
Takeaways & Limitations
Fine-grained scene knowledge and stage-specific experts improve semantic control, reduce object-attribute mismatching, and support high-quality Chinese text-to-image generation.
Takeaways & Limitations
Character rendering remains difficult, particularly for Chinese characters, which may be rendered as meaningless strokes.
Abstract
from arXiv · showhide
Recent progress in diffusion models has revolutionized the popular technology of text-to-image generation. While existing approaches could produce photorealistic high-resolution images with text conditions, there are still several open problems to be solved, which limits the further improvement of image fidelity and text relevancy. In this paper, we propose ERNIE-ViLG 2.0, a large-scale Chinese text-to-image diffusion model, to progressively upgrade the quality of generated images by: (1) incorporating fine-grained textual and visual knowledge of key elements in the scene, and (2) utilizing different denoising experts at different denoising stages. With the proposed mechanisms, ERNIE-ViLG 2.0 not only achieves a new state-of-the-art on MS-COCO with zero-shot FID score of 6.75, but also significantly outperforms recent models in terms of image fidelity and image-text alignment, with side-by-side human evaluation on the bilingual prompt set ViLG-300.
1. Introduction
ERNIE-ViLG 2.0 addresses text-image misalignment and the difficulty of using one denoising network across different stages by adding scene knowledge and stage-specific experts. It achieves state-of-the-art MS-COCO performance and stronger human-evaluated fidelity and alignment.
- Motivation: Text-image models may miss important elements and interactions because all tokens and image regions contribute equally during learning.This can produce attribute confusion, especially in prompts containing multiple objects with specific attributes.
- Motivation: A single U-Net must learn different denoising capabilities across stages, although early steps remove noise and later steps refine details.The paper identifies this shared-parameter design as a limitation of existing models.
- Approach: ERNIE-ViLG 2.0 extracts key scene elements with a text parser and object detector, guides their alignment, and assigns specialized experts to denoising stages.The method is designed to model relationships among objects and attributes while decoupling denoising capabilities across steps.
- Results: 6.75 zero-shot FID-30k on MS-COCO establishes ERNIE-ViLG 2.0 as a new state-of-the-art result.The model also scales to 24B parameters and receives stronger human evaluations for image-text alignment and image fidelity than recent methods.
2. Method
The method enhances diffusion training with textual and visual knowledge for fine-grained scene alignment, then assigns specialized denoising experts to blocks of timesteps. It preserves the diffusion pipeline while emphasizing important words, image regions, and stage-specific denoising behavior.
- Diffusion Framework: Diffusion training encodes text with a text encoder and uses a conditioned denoising network to generate images from Gaussian noise.ERNIE-ViLG 2.0 enhances both components with textual and visual knowledge of key scene elements.
- Diffusion Framework: Cross-modal attention concatenates projected image and text representations so image features can interact with the text prompt.The U-Net representation and encoded text representation are combined through an attention layer.
- Textual Knowledge: Keyword knowledge strengthens attention involving image tokens and selected text keywords, including special adjective and noun markers.The scaling factor increases attention weights for interactions involving the selected keywords.
- Visual Knowledge: Visual knowledge uses an object detector on part of the training data and assigns higher loss weights to regions containing selected salient objects.The method can emphasize regions corresponding to objects such as dogs and cats, but selected objects may require category and attribute labels to align them with text.
- Training and Inference: The added knowledge strategies are applied during training so the model can use their learned signals during inference without special tokens or attention strengthening.The approach randomly equips part of the training samples with the enhancement strategies.
- Mixture-of-Denoising-Experts: MoDE divides denoising timesteps into consecutive blocks and assigns each block a specialized expert network.Adjacent timesteps are grouped because their inputs and denoising tasks are similar; the model uses one expert network for each block.
3. Experiments
Experiments evaluate ERNIE-ViLG 2.0 on automatic MS-COCO metrics, bilingual human preferences, and ablations of knowledge enhancement, expert count, and denoising attention. The model achieves strong quantitative and human-evaluated results while analyses associate textual and visual knowledge with complementary improvements and show denoising attention changes across timesteps.
- Automatic Evaluation: 6.75 zero-shot FID-30k on MS-COCO 256 × 256 establishes ERNIE-ViLG 2.0 as state of the art.The evaluation randomly selects 30,000 validation images and translates English captions to Chinese.
- Human Evaluation: 56.5%±3.8% and 68.2%±3.8% user preference rates for image-text alignment exceed DALL-E 2 and Stable Diffusion, respectively.For image fidelity, the corresponding preferences are 58.8%±3.6% and 66.5%±3.5%.
- Ablation Studies: Textual knowledge improves fine-grained semantic control, while visual knowledge promotes fidelity; combining them improves fine-grained image-text alignment.Object knowledge alone does not steadily improve performance, whereas synthetic descriptions help exploit visual knowledge.
- Ablation Studies: Increasing denoising experts gradually improves performance, with 10 experts producing clearer clock numbers, more harmonious proportions, and more photorealistic images than one.The analysis reports that more expert networks outperform using one network trained on more samples.
- Attention Analysis: Cross-attention is broadly distributed near timestep 1,000 because noisy inputs lack a clear layout and image tokens attend to the prompt to form an image skeleton.The visualization covers attention from image features to text representations across a 1,000-step denoising process.
4. Related Work
Related work spans text-to-image generation with GANs and transformers, alongside knowledge-enhanced language and vision-language models. These prior approaches motivate incorporating structured knowledge to focus learning on important elements and improve language-image alignment.
- Text-to-Image Generation: Text-to-image generation synthesizes images from natural-language descriptions, with prior approaches including GANs and transformer-based models.Examples include ERNIE-ViLG, DALL-E, CogView, Make-A-Scene, and Parti.
- Knowledge Enhancement: Knowledge-enhanced models use masking, knowledge-aware pretraining, object tags, scene graphs, and document layouts to improve language or vision-language understanding.These methods provide structured signals that help models focus on relevant elements and better align language with visual content.
5. Risks, Limitations, and Future Work
ERNIE-ViLG 2.0 remains limited in character rendering, especially for difficult Chinese characters, while its broader use also raises misuse and dataset-bias risks. The authors report that more denoising experts improve performance but explored only up to 10 because of computation limits.
- Model Usage and Data Bias: Large-scale image-text training introduces risks of malicious use and social or cultural bias inherited from web-crawled datasets.The paper notes that generated images may reflect biases in the images and alt-text used for training.
- Character Rendering: Character rendering remains challenging because training mixes Chinese text-image pairs with English pairs translated into Chinese.The model may need to learn corresponding characters in both languages simultaneously.
- Character Rendering: For difficult cases, the model learns the requested character position but produces meaningless strokes rather than correct characters.Figure 8 includes successful simple cases and a failure case involving Chinese characters.
- Variation of Mixture-of-Denoising-Experts: Using more denoising experts leads to better model performance, but computation limited experiments to at most 10 experts.The authors suggest exploring more experts and multiple text encoders in future work.
6. Conclusions
ERNIE-ViLG 2.0 combines scene knowledge with timestep-specific denoising experts to improve fine-grained semantic control, and scales to 24B parameters. It achieves state-of-the-art MS-COCO performance and is preferred over strong baselines in bilingual human evaluation for text relevancy and image fidelity.
- Conclusions: The model incorporates visual and textual scene knowledge and introduces MoDE to disentangle parameters across denoising timesteps.The model is scaled to 24B parameters with relatively short inference time.
- Conclusions: ERNIE-ViLG 2.0 achieves state-of-the-art performance on MS-COCO, with each proposed mechanism contributing to the final results.The conclusion attributes the outcome to the combined proposed mechanisms without reporting a separate value here.
- Conclusions: Human evaluation on the bilingual ViLG-300 prompt set indicates that ERNIE-ViLG 2.0 is preferred over strong baselines in text relevancy and image fidelity.The prompt set was collected to enable fair comparisons between Chinese and English text-to-image models.
- Future Work: Further analysis suggests that different knowledge sources improve different aspects of generation and that more experts produce better image quality.The authors plan to enrich external image-text alignment knowledge and expand the use of multiple experts.
A. Detailed Training Process
Training uses paired images and text, extracts textual keywords and salient image regions, and applies this extra information in knowledge-enhanced training across two stages.
- Training Inputs: The training pipeline extracts text keywords with jieba and salient image regions with an object detector from paired image-text inputs.These additional signals are used for knowledge-enhanced training.
- Training Stages: The first training stage trains a 2.2B-parameter U-Net and a 1.3B-parameter text encoder for 350,000 steps.The passage describes this as the first of two training stages.
- Training Stages: The reported training process consists of two stages for progressively training the model components.The supplied passage introduces the second stage but does not provide its full details.
B. Detailed Automatic Evaluation
Automatic evaluation compares model size, denoising experts, reranking, and FID-30K. The final 24B model with 10 experts outperforms the similarly sized Parti model while using fewer reranking images.
- Automatic Evaluation: 8.07 FID-30K is achieved by the 3.5B model with one denoising expert without reranking, versus 10.39 for DALL-E 2 at similar size.The comparison places the single-expert first-stage model above DALL-E 2 on this reported metric.
- Automatic Evaluation: 7.23 FID-30K is achieved by the final 24B model with 10 experts without reranking.This is the reported no-reranking result for the larger model.
- Automatic Evaluation: 6.75 FID-30K with 4 reranking images lets the 24B model outperform Parti’s 7.23 with 16 reranking images.The models have a similar number of parameters, while ERNIE-ViLG 2.0 uses fewer reranking images.
- Automatic Evaluation: The comparisons indicate that both extra knowledge and model scaling contribute to the final performance.This conclusion is drawn from the reported model-size and training-stage comparisons.
C. Detailed Human Evaluation
ViLG-300 was constructed as a bilingual evaluation set from filtered DrawBench and ERNIE-ViLG prompts, with manual translation and proofreading to create parallel Chinese-English inputs.
- ViLG-300 combines 150 randomly sampled prompts from filtered DrawBench and ERNIE-ViLG prompt pools.The construction process first removed language-related DrawBench prompts and ERNIE-ViLG prompts, leaving 162 and 398 prompts, respectively.
- The final set contains parallel Chinese and English prompts produced through manual translation and proofreading.
C.2. Detailed Results on ViLG-300
On ViLG-300, ERNIE-ViLG 2.0 performs strongly on colors and complex scenes, with particularly impressive results in Geography, Scene, and Cartoon categories.
- ERNIE-ViLG 2.0 is especially skilled at prompts involving colors and complex scenes.
- The model shows impressive performance in categories including Geography, Scene, and Cartoon.
- The section includes detailed comparisons with DALL-E 2 and Stable Diffusion on ViLG-300.
D.1. Knowledge Enhancement Ablation
Knowledge-enhancement strategies accelerate diffusion-model convergence and improve different aspects of text-image alignment in complementary categories, while CLIP remains limited for multi-object relationships.
- Knowledge-enhancement strategies significantly accelerate diffusion-model convergence.At the beginning of training, the knowledge-enhanced model reaches or exceeds the baseline trained with twice as many samples.
- The knowledge-enhanced model can match or exceed baselines trained with twice as many samples early in training.The reported comparisons are 100M versus 200M samples and 200M versus 400M samples.
- Different knowledge strategies improve different categories, indicating complementary gains in text-image alignment.
- CLIP does not adequately capture relationships between multiple objects, motivating more accurate automatic evaluation for fine-grained semantic control.
D.2. Mixture-of-Denoising-Experts Ablation
The ablations examine training scale, denoising-expert count, human comparisons, knowledge combinations, and image quality. They associate expert decoupling with improved performance and more natural visual details, while textual and visual knowledge provide complementary benefits.
- Mixture-of-Denoising-Experts Ablation: A single expert trained with 400M samples performs about as well as two experts trained with 200M samples each, but later lags behind the two-expert setting.
- Mixture-of-Denoising-Experts Ablation: Decoupling denoising capabilities across stages is reported as effective, and scaling the U-Net can further improve text-to-image performance.
- Comparison of Image Quality: ERNIE-ViLG 2.0 directly generates 1024×1024 images, whereas Stable Diffusion produces 512×512 images and DALL-E 2 uses cascaded upscaling.
- Comparison of Image Quality: Compared with DALL-E 2, ERNIE-ViLG 2.0 produces more natural and photorealistic textures in magnified regions.
- Knowledge Enhancement: Textual and visual knowledge have non-overlapping impacts, and combining them facilitates accurate semantic control and high image fidelity.
- Mixture-of-Denoising-Experts Ablation: Increasing the number of denoising experts makes generated-image textures more natural and photorealistic.