Source-linked AI summary
Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding
Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, Dayou Chen, Jiajun He, Jiahao Li, Wenyue Li, Chen Zhang, Rongwei Quan, Jianxiang Lu, Jiabin Huang, Xiaoyan Yuan, Xiaoxiao Zheng, Yixuan Li, Jihong Zhang, Chao Zhang, Meng Chen, Jie Liu, Zheng Fang, Weiyan Wang, Jinbao Xue, Yangyu Tao, Jianchen Zhu, Kai Liu, Sihuan Lin, Yifu Sun, Yun Li, Dongdong Wang, Mingtao Chen, Zhichao Hu, Xiao Xiao, Yan Chen, Yuhong Liu, Wei Liu, Di Wang, Yong Yang, Jie Jiang, Qinglin Lu
TL;DR
Chinese prompt understanding and generation quality remain challenging for text-to-image systems. Hunyuan-DiT combines a diffusion transformer, bilingual and multilingual text encoders, iterative data optimization, caption refinement, and multi-turn dialogue, achieving state-of-the-art Chinese-to-image performance among open-source models. Its evaluation protocol uses more than 50 professional evaluators and covers multiple quality dimensions.
Problem
Existing diffusion text-to-image models have limited direct Chinese prompt understanding, while prior Chinese-focused models still need improved generation quality.
Method
Hunyuan-DiT combines a diffusion-transformer model with bilingual CLIP and multilingual T5 encoders, iterative data processing, MLLM caption refinement, evaluation feedback, and multi-turn dialogue.
Results
Hunyuan-DiT achieves state-of-the-art Chinese-to-image performance among open-source models and performs similarly to top closed-source models in subject clarity and aesthetics.
Takeaways & Limitations
The paper presents Hunyuan-DiT as a recipe for training text-to-image models with English and Chinese understanding, detailed prompting, and interactive generation.
Takeaways & Limitations
Adversarial training adds modules that substantially increase GPU-memory and training-time demands.
Abstract
from arXiv · showhide
We present Hunyuan-DiT, a text-to-image diffusion transformer with fine-grained understanding of both English and Chinese. To construct Hunyuan-DiT, we carefully design the transformer structure, text encoder, and positional encoding. We also build from scratch a whole data pipeline to update and evaluate data for iterative model optimization. For fine-grained language understanding, we train a Multimodal Large Language Model to refine the captions of the images. Finally, Hunyuan-DiT can perform multi-turn multimodal dialogue with users, generating and refining images according to the context. Through our holistic human evaluation protocol with more than 50 professional human evaluators, Hunyuan-DiT sets a new state-of-the-art in Chinese-to-image generation compared with other open-source models. Code and pretrained models are publicly available at github.com/Tencent/HunyuanDiT
1 Introduction
Hunyuan-DiT addresses limited Chinese prompt understanding by combining architectural, data, caption-refinement, and dialogue innovations. It achieves strong open-source Chinese-to-image performance while supporting detailed prompts, multiple resolutions, and interactive generation.
- Existing diffusion text-to-image models produce high-quality images but generally lack direct Chinese prompt understanding, while prior Chinese-focused models still need better generation quality.
- Hunyuan-DiT combines a diffusion-transformer architecture with bilingual CLIP and multilingual T5 encoders to improve language understanding and context length.
- The system uses an iterative data convoy pipeline, MLLM-refined structural captions, and evaluation feedback to optimize data and the text-to-image model.
- Hunyuan-DiT supports Chinese and English prompts, long text up to 256 tokens, multiple resolutions, and multi-turn dialogue for image generation and refinement.
- Hunyuan-DiT achieves state-of-the-art performance among open-source models and performs similarly to leading closed-source models in subject clarity and aesthetics.
2 Methods
Hunyuan-DiT combines a latent-space diffusion transformer with bilingual text encoding, multi-resolution positional encoding, training-stability techniques, and an iterative data pipeline. The system also supports multimodal dialogue and evaluation-guided data updates, while adversarial acceleration introduces substantial training costs.
- Diffusion Transformer: Hunyuan-DiT compresses images with a pretrained VAE and uses a transformer diffusion model conditioned by bilingual CLIP and multilingual T5 encoders.The model operates in the VAE latent space and fuses text information through cross-attention.
- Diffusion Transformer: Cross-attention replaces class-conditional adaptive normalization for fine-grained text conditioning, while decoder skip modules fuse encoder features without upsampling.The architecture patchifies latent inputs into tokens and uses encoder and decoder transformer blocks containing self-attention, cross-attention, and FFN modules.
- Positional Encoding: Centralized Interpolative Positional Encoding maps different resolutions into a shared [0, S] range, enabling faster convergence and generalization to new resolutions.Extended Positional Encoding is described as suboptimal because it cannot align different resolutions or cover cases where both dimensions are large.
- Training Stability: Training stability is improved with QK-Norm, decoder post-skip normalization, and FP32 computation for operations prone to FP16 overflow.These techniques target unstable attention computation, loss explosion, and numerical errors, respectively.
- Data Pipeline: The data pipeline acquires, interprets, layers, applies, and evaluates data through category balancing and iterative comparison with the original model.It covers subject and style diversity, uses quality tiers, and supports specialized optimization and continual data updates.
- System Interaction and Efficiency: A three-turn multimodal dialogue system uses a special <draw> token to trigger detailed prompts for Hunyuan-DiT, while adversarial acceleration requires severe additional GPU memory and training time.The dialogue component is trained on three-turn multimodal conversations; the acceleration limitation concerns adversarial training cost.
3 Evaluation Protocol
The evaluation protocol combines multidimensional metrics, a hierarchical prompt dataset, trained professional evaluators, and aggregation procedures to assess text-to-image performance comprehensively. It also evolves to address new evaluation challenges and supports multi-turn interaction assessment.
- Evaluation Dimensions: The protocol evaluates text-image consistency, AI artifacts, subject clarity, and overall aesthetics, with additional dimensions for multi-turn interaction.Multi-turn evaluation includes instruction compliance, subject consistency, and multi-turn prompt enhancement performance.
- Evaluation Execution: More than 50 professional evaluators independently assess shared image sets after standard training, and multi-person correction reduces subjective bias.The evaluators are selected for professional knowledge and evaluation experience.
- Dataset Construction: The evaluation dataset spans eight level-1 categories, more than 70 level-2 categories, and over 3,000 prompts across easy, medium, and hard difficulty levels.Prompts cover functional applications, character roles, Chinese elements, multi-turn generation, artistic styles, and subject details.
- Score Calculation: Prompt pass rates are averaged within level-2 categories, aggregated across level-1 categories, and combined using expert-informed category weights into an overall pass rate.The protocol therefore produces scores at prompt, level-2, level-1, and overall levels.
- Protocol Evolution: The evaluation framework is designed to evolve by adding dimensions, enriching feedback, dynamically adjusting datasets, and improving evaluation efficiency.The proposed feedback includes identifying text-image inconsistency locations and distortion spots.
4 Results
Hunyuan-DiT is compared with open- and closed-source state-of-the-art models using four evaluation dimensions and is further examined through ablations of architecture, positional encoding, and text encoding. The reported results favor its skip module, RoPE, combined bilingual CLIP and multilingual T5 encoders, and length-wise feature concatenation.
- Quantitative Evaluation: Hunyuan-DiT achieves the best scores among open-source models on text-image consistency, AI-artifact exclusion, subject clarity, and aesthetics.It ranks third overall among all compared models and has 1.5B parameters.
- Quantitative Evaluation: Hunyuan-DiT performs similarly to MidJourney v6 and DALL-E 3 in subject clarity and image aesthetics.The comparison includes open-source models such as Playground 2.5, PixArt-α, and SDXL, alongside closed-source systems.
- Prompt Enhancement: Prompt enhancement with an MLLM improves consistency between generated images and text descriptions for simple abstract concept prompts.The effect is illustrated qualitatively in Figure 14.
- Architecture Ablation: Removing long skip connections increases FID and decreases CLIP score, supporting the skip module’s role in feature fusion between encoder and decoder stages.The skip module mimics long skip connections used in U-Nets.
- Positional Encoding Ablation: RoPE outperforms sinusoidal positional encoding during most training stages and accelerates convergence, while adding one-dimensional RoPE to text embeddings yields no significant gains.The authors hypothesize that RoPE’s absolute- and relative-position encoding contributes to its advantage.
- Text Encoder Ablation: Combining bilingual CLIP with multilingual T5 significantly enhances FID and CLIP score, and concatenating their features along the text-length dimension performs best.The authors attribute the concatenation result to the Transformer’s global attention over individual text slots.
5 Conclusions
The report presents Hunyuan-DiT as a complete recipe spanning model design, data processing, and evaluation for English- and Chinese-language text-to-image generation. It reports top Chinese-to-image performance among open-source models and aims to provide a useful recipe for the community.
- Conclusion: Hunyuan-DiT combines model design, data processing, and evaluation into a pipeline for understanding English and Chinese prompts.The conclusion frames these combined efforts as the basis of the system’s performance.
- Conclusion: Hunyuan-DiT reaches top performance in Chinese-to-image generation among open-source models.The authors present the system as a useful recipe for training better text-to-image models.
A Additional Materials
The supplementary materials describe hierarchical subject and style organization in the training data, the full data pipeline, the data-convoy mechanism, and category weights used in evaluation.
- Training Data Hierarchies: The training data hierarchy covers subjects across more than ten thousand sub-categories and styles across over a hundred styles.Subject categories include humans, landscapes, plants, animals, goods, transportation, and games; styles include anime, 3D, painting, realistic, and traditional styles.
- Data Pipeline: The supplementary figures illustrate the complete data pipeline and the data-convoy mechanism used in model development.The supplied passages identify these mechanisms through Figures 20 and 21 without detailing their internal steps.
- Evaluation Weights: Evaluation category weights are documented separately, with the listed categories summing to 90% because 10% is reserved for internal multi-turn text-to-image evaluation.For comparisons with state-of-the-art models, only the categories shown in the table are considered.