Source-linked AI summary
HiDream-I1: A High-Efficient Image Generative Foundation Model with Sparse Diffusion Transformer
Qi Cai, Jingwen Chen, Yang Chen, Yehao Li, Fuchen Long, Yingwei Pan, Zhaofan Qiu, Yiheng Zhang, Fengbin Gao, Peihan Xu, Yimeng Wang, Kai Yu, Wenxuan Chen, Ziwei Feng, Zijian Gong, Jianzhuang Pan, Yi Peng, Rui Tian, Siyu Wang, Bo Zhao, Ting Yao, Tao Mei
TL;DR
High-quality image generation often incurs substantial computational complexity and latency. HiDream-I1 addresses this with a sparse MoE-based DiT, offering scalable variants and extensions for editing and interactive creation. It reports state-of-the-art quality, including the highest average HPSv2.1 score among evaluated models.
Problem
State-of-the-art image generators often face increased computational complexity and longer inference times, limiting real-time, cost-effective deployment.
Method
HiDream-I1 uses a sparse DiT with dual-stream modality processing, single-stream interaction, dynamic MoE routing, and GAN-powered diffusion distillation.
Results
33.82 HPSv2.1: HiDream-I1 achieves the highest average human-preference score among evaluated models and ranks first across four visual styles.
Takeaways & Limitations
Three deployment variants and extensions for instruction-based editing and interactive image agents broaden HiDream-I1 from image generation to interactive visual content creation.
Abstract
from arXiv · showhide
Recent advancements in image generative foundation models have prioritized quality improvements but often at the cost of increased computational complexity and inference latency. To address this critical trade-off, we introduce HiDream-I1, a new open-source image generative foundation model with 17B parameters that achieves state-of-the-art image generation quality within seconds. HiDream-I1 is constructed with a new sparse Diffusion Transformer (DiT) structure. Specifically, it starts with a dual-stream decoupled design of sparse DiT with dynamic Mixture-of-Experts (MoE) architecture, in which two separate encoders are first involved to independently process image and text tokens. Then, a single-stream sparse DiT structure with dynamic MoE architecture is adopted to trigger multi-model interaction for image generation in a cost-efficient manner. To support flexiable accessibility with varied model capabilities, we provide HiDream-I1 in three variants: HiDream-I1-Full, HiDream-I1-Dev, and HiDream-I1-Fast. Furthermore, we go beyond the typical text-to-image generation and remould HiDream-I1 with additional image conditions to perform precise, instruction-based editing on given images, yielding a new instruction-based image editing model namely HiDream-E1. Ultimately, by integrating text-to-image generation and instruction-based image editing, HiDream-I1 evolves to form a comprehensive image agent (HiDream-A1) capable of fully interactive image creation and refinement. To accelerate multi-modal AIGC research, we have open-sourced all the codes and model weights of HiDream-I1-Full, HiDream-I1-Dev, HiDream-I1-Fast, HiDream-E1 through our project websites: https://github.com/HiDream-ai/HiDream-I1 and https://github.com/HiDream-ai/HiDream-E1. All features can be directly experienced via https://vivago.ai/studio.
1. Introduction
HiDream-I1 addresses the computational cost and latency of high-quality image generation with a sparse DiT foundation model, while extending it to scalable variants, editing, and interactive image creation.
- Its sparse DiT uses separate dual-stream processing for image and text tokens, followed by single-stream multimodal interaction with dynamic MoE routing.
- Three variants target different deployment needs: Full prioritizes quality, Dev balances quality and computation in 28 steps, and Fast uses 14 steps for real-time applications.
- HiDream-E1 adds instruction-based image editing, while HiDream-A1 combines generation and editing into an interactive image agent.
- The paper highlights sparse DiT and GAN-powered distillation as mechanisms for combining computational efficiency with image realism, clarity, speed, and quality.
- HiDream-I1 reports leading results in human preference and semantic alignment across HPS, GenEval, and DPG-Bench benchmarks.
2. Data Pre-processing
The data pipeline builds a broad, diverse training corpus through collection, deduplication, safety and quality filtering, and detailed image captioning.
- The pipeline combines systematic data collection, deduplication, multifaceted filtering, and annotation to prepare large-scale training data.
- Data collection aggregates web datasets and internal copyrighted images with accompanying metadata while targeting diverse styles, subjects, resolutions, and aspects.
- A two-stage deduplication process uses SSCD features and k-means clustering before exact intra-cluster similarity searches.
- Approximately 20% of initially collected images were removed through deduplication, reducing data redundancy.
- Filtering removes potentially unsafe, aesthetically poor, and watermarked images using specialized pretrained detectors and scoring models.
- MiniCPM-V 2.6 automatically generates detailed captions from image content and existing metadata, with instructions emphasizing factual visual descriptions and varied lengths.
3. Model Architecture: HiDream-I1
HiDream-I1 uses flow matching with a hybrid text encoder and a sparse DiT that first processes modalities separately, then combines them for multimodal generation.
- Flow matching learns a continuous-time transformation from Gaussian noise to target images by predicting a velocity field conditioned on noisy images, text, and timestep.
- The model architecture contains a hybrid text encoding module and a sparse MoE-enhanced DiT backbone operating on latent representations.
- HiDream-I1 integrates four text-representation sources, including long-context CLIP, T5, and selected intermediate LLM layers, to form textual conditioning.
- Dual-stream DiT blocks independently process latent image tokens and text tokens before their interaction through attention mechanisms.
- After the dual-stream stages, image and text tokens are concatenated and processed jointly by subsequent single-stream transformer blocks.
- Sparse MoE routers dynamically assign tokens to small subsets of specialized FFN experts, replacing standard dense feed-forward networks.
4. Model Training Strategy
HiDream-I1 is trained in stages that first establish latent-space generation and then refine quality, prompt alignment, and user preference.
- Training uses multi-stage Latent Flow Matching, beginning with multi-resolution pre-training and continuing with post-hoc alignment tuning.
- Pre-training optimizes the sparse DiT backbone to model the flow from noise to complex image representations in latent space.
- Refinement uses high-quality human-annotated image-text pairs to improve prompt fidelity, aesthetic quality, and preference alignment.
- The refinement model is fine-tuned for 20,000 steps with learning rate 0.00001 and global batch size 64.
5. Inference Acceleration via GAN-powered Diffusion Model Distillation
HiDream-I1 distills the full model into faster variants that approximate its generation with fewer sampling steps. GAN-powered distillation combines trajectory alignment with adversarial training to preserve perceptual quality and sharpness.
- HiDream-I1-Dev targets 28 sampling steps and HiDream-I1-Fast targets 16, reducing inference steps from the approximately 50 required by HiDream-I1-Full.Both accelerated variants are distilled from the full model.
- DMD trains the student models to approximate the teacher’s generative function by aligning their predicted trajectory distributions.The teacher is HiDream-I1-Full, while the students are HiDream-I1-Dev and HiDream-I1-Fast.
- Adversarial training supplements DMD by having a discriminator assess the realism of images decoded from student-generated latents against real images.The discriminator uses multi-level features extracted by the frozen teacher backbone.
- The final GAN-powered distillation objective is Ltotal = LDMD + λadvLadv.
6. Extension for Image Editing: HiDream-E1
HiDream-E1 extends HiDream-I1 to instruction-based editing by conditioning generation on a source image and textual instruction. Its training uses paired source, instruction, and target images to support visually grounded modifications.
- HiDream-E1 enables modifications to a source image based on a textual editing instruction.
- Source and target images are encoded into VAE latents, spatially concatenated, and used with the instruction to generate the target latent through latent flow matching.The source latent provides visual context for editing.
- A spatially weighted loss emphasizes latent regions where target and source differ, prioritizing edited areas while helping preserve unchanged context.
- The editing model is fine-tuned on 5 million source image, editing instruction, and target image triplets.At inference, users provide the source image and editing instruction.
7. Extension for Image Agent: HiDream-A1
HiDream-A1 is a unified multimodal agent that integrates image generation, image editing, and interactive understanding within a conversational interface. It routes user requests through coordination and planning modules to select the appropriate capability.
- HiDream-A1 integrates text-to-image generation, image editing, and interactive understanding in a conversational AI interface.
- The Coordinator classifies user inputs as Generation or Chat, while the Planner organizes steps for generation tasks.User input can include natural language and visual inputs.
- For generation tasks, the Planner can use HiDream-I1 for image generation or HiDream-E1 for instruction-based image editing.
8. Evaluation
HiDream-I1 and HiDream-E1 are evaluated on prompt adherence, human preference, and instruction-based editing benchmarks. HiDream-I1 leads the reported DPG-Bench, GenEval, and HPSv2.1 results, while HiDream-E1 leads both editing benchmarks.
- Prompt Adherence: HiDream-I1 achieves the highest DPG-Bench overall score of 85.89 for prompt alignment accuracy.It is particularly strong on Relation (93.74) and Other (91.83).
- Prompt Adherence: HiDream-I1 ranks first on GenEval with an overall metric of 0.83.It also achieves 1.00 single-object accuracy and 0.98 two-object accuracy.
- Human Preference: HiDream-I1 achieves the highest average HPSv2.1 score of 33.82 and ranks first across Animation, Concept Art, Painting, and Photo.HPSv2.1 evaluates predicted human preference for image quality across styles.
- Instruction-Based Editing: HiDream-E1 achieves the highest overall average scores on EmuEdit (6.40) and ReasonEdit (7.54).EmuEdit contains 3,589 samples across 10 editing task types, while ReasonEdit contains 197 challenging samples.
9. Conclusion
HiDream-I1 balances state-of-the-art image quality with efficiency, supports scalable deployment through three variants, and extends into interactive visual creation through editing and agent capabilities.
- HiDream-I1 balances state-of-the-art image quality with exceptional efficiency and delivers high-fidelity image synthesis within seconds.
- Three variants support deployment from high-end creative workflows to real-time applications.The variants are HiDream-I1-Full, HiDream-I1-Dev, and HiDream-I1-Fast.
- Experiments on GenEval, DPG-Bench, and HPSv2.1 validate HiDream-I1’s superiority against state-of-the-art approaches.
- HiDream-E1 enables precise instruction-based editing, while HiDream-Agent unifies generation, editing, and visual understanding through a natural-language interface.
A. Contributions and Acknowledgments
The contributor list is organized alphabetically by first name.
- Contributors are listed alphabetically by first name.