PaperlayerView exact revision on arXiv

Source-linked AI summary

HiDream-I1: A High-Efficient Image Generative Foundation Model with Sparse Diffusion Transformer

Qi Cai, Jingwen Chen, Yang Chen, Yehao Li, Fuchen Long, Yingwei Pan, Zhaofan Qiu, Yiheng Zhang, Fengbin Gao, Peihan Xu, Yimeng Wang, Kai Yu, Wenxuan Chen, Ziwei Feng, Zijian Gong, Jianzhuang Pan, Yi Peng, Rui Tian, Siyu Wang, Bo Zhao, Ting Yao, Tao Mei

arXiv:2505.22705v1cs.CVcs.MM

TL;DR

Many state-of-the-art image generators face high computational complexity and long inference times, challenging real-time, cost-effective deployment. HiDream-I1 combines dual-stream processing of image and text tokens with single-stream multimodal interaction, using dynamic Mixture-of-Experts in both sparse Diffusion Transformer stages. Experiments on GenEval, DPG-Bench, and HPSv2.1 validate HiDream-I1’s superiority against state-of-the-art approaches; it achieves the highest average HPSv2.1 score (33.82) among evaluated models.

  • Problem

    Many state-of-the-art image generators face high computational complexity and long inference times, challenging real-time, cost-effective deployment.

    Source: §1 · p. 4

  • Method

    HiDream-I1 combines dual-stream processing of image and text tokens with single-stream multimodal interaction, using dynamic Mixture-of-Experts in both sparse Diffusion Transformer stages.

    Source: §1 · p. 4

  • Results

    Experiments on GenEval, DPG-Bench, and HPSv2.1 validate HiDream-I1’s superiority against state-of-the-art approaches; it achieves the highest average HPSv2.1 score (33.82) among evaluated models.

    Source: §8.2 · p. 14, §9 · p. 15

  • Takeaways & Limitations

    Three variants target different deployment needs, from quality-focused workflows to real-time image generation.

    Source: §1 · p. 4

Abstract

from arXiv · show

Recent advancements in image generative foundation models have prioritized quality improvements but often at the cost of increased computational complexity and inference latency. To address this critical trade-off, we introduce HiDream-I1, a new open-source image generative foundation model with 17B parameters that achieves state-of-the-art image generation quality within seconds. HiDream-I1 is constructed with a new sparse Diffusion Transformer (DiT) structure. Specifically, it starts with a dual-stream decoupled design of sparse DiT with dynamic Mixture-of-Experts (MoE) architecture, in which two separate encoders are first involved to independently process image and text tokens. Then, a single-stream sparse DiT structure with dynamic MoE architecture is adopted to trigger multi-model interaction for image generation in a cost-efficient manner. To support flexiable accessibility with varied model capabilities, we provide HiDream-I1 in three variants: HiDream-I1-Full, HiDream-I1-Dev, and HiDream-I1-Fast. Furthermore, we go beyond the typical text-to-image generation and remould HiDream-I1 with additional image conditions to perform precise, instruction-based editing on given images, yielding a new instruction-based image editing model namely HiDream-E1. Ultimately, by integrating text-to-image generation and instruction-based image editing, HiDream-I1 evolves to form a comprehensive image agent (HiDream-A1) capable of fully interactive image creation and refinement. To accelerate multi-modal AIGC research, we have open-sourced all the codes and model weights of HiDream-I1-Full, HiDream-I1-Dev, HiDream-I1-Fast, HiDream-E1 through our project websites: https://github.com/HiDream-ai/HiDream-I1 and https://github.com/HiDream-ai/HiDream-E1. All features can be directly experienced via https://vivago.ai/studio.

1. Introduction

State-of-the-art image generation has faced a trade-off between quality and computational cost; HiDream-I1 is an open-source 17-billion-parameter model designed to deliver high-quality images within seconds.

Introduction

  • HiDream-I1 combines dual-stream image and text processing with single-stream multimodal interaction, using dynamic MoE routing in both stages to balance generation quality and computational efficiency.The sparse DiT design routes data through specialized experts based on input characteristics.Source: §1 · p. 4
  • HiDream-I1 offers Full, Dev, and Fast variants using over 50, 28, and 14 diffusion steps, respectively, for different quality, speed, and computational-demand needs.Full targets utmost image quality; guidance-distilled Dev balances quality and computational demand; Fast is aimed at real-time applications.Source: §1 · p. 4
  • HiDream-E1 extends HiDream-I1 with image-conditioned, natural-language instruction editing, while HiDream-A1 combines generation and interactive editing in a conversational image agent.The introduction says HiDream-A1 also integrates image understanding and is intended to let users create and edit without switching platforms or adjusting complex parameters.Source: §1 · pp. 4–5
  • GAN-powered diffusion distillation is presented as a way to improve speed while preserving image detail, sharpening edges, realism, and clarity.The approach incorporates generative adversarial learning to address details and edges commonly sacrificed in typical diffusion model distillation.Source: §1 · p. 4
  • HiDream-I1 is reported to achieve state-of-the-art performance on HPSv2.1 across styles and top semantic relevance on GenEval and DPG-Bench.The introduction describes HPSv2.1 as assessing semantic relevance, image quality, and aesthetics, and characterizes GenEval and DPG-Bench as evaluating prompt-image semantic relevance.Source: §1 · p. 5

2. Data Pre-processing

HiDream-I1’s data-preprocessing pipeline combines broad data collection, deduplication, filtering, and annotation to curate training data.

2.1. Data Collection

  • The collection strategy combined web-sourced images and accompanying text with internal copyrighted images, aiming for breadth across styles, subjects, resolutions, and aspect ratios.Source: §2.1 · p. 5

2.2. Data Deduplication

  • A two-stage process used SSCD features and k-means clustering, then Faiss similarity searches within clusters; it removed approximately 20% of initially collected images.The process targeted exact and visually similar duplicates while avoiding computationally prohibitive all-pairs comparisons.Source: §2.2 · p. 6

2.3. Data Filtering

  • Filtering removed potentially inappropriate, low-aesthetic, watermarked, low-Top-IQ, and unusually low-bytes-per-pixel images.The filters used an NSFW classifier, aesthetic scoring model, watermark detector, Top-IQ metric, and JPEG-based bytes-per-pixel checks.Source: §2.3 · p. 6

2.4. Data Annotation

  • MiniCPM-V 2.6 generated image captions using image content and existing metadata as context, with instructions to describe visual details objectively and limit hallucination.A prompting strategy encouraged varied caption lengths to reflect typical user prompts.Source: §2.4 · pp. 6–7

3. Model Architecture: HiDream-I1

HiDream-I1 generates images in latent space with a flow-matching objective and a sparse DiT backbone that combines hybrid text conditioning with staged image–text processing.

3. Model Architecture: HiDream-I1

  • The model predicts a velocity field from noisy image latents, text embeddings, and timestep, and minimizes mean squared error between predicted and target velocities.It uses a linear interpolation path from Gaussian noise to the target image, with constant velocity equal to the target minus the noise sample.Source: §3 · p. 7

3.1. Hybrid Text Encoding

  • Four text sources—Long-Context CLIP, T5-XXL, and intermediate Llama 3.1 8B Instruct layers—provide pooled global and concatenated sequence conditioning for the DiT.T5 and selected LLM-layer sequences are projected and concatenated into the primary text-conditioning sequence; pooled CLIP features provide global conditioning.Source: §3.1 · p. 8

3.2. Sparse Diffusion Transformer (DiT) Backbone

  • The backbone first processes patchified image latents and text tokens in parallel dual-stream blocks, then concatenates their outputs for subsequent single-stream blocks.Sparse MoE feed-forward computation is used in both stages.Source: §3.2 · p. 8
  • Sparse MoE routers dynamically send each token to a small subset of specialized FFN experts, including a shared expert, expanding capacity while maintaining computational efficiency relative to equally sized dense models.The expert FFNs use SwiGLU Shazeer (2020).Source: §3.2 · p. 9
  • Pooled Long-CLIP features and sinusoidal timestep embeddings condition every transformer block through adaLN, while QK-normalization in self-attention enhances training stability.adaLN modulates token scale and shift parameters.Source: §3.2 · p. 9

4. Model Training Strategy

HiDream-I1 uses Latent Flow Matching in a multi-stage training strategy, first building generative capabilities in compressed latent space and then refining quality, alignment, and user preference.

4. Model Training Strategy

  • Training comprises multi-resolution pre-training followed by post-hoc alignment tuning.The pre-training phase builds foundational generative capabilities in compressed latent space; refinement then targets quality, alignment, and user preference.Source: §4 · p. 9

4.1. Pre-training Stage

  • The pre-training stage trains the sparse DiT backbone to model flows from noise to complex latent-space image representations.Training progresses through image resolutions up to 256 × 256 for 600,000 steps, 512 × 512 for 200,000 steps, and 1,024 × 1,024 for 200,000 steps, with batch sizes of 24, 8, and 2 per GPU, respectively; images maintain their aspect ratio.Source: §4.1 · p. 9

4.2. Post-Training Stage

  • Post-training fine-tunes the model on high-quality, human-annotated image-text pairs to improve prompt fidelity, aesthetic quality, and preference alignment.The images have higher aesthetic quality and rich structure, and human annotators verify text-image alignment; fine-tuning runs for 20,000 steps at learning rate 0.00001 with global batch size 64.Source: §4.2 · pp. 9–10

5. Inference Acceleration via GAN-powered Diffusion Model Distillation

HiDream-I1-Full’s roughly 50-step generation is distilled into faster Dev and Fast variants. The distillation combines trajectory-distribution matching with adversarial training to support stable, sharp image generation with fewer steps.

5. Inference Acceleration via GAN-powered Diffusion Model Distillation

  • HiDream-I1-Dev targets 28 sampling steps and HiDream-I1-Fast targets 16, compared with approximately 50 steps for HiDream-I1-Full.The reduced-step variants are created through knowledge distillation.Source: §5 · p. 10
  • DMD (Yin et al., 2024) aligns the student’s predicted trajectory distribution with the teacher’s to enable stable generation with fewer sampling steps.HiDream-I1-Full is the teacher, and HiDream-I1-Dev and HiDream-I1-Fast are the student models.Source: §5 · p. 10
  • Adversarial training supplements DMD to maintain perceptual quality and sharpness, using a discriminator that assesses decoded student images against real images.The discriminator uses multi-level features from the frozen teacher backbone; the total objective is Ltotal = LDMD + λadvLadv.Source: §5 · p. 10

6. Extension for Image Editing: HiDream-E1

HiDream-E1 extends HiDream-I1 to instruction-based image editing, generating modifications to a source image from a text instruction. It uses source-image conditioning and instruction-guided training to produce edits while preserving unchanged context.

6. Extension for Image Editing: HiDream-E1

  • HiDream-E1 extends HiDream-I1 to edit a source image according to a textual instruction.At inference, users provide the source image and editing instruction, and the model generates the modified image.Source: §6 · p. 10
  • Source and target images are encoded as VAE latents, concatenated spatially, and used to train generation of the target latent conditioned on the source latent and instruction.Training uses latent flow matching, with the source latent providing visual context.Source: §6 · p. 10
  • A spatially weighted loss emphasizes latent regions that differ between target and source, prioritizing edited areas while helping preserve unchanged visual context.The loss assigns greater weight to errors in regions where the target latent significantly differs from the source latent.Source: §6 · p. 10
  • HiDream-E1 was trained by fine-tuning pre-trained HiDream-I1 on 5 million source-image, editing-instruction, and target-image triplets.At inference, the user supplies a source image and editing instruction.Source: §6 · p. 10

7. Extension for Image Agent: HiDream-A1

HiDream-A1 is a conversational multimodal agent that combines image generation, image editing, and interactive understanding through natural-language dialogue.

7. Extension for Image Agent: HiDream-A1

  • HiDream-A1 integrates text-to-image generation, image editing, and interactive understanding in a conversational AI interface for visual creation and manipulation.Users can provide natural-language or visual input.Source: §7 · p. 12
  • A Coordinator manages the workflow and routes requests to either Generation or Chat.For Generation, a Planner determines the steps needed to fulfill the request.Source: §7 · p. 12
  • For generation and instruction-based editing, the Planner can use HiDream-I1 and HiDream-E1, respectively.Figure 4 shows the agent interface and examples of both tasks.Source: §7 · p. 12

8. Evaluation

The evaluation measures HiDream-I1’s prompt adherence and human preference, and HiDream-E1’s instruction-following and preservation of untargeted image regions.

8. Evaluation

8.1. Prompt Adherence Evaluation of HiDream-I1

  • HiDream-I1 leads GenEval at 0.83 overall, with best scores in single-object (1.00), two-object (0.98), counting (0.79), color (0.91), and color attribution (0.72).The benchmark targets challenging compositional generation tasks (Ghosh et al., 2023).Source: Table 2 · p. 13, §8.1 · p. 13

8.2. Human Preference Evaluation of HiDream-I1

  • HiDream-I1 achieves the highest average HPSv2.1 score (33.82), ranking first in Animation (35.05), Concept Art (33.74), Painting (33.88), and Photo (32.61).HPSv2.1 predicts human judgments of image quality across styles (Wu et al., 2023).Source: Table 3 · p. 13, §8.2 · pp. 13–14

8.1. Prompt Adherence Evaluation of HiDream-I1

  • HiDream-I1 leads DPG-Bench with an overall score of 85.89, including Relation 93.74 and Other 91.83, alongside competitive entity and attribute generation.Source: §8.1 · p. 13

8.3. Evaluation of HiDream-E1

  • HiDream-E1 leads EmuEdit (6.40 average) and ReasonEdit (7.54), demonstrating strong performance on broad editing tasks and challenging instructions.It tops six EmuEdit categories: Global edits (5.32), Text manipulation (6.45), Color adjustments (7.57), Style transfer (6.49), Object removal (5.99), and Local edits (6.35).Source: Table 4 · p. 14, §8.3 · p. 14

9. Conclusion · Appendix

HiDream-I1 is presented as an open-source sparse-DiT model balancing image quality and efficiency, with variants for different deployment needs. Its extensions add instruction-based editing and interactive visual creation.

9. Conclusion

  • HiDream-I1 uses a sparse Diffusion Transformer to produce high-fidelity images within seconds while balancing quality and efficiency.The conclusion characterizes it as a powerful open-source image generative foundation model.Source: §9 · p. 14
  • HiDream-I1-Full, HiDream-I1-Dev, and HiDream-I1-Fast support deployment from high-end creative workflows to real-time applications.The variant list spans the end of one passage and the beginning of the next.Source: §9 · pp. 14–15
  • Experiments on GenEval, DPG-Bench, and HPSv2.1 validate HiDream-I1’s superiority over state-of-the-art approaches.Source: §9 · p. 15
  • HiDream-E1 extends HiDream-I1 with precise, instruction-based image editing.Source: §9 · p. 15
  • HiDream-Agent unifies image generation, editing, and visual understanding through a natural-language interface.Source: §9 · p. 15
  • Together, HiDream-I1, HiDream-E1, and HiDream-Agent form a multimodal framework for interactive visual content creation.Source: §9 · p. 15

Related papers

Loading 2505.22705v1…