PaperlayerView exact revision on arXiv

Source-linked AI summary

HiDream-O1-Image: A Natively Unified Image Generative Foundation Model with Pixel-level Unified Transformer

Qi Cai, Jingwen Chen, Chengmin Gao, Zijian Gong, Yehao Li, Yingwei Pan, Yi Peng, Zhaofan Qiu, Kai Yu, Yiheng Zhang, Hao Ai, Siying Bai, Yang Chen, Zhihui Chen, Fengbin Gao, Ying Guo, Dong Li, Zhen Shen, Leilei Shi, Jing Wang, Siyu Wang, Yimeng Wang, Rui Zheng, Ting Yao, Tao Mei

arXiv:2605.11061v1cs.CVcs.MM

TL;DR

Visual generative models often rely on fragmented encoding modules, leaving open whether pixel-space diffusion can serve as a versatile generalist reasoning framework. HiDream-O1-Image unifies pixels, text, and task conditions in one token space, and experiments report effectiveness across benchmarks and scales, including comparable editing results from its 8B model against larger competitors.

  • Problem

    Whether pixel-space diffusion can scale from a specialized generator into a versatile generalist reasoning framework, requiring structural unification instead of modular pipelines, is an open question.

    Source: §1 · p. 6

  • Method

    HiDream-O1-Image uses an end-to-end Pixel-level Unified Transformer to map image pixels, text, and task conditions into a shared token space.

    Source: §1 · p. 7, §3.2 · p. 11

  • Results

    Experiments across multiple benchmarks confirm effectiveness across scales, with the 8B model's editing results comparable to 16.8B FLUX.1 Kontext and 27B Qwen-Image-Edit.

    Source: §7 · p. 18, §9 · p. 21

  • Takeaways & Limitations

    The conclusion positions HiDream-O1-Image as a versatile generalist framework for diverse synthesis tasks rather than a specialized text-to-image generator.

    Source: §9 · p. 21

Abstract

from arXiv · show

The evolution of visual generative models has long been constrained by fragmented architectures relying on disjoint text encoders and external VAEs. In this report, we present HiDream-O1-Image, a natively unified generative foundation model via pixel-space Diffusion Transformer, that pioneers a paradigm shift from modular architectures to an end-to-end in-context visual generation engine. By mapping raw image pixels, text tokens, and task-specific conditions into a single shared token space, HiDream-O1-Image achieves a structural unification of multimodal inputs within an Unified Transformer (UiT) architecture. This native encoding paradigm eliminates the need for separate VAEs or disjoint pre-trained text encoders, allowing the model to treat diverse generation and editing tasks as a consistent in-context reasoning process. Extensive experiments show that HiDream-O1-Image excels across various generation tasks, including text-to-image generation, instruction-based editing, and subject-driven personalization. Notably, with only 8B parameters, HiDream-O1-Image (8B) achieves performance parity with or even surpasses established state-of-the-art models with significantly larger parameters (e.g., 27B Qwen-Image). Crucially, to validate the immense scalability of this paradigm, we successfully scale the architecture up to over 200B parameters. Experimental results demonstrate that this massive-scale version HiDream-O1-Image-Pro (200B+) unlocks unprecedented generative capabilities and superior performance, establishing new state-of-the-art benchmarks. Ultimately, HiDream-O1-Image highlights the immense potential of natively unified architectures and charts a highly scalable path toward next-generation multimodal AI.

1. Introduction

HiDream-O1-Image unifies raw pixels, text, and task conditions in a shared token space, treating diverse visual generation and editing tasks as in-context reasoning. The paper also introduces a reasoning-driven prompt agent and reports broad task coverage and scalability beyond 200B parameters.

1. Introduction

  • Existing latent diffusion models compress images through VAEs and use separate language encoders, creating information bottlenecks that can discard high-frequency visual detail.Pixel-space DiTs preserve raw-image detail but generally retain disjoint text encoders and are specialized mainly for text-to-image generation, limiting semantic alignment and broader task generalization.Source: §1 · p. 6
  • A reasoning-driven prompt agent refines complex user instructions before generation to improve generalizability and instruction following, particularly on tasks requiring logical deduction.The paper says the agent is open-source.Source: §1 · p. 7
  • The model supports image, text, and additional condition or reference inputs for text-to-image generation, instruction-based editing, and subject-driven personalization, including outputs up to 2,048 × 2,048.The paper also describes cinematic shots, artistic styles, long text rendering, and storyboard multi-panel generation as supported scenarios.Source: §1 · p. 7
  • HiDream-O1-Image maps raw pixels, text tokens, and task conditions into one shared token space, replacing external VAEs and disjoint text encoders with an end-to-end Pixel-level Unified Transformer.This framing treats generation and editing as a consistent in-context visual reasoning process rather than separate tasks handled by specialized modules.Source: §1 · pp. 6–7
  • The paper reports that the 8B model matches or surpasses larger open-source models such as 27B Qwen-Image and leading commercial models, and scales the architecture beyond 200B parameters.At 200B+, it reports superior generative capabilities, visual fidelity, and reasoning, with new state-of-the-art benchmarks across a wide range of generation tasks.Source: §1 · p. 7

2. Data Curation and Prompt Construction

HiDream-O1-Image’s data engine converts heterogeneous sources into image-text pairs, editing triplets, and subject-reference samples through collection, deduplication, quality and safety filtering, and VLM-based prompt construction. The curation broadens coverage across generation, editing, personalization, and multi-panel tasks while tailoring prompts to each format.

2. Data Curation and Prompt Construction

  • The data engine converts heterogeneous raw sources into image-text pairs, editing triplets, and subject-reference samples through collection, deduplication, filtering, and VLM-based prompt construction.The pipeline is summarized in Figure 6.Source: §2 · p. 8

2.1. Source Data Collection

  • Text-to-image sources include graphic layouts, long-text images, and diverse style-oriented samples to support typography, structured composition, and varied visual aesthetics.Source: §2.1 · p. 8
  • Editing, personalization, and multi-panel data cover before-after changes, identity consistency across varied references, and sequential or panelwise composition.Sources include public editing datasets, synthesized pairs, video-derived samples, person and object reference sets, Internet grid images, and video frame compositions.Source: §2.1 · p. 8

2.2. Data Deduplication

  • Deduplication uses SSCD features, 16,000 k-means clusters fitted on 2 million samples, and within-cluster Faiss nearest-neighbor search to retain one representative per near-duplicate group.The method addresses the infeasibility of pairwise comparison across the full corpus (45); it uses SSCD (38) and Faiss (10).Source: §2.2 · p. 9
  • Approximately 20% of initial candidates are removed by redundancy control while semantic coverage is preserved.Source: §2.2 · p. 9

2.3. Data Quality and Safety Filtering

  • Filtering removes unsafe, low-aesthetic, watermarked, technically poor, or task-inconsistent samples using complementary classifiers, scoring models, detectors, and VLM checks.Editing pairs are checked for meaningful visual changes, IP groups for identity matches, and technical quality using Top-IQ (3) and JPEG bytes-per-pixel estimates; tools include an NSFW classifier (26), aesthetic model (25), and watermark detector (27).Source: §2.3 · p. 9

2.4. Prompt Construction

  • Qwen3-VL (1) turns metadata and visual-textual signals into prompts tailored to natural images, layouts, editing, identity personalization, and multi-panel samples, with short, medium, and detailed variants.Prompts preserve salient visual attributes and layout text; editing instructions describe intended changes while preserving other regions, and personalization prompts retain identity-defining appearance. Samples are dropped when prompting fails or produces preset sensitive terms.Source: §2.4 · pp. 9–10

3. Model Architecture: HiDream-O1-Image

HiDream-O1-Image is a pixel-space Unified Transformer instantiated at 8B and 200B+ parameters to combine high-fidelity image synthesis with in-context reasoning.

3.1. Reasoning-Driven Prompt Agent

  • A Gemma (16)-based prompt agent reasons through layout, attributes, physical logic, and context before refining complex user instructions into clearer textual conditions for generation and editing.It generates a chain of thought before producing the final prompt.Source: §3.1 · p. 11

3.2. Unified Multimodal Tokenization

  • Refined text prompts, task-specific conditions, and noisy target-image patches are mapped into one token space, jointly reasoned over by unified Transformer blocks, and decoded into clean image patches.Text uses the backbone vocabulary, condition images use SigLip-2 (49) features, and target images are patch-embedded after noise interpolation.Source: §3.2 · pp. 11–12

3.3. Unified Transformer (UiT) Architecture

  • A decoder-only Transformer backbone supports 8B and 200B+ variants; the 8B model starts from Qwen3-VL-8B-Instruct (1), while patch embeddings enable pixel-space diffusion at both scales.Both variants use RMSNorm (62), SwiGLU (44), RoPE (46), and a specialized timestep token.Source: §3.3 · p. 12
  • Condition and text tokens use causal masking, while generation tokens use full attention to aggregate global context for spatially coherent image synthesis.This hybrid attention preserves autoregressive structure for language modeling while supporting image synthesis.Source: §3.3 · p. 12

3.4. Overall Objective

  • The objective couples flow matching for image prediction with LPIPS (63) and perceptual DINO losses to balance structural regression with perceptual alignment in pixel-space synthesis.The perceptual constraints address the difficulty of modeling long-range semantic coherence with pixel-space diffusion.Source: §3.4 · p. 12

4. Model Training

HiDream-O1-Image is trained through progressive, resolution-increasing generalist pre-training, followed by supervised fine-tuning and preference-based reinforcement learning to refine reasoning and generation quality.

4.1. Progressive Generalist Pre-training

  • Three-stage pre-training progresses from 512 × 512 foundational alignment to 1,024 × 1,024 in-context learning and ultra-high-resolution refinement, while preserving image aspect ratios.Stage I jointly trains text-to-image generation, language modeling, and multimodal understanding; Stage II adds in-context generation and editing, including personalization; Stage III targets fine detail and perceptual quality on images exceeding 2,048 × 2,048.Source: §4.1 · p. 13

4.2. Post-training

  • Supervised fine-tuning uses several hundred thousand curated samples and reasoning trajectories, with uniform timestep sampling to emphasize late-stage denoising.The data target compositional coherence, lighting consistency, photorealism, and stylistic fidelity, while reasoning trajectories fine-tune the Prompt Agent to produce structurally aligned, unambiguous prompts.Source: §4.2 · p. 13
  • RLHF applies GRPO (30) to align generation with human preferences using aggregated OCR, aesthetic, instruction-following, and reasoning rewards.The composite objective targets photorealism, aesthetic quality, text-rendering accuracy, semantic consistency, and logical reasoning while suppressing artifacts.Source: §4.2 · pp. 13–14

5. Adversarial Diffusion Distillation for Fast Inference

HiDream-O1-Image-Dev accelerates inference with a 28-step sampler distilled from the full model using DMD, diffusion supervision, and adversarial learning to preserve generative dynamics and perceptual fidelity.

Adversarial Diffusion Distillation for Fast Inference

  • The full HiDream-O1-Image model typically uses around 50 denoising steps, while HiDream-O1-Image-Dev uses a 28-step sampler for faster generation.The shorter trajectory is intended to improve efficiency in latency-sensitive scenarios.Source: §5 · p. 14
  • DMD (61) trains the student to align its reduced-step trajectory distribution with the full model while inheriting the teacher’s main generative dynamics.The student approximates the teacher’s generation behavior under fewer sampling steps.Source: §5 · pp. 14–15
  • Standard diffusion loss provides auxiliary supervision that improves training stability and mitigates optimization oscillation during distillation.Source: §5 · p. 15
  • Adversarial distillation pairs the student generator with a discriminator comparing real and reconstructed pixel-space images, guided by multilevel features from the frozen teacher backbone.The total objective is Ltotal = LDMD + 𝜆diffLdiff + 𝜆advLadv.Source: §5 · p. 15

6. Performance Comparisons for Text-to-Image Generation

HiDream-O1-Image performs strongly in general text-to-image synthesis and fine-grained text rendering, while supporting cinematic, artistic, and storyboard-generation scenarios. Its pixel-space approach is presented as enabling precise text-image alignment and controllable visual anchors for video workflows.

6.1. General Text-to-Image Synthesis

  • The 8B model surpasses similar-scale open-source counterparts, while HiDream-O1-Image-Pro (200B+) outperforms leading closed-source models, including GPT Image 2 and Seedream-4.0.The comparisons cover GenEval (14), DPG (19), and HPSv3 (33).Source: §6.1 · pp. 15–16, Table 2 · p. 16, Table 3 · p. 16

6.2. High-Fidelity Text Rendering

  • HiDream-O1-Image attains the highest scores across most CVTG-2K metrics; on LongText-Bench (13), its 8B model performs comparably to the best competitor, 27B Qwen-Image, while its 200B+ model establishes a new state of the art in ultra-long text rendering.Source: §6.2 · pp. 16–17
  • Pixel-space generation avoids disjoint-modality translation bottlenecks and lossy VAE compression, supporting precise text-image alignment and mitigating structural distortions in rendered text.The paper attributes these benefits to its end-to-end pixel-space generative framework.Source: §6.2 · pp. 17–18

6.3. Versatility Across Diverse Generation Scenarios

  • HiDream-O1-Image handles cinematic shots, artistic styles, complex long text, and coherent multi-panel storyboards in a single inference pass.The paper illustrates these scenarios in Figure 9.Source: §6.3 · p. 18
  • For video workflows, the model supports 15 cinematic shots and perspectives at the initial-keyframe stage, providing controllable spatial and camera composition that streamlines subsequent video synthesis.The paper frames high-quality static images as foundational keyframes for downstream video generation.Source: §6.3 · p. 18

7. Performance Comparisons for Image Editing

The section compares HiDream-O1-Image on GEdit and ImgEdit, reporting comparable results to larger competitors at 8B parameters and state-of-the-art performance from the 200B+ model on complex edits.

7. Performance Comparisons for Image Editing

  • Across both benchmarks and model scales, HiDream-O1-Image shows strong instruction following and high generation fidelity, while preserving unedited regions.Source: §7 · pp. 18, 20
  • Shared visual-linguistic tokens enable precise semantic-to-spatial grounding, while joint language-modeling and multimodal-understanding objectives support intent interpretation and preservation of original visual context.The paper links these mechanisms to detail fidelity and an effective in-context generation process.Source: §7 · p. 18
  • On GEdit and ImgEdit, the 8B model achieves comparable results against larger competitors, while the 200B+ model sets a new state of the art for complex, fine-grained edits.The reported comparisons cover models including 16.8B FLUX.1 Kontext and 27B Qwen-Image-Edit.Source: §7 · pp. 18, 20, Table 6 · p. 20, Table 7 · p. 20

8. Performance Comparisons for Subject-driven Personalization

HiDream-O1-Image is evaluated on multi-subject personalization, where its shared token space is presented as a driver of stronger subject preservation and composition. The reported gains extend to settings with up to 11 subjects.

8. Performance Comparisons for Subject-driven Personalization

  • UniSubject contains 300 cases and 1.8K subjects, pairing one human with 1–10 reference objects and a compositional prompt; evaluation uses Q-PF, Q-SC, Overall Score, and HPSv3 preference.Q-PF measures prompt alignment, Q-SC measures pairwise subject preservation, and Overall Score is the mean of Q-PF and Q-SC.Source: §8 · p. 20
  • HiDream-O1-Image (8B) raises Q-O from 7.19 to 7.50 for 4–8 subjects and from 6.73 to 7.48 for 9–11 subjects, while the 200B+ model pushes performance further.The 200B+ model demonstrates superior multi-subject compositionality and identity preservation in extreme scenarios.Source: Table 8 · p. 21, §8 · p. 21
  • The shared token space anchors textual concepts to fine-grained visual prompt tokens, forming a fused subject representation that mitigates instruction–reference interference as subject counts increase.The paper attributes consistent performance boosts in the 4–8 and 9–11 subject settings to this mechanism.Source: §8 · p. 21
  • Figure 10 shows HiDream-O1-Image preserving multiple subjects better than Qwen-Image in three subject-driven personalization examples.Source: §8 · p. 21

9. Conclusions · Appendix

HiDream-O1-Image unifies pixels, text, and task conditions in one shared token space, enabling diverse synthesis tasks through a consistent in-context process. The report finds strong results at 8B parameters and new state-of-the-art records at 200B+.

9. Conclusions

  • Across GenEval, CVTG-2K, GEdit, and UniSubject, the 8B model is competitive or superior to larger latent-space DiTs, while the 200B+ model establishes new state-of-the-art records.The conclusion reports effectiveness across multiple benchmarks and scales.Source: §9 · p. 21
  • The model maps raw pixels, text, and task conditions into one shared token space, removing separate VAE and text-encoder boundaries.This unification supports a consistent in-context generation process and a generalist framework for diverse synthesis tasks.Source: §9 · p. 21
  • Figure 10 compares HiDream-O1-Image with Qwen-Image on subject-driven customized generation.The supplied caption identifies the comparison's models and task but does not report its outcome.Source: Figure 10 · p. 22

Related papers

Loading 2605.11061v1…