Source-linked AI summary
DreamO: A Unified Framework for Image Customization
Chong Mou, Yanze Wu, Wenxu Wu, Zinan Guo, Pengze Zhang, Yufeng Cheng, Yiming Luo, Fei Ding, Shiwen Zhang, Xinghui Li, Mengtian Li, Mingcong Liu, Yi Zhang, Shaojin Wu, Songtao Zhao, Jian Zhang, Qian He, Xinglong Wu
TL;DR
Existing image-customization methods are often task-specific, leaving unified multi-condition customization an open challenge. DreamO builds a unified framework on a pre-trained DiT using routing constraints, placeholder conditioning, large-scale multi-task data, and progressive training. Experiments show high-quality performance across diverse customization tasks and flexible integration of different control conditions.
Problem
Existing image-customization approaches are predominantly task-specific, limiting unified integration of different condition types.
Method
DreamO embeds heterogeneous conditions into a pre-trained DiT using feature routing, placeholders, large-scale multi-task training, and progressive optimization.
Results
DreamO performs a wide range of image customization tasks with high-quality results and flexible multi-condition control.
Takeaways & Limitations
A single pre-trained DiT framework can support diverse condition types while maintaining high-fidelity consistency and disentangling heterogeneous control signals.
Abstract
from arXiv · showhide
Recently, extensive research on image customization (e.g., identity, subject, style, background, etc.) demonstrates strong customization capabilities in large-scale generative models. However, most approaches are designed for specific tasks, restricting their generalizability to combine different types of condition. Developing a unified framework for image customization remains an open challenge. In this paper, we present DreamO, an image customization framework designed to support a wide range of tasks while facilitating seamless integration of multiple conditions. Specifically, DreamO utilizes a diffusion transformer (DiT) framework to uniformly process input of different types. During training, we construct a large-scale training dataset that includes various customization tasks, and we introduce a feature routing constraint to facilitate the precise querying of relevant information from reference images. Additionally, we design a placeholder strategy that associates specific placeholders with conditions at particular positions, enabling control over the placement of conditions in the generated results. Moreover, we employ a progressive training strategy consisting of three stages: an initial stage focused on simple tasks with limited data to establish baseline consistency, a full-scale training stage to comprehensively enhance the customization capabilities, and a final quality alignment stage to correct quality biases introduced by low-quality data. Extensive experiments demonstrate that the proposed DreamO can effectively perform various image customization tasks with high quality and flexibly integrate different types of control conditions.
1 INTRODUCTION
Image customization has advanced across identity, appearance, try-on, and style, but existing methods are largely task-specific. DreamO addresses this gap with a unified framework for diverse conditions and multi-condition interactions.
- Image customization targets consistency in attributes including identity, object appearance, virtual try-on, and style, yet unified customization remains challenging.
- DreamO uses a pre-trained DiT and a single model to support identity, subject, try-on, style, and interactions among condition types.The framework follows a unified sequence-conditioning format and trains with additional parameters.
- Feature routing constrains DiT representations to preserve content fidelity and disentangle different control conditions.
- A placeholder strategy links textual descriptions to condition images and controls condition placement in generated results.
- Progressive training supports convergence in multi-task and complex-task settings, while large-scale data broadens customization capabilities.
- Extensive experiments report high-quality customization across varied tasks and strong flexibility for multi-condition scenarios.
2 RELATED WORKS
Diffusion models have expanded controllable image generation from text and spatial conditions to image-based identity and style control. Recent DiT-based approaches unify inputs and tasks, while routing constraints exploit spatial-semantic correspondences for more precise condition influence.
- DiT architectures have emerged as scalable diffusion-model choices with improved performance over earlier UNet-based designs.
- Diffusion models support text conditioning, spatial control through added modules, and unspatial control such as identity and style through image prompts.
- Recent DiT-based methods concatenate text, image, and condition tokens, then apply task-specific LoRA training or multi-stage full-model optimization.
- Cross-attention maps establish spatial-semantic correspondence between linguistic tokens and generated regions, motivating routing constraints for identity and subject generation.
3 METHOD
DreamO builds unified image customization on a DiT model by processing image, text, and condition tokens together, while routing constraints and progressive training support fidelity, condition disentanglement, and multi-task convergence.
- Unified DiT framework: DreamO reuses Flux’s VAE to encode flexible-resolution condition images, concatenates image, text, and condition tokens, and feeds them into Flux.Higher resolutions are recommended for detail-rich images, while lower resolutions reduce compression costs for simpler images.
- Routing constraint: A feature-routing constraint uses condition-to-image cross-attention to focus each reference image on its target subject and improve reference consistency.The dense similarity map is averaged across condition tokens, and an MSE loss compares the resulting response with the target subject mask.
- Routing constraint: Placeholder-to-image routing associates each condition image with a corresponding placeholder, establishing textual-to-condition correspondences in multi-condition prompts.The intended similarity is 1 for the matching placeholder-condition pair and 0 for all other pairs.
- Routing constraint: The final objective combines diffusion, image-routing, and placeholder-routing losses, while normal text without placeholders is introduced with 50% probability.The routing losses add little reported training overhead: 2.5s/iter versus 3s/iter.
- Results: Despite separate task-specific data construction, DreamO develops an ID-and-try-on combination capability absent from the training data.The paper also reports high-fidelity identity injection across varied scenes compared with PhotoMaker, InstantID, and PuLID.
- Progressive training: Progressive training first establishes subject-driven consistency, then tunes on all data; direct full-data training otherwise makes convergence difficult and can shift Flux’s generation prior.The first stage uses Subject200K data generated by Flux, which facilitates fast convergence; the second stage performs full-data tuning across subtasks.
4 EXPERIMENT
DreamO is evaluated across identity, subject, virtual try-on, and style customization, including single- and multi-condition settings. The experiments report strong fidelity, text alignment, and image quality, while ablations examine routing constraints and progressive training.
- DreamO injects identity information with high fidelity across varied scenes while retaining customization flexibility.
- DreamO achieves higher subject fidelity and better text consistency than single-task and unified baselines in single- and multi-subject scenarios.
- DreamO maintains both clothing fidelity to references and text alignment in virtual try-on, unlike a comparison method that loses text alignment.
- DreamO shows weaker content intrusion, better text alignment, and higher style fidelity than recent style-customization methods.
- Quantitative evaluations report strong face similarity, subject consistency, text following, try-on performance, and style consistency across task-specific benchmarks.Identity uses Face-Sim and CLIP similarity; subject evaluation uses CLIP and Dino similarity, while try-on and style use task-specific similarity measures.
- A user study across style, object, identity, and try-on customization finds DreamO performs better in text alignment, reference alignment, and image quality.Twenty volunteers rated six test samples per task on a 0–5 satisfaction scale.
- Removing routing constraints degrades reference fidelity and causes condition coupling, while placeholder-to-image routing enables individual control of specific reference objects.The placeholder routing ablation specifically evaluates whether placeholders bind to their corresponding images during multi-subject customization.
5 CONCLUSION
DreamO unifies diverse image-customization conditions within a single pre-trained DiT architecture. Comprehensive experiments show high-quality performance across a wide range of customization tasks.
- DreamO supports identity, style, subject, and try-on customization within one pre-trained DiT architecture.
- Comprehensive experiments demonstrate high-quality results across a wide range of image-customization tasks.
RES
Figure 13 presents DreamO's capability in style-driven image customization. The figure focuses specifically on style-driven customization results.
- Figure 13 presents DreamO's style-driven image customization capability.
- The figure focuses on customization driven by image style.
- DreamO's customization capability is illustrated specifically for style-driven generation.