Source-linked AI summary
OminiControl: Minimal and Universal Control for Diffusion Transformer
Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, Xinchao Wang
TL;DR
Existing image-conditioning methods either add substantial architectural complexity or remain biased toward particular control tasks, especially when applied to DiT models. OminiControl reuses DiT components with unified token processing, adaptive positioning, and Subjects200K data, and experiments report effectiveness across aligned and non-aligned tasks with only 0.1% additional parameters.
Problem
Existing image-control methods require substantial architectural modifications, usually favor either spatially aligned or non-aligned tasks, and perform suboptimally on DiT models.
Method
OminiControl reuses DiT’s VAE encoder and transformer blocks, concatenates condition and noisy image tokens, and adapts position indices to task requirements.
Results
0.1% additional parameters supports effective control across spatially aligned and non-aligned tasks, with reported improvements over specialized methods on evaluated tasks.
Takeaways & Limitations
Effective and flexible image-conditional control in DiT can be achieved with minimal architectural overhead, supported by the over-200,000-image Subjects200K dataset.
Takeaways & Limitations
Joint attention does not inherently provide adjustable conditioning strength, requiring an attention-bias mechanism for inference-time control.
Abstract
from arXiv · showhide
We present OminiControl, a novel approach that rethinks how image conditions are integrated into Diffusion Transformer (DiT) architectures. Current image conditioning methods either introduce substantial parameter overhead or handle only specific control tasks effectively, limiting their practical versatility. OminiControl addresses these limitations through three key innovations: (1) a minimal architectural design that leverages the DiT's own VAE encoder and transformer blocks, requiring just 0.1% additional parameters; (2) a unified sequence processing strategy that combines condition tokens with image tokens for flexible token interactions; and (3) a dynamic position encoding mechanism that adapts to both spatially-aligned and non-aligned control tasks. Our extensive experiments show that this streamlined approach not only matches but surpasses the performance of specialized methods across multiple conditioning tasks. To overcome data limitations in subject-driven generation, we also introduce Subjects200K, a large-scale dataset of identity-consistent image pairs synthesized using DiT models themselves. This work demonstrates that effective image control can be achieved without architectural complexity, opening new possibilities for efficient and versatile image generation systems.
1. Introduction
OminiControl is introduced to address the parameter, task-coverage, and DiT-compatibility limitations of existing image-control methods. It reuses DiT components and combines unified token processing, adaptive positioning, and adjustable conditioning to support aligned and non-aligned tasks.
- Existing image-control methods require substantial architectural modifications and dedicated control modules.
- Task bias limits most methods to either spatially aligned controls or spatially non-aligned controls, rarely both.
- 0.1% additional parameters enables OminiControl to reuse DiT’s VAE encoder and transformer blocks for image conditioning.
- Unified sequence processing concatenates condition tokens with noisy image tokens, allowing attention to learn spatial or semantic relationships across tasks.
- Adaptive position indices and inference-time attention bias support both task flexibility and dynamic adjustment of image-condition influence.
- Subjects200K provides over 200,000 diverse, identity-consistent images synthesized through an automated pipeline for subject-driven generation.
2. Related works
Related work establishes text-to-image generation and expands diffusion models with image and spatial controls. Existing approaches often add condition features to hidden states, limiting non-aligned control, while IP-Adapter introduces an additional encoder for cross-attention.
- ControlNet provides spatially aligned control, while T2I-Adapter improves efficiency through lightweight adapters.
- UniControl uses Mixture-of-Experts to unify different spatial conditions and reduce model size.
- Feature addition to denoising-network hidden states inherently limits effectiveness for spatially non-aligned tasks such as subject-driven generation.
3. Methods
OminiControl integrates image conditions into DiT models by reusing existing components, jointly processing condition and image tokens, and adapting position indices to the control task. The framework is designed to support both spatially aligned and non-aligned control while keeping additional architecture and parameters minimal.
- Control with flexibility: OminiControl adds only 0.1% additional parameters while targeting universal control across diverse tasks.The framework also introduces an attention bias mechanism to adjust image-condition influence at inference time.
- Minimal architecture: OminiControl reuses the DiT VAE encoder and transformer blocks, applying lightweight LoRA fine-tuning instead of adding separate control modules.Condition images are projected into the same latent space as noisy image tokens before joint processing.
- Unified sequence processing: Direct feature addition is limited for non-aligned tasks and constrains interactions between condition and image tokens.The approach assumes spatial correspondence that may not exist in subject-driven generation.
- Unified sequence processing: Unified sequence processing concatenates noisy image, text, and condition tokens as [X; CT; CI] for flexible multi-modal attention.This lets relationships emerge between token pairs without imposing rigid spatial constraints.
- Unified sequence processing: Attention maps show that unified token processing handles both spatially aligned and non-aligned tasks.The Canny-to-image task exhibits spatial interactions, while subject-driven generation shows subject-focused attention.
- Position-aware token interaction: Shared position indices support aligned tasks, whereas shifted indices improve non-aligned-task performance by avoiding spatial overlap.A fixed offset such as (0, 32) accelerates convergence and improves final performance; the dynamic strategy selects indices by task type.
4. Experiment
Experiments evaluate OminiControl across spatially aligned and subject-driven generation, showing strong controllability, image quality, parameter efficiency, and flexible condition-strength control. Ablations examine LoRA rank, integration depth, and critical attention components.
- Setup: Experiments cover spatially aligned tasks and subject-driven generation using FLUX.1, with comparisons against ControlNet, T2I-Adapter, ControlNetPro, IP-Adapter, and FLUX.1 Tools.Spatially aligned tasks include Canny-to-image, depth-to-image, inpainting, and colorization.
- Spatially aligned tasks: 0.38 F1-Score is achieved on depth-to-image, while deblurring FID improves from 30.38 to 11.49 versus ControlNetPro.MSE is reduced by 77% for deblurring and 93% for colorization compared with ControlNetPro.
- Subject-driven generation: 75.8% modification accuracy exceeds IP-Adapter (FLUX)’s 57.7%, while 50.6% identity preservation exceeds IP-Adapter (SD 1.5)’s 29.4%.Best-seed results reach 90.7% modification accuracy and 82.3% identity preservation.
- Parameter efficiency: 14.5M trainable parameters, approximately 0.1% of FLUX.1’s 12B parameters, are required, compared with 27.5% for ControlNet and 7.6% for IP-Adapter.Using FLUX.1’s original VAE encoder still requires only 0.4% additional parameters.
- Condition strength control: Condition strength factor γ controls image-condition influence for both spatially aligned and subject-driven generation.The mechanism is evaluated qualitatively through generated results with varying γ.
- Ablation studies: LoRA rank 16 performs best across multiple aspects, while rank 1 remains competitive for text-image alignment.Ablations also report that full-depth condition integration achieves the best performance.
- Ablation studies: Normalization layers and query and key attention projections are critical for control quality, whereas value projections have less impact.Removing LoRA from normalization, query, or key components significantly degrades conditional rendering.
5. Conclusion
OminiControl provides parameter-efficient image-conditional control across spatially aligned and non-aligned DiT tasks, supported by the Subjects200K dataset. Its unified sequence approach improves flexibility but increases inference token processing.
- Conclusion: OminiControl uses unified tokens for image-conditional DiT control across diverse tasks with only 0.1% additional parameters.Subjects200K contains over 200,000 subject-consistent images supporting subject-driven generation.
- Limitation: The unified sequence approach increases the number of tokens processed through the network, potentially limiting inference computational efficiency.Improving token efficiency while preserving control capabilities is identified as future research.
A. Details of Subjects200K datasets
Subjects200K is a synthetic dataset designed to provide high-quality, identity-consistent image pairs at scale. Its pipeline generates structured descriptions, synthesizes paired views, and applies stringent quality assessment.
- Overview: Subjects200K addresses limitations in the scale and image quality of previous datasets using FLUX.1-dev to generate consistent subject images under varied conditions.
- Dataset structure: The dataset contains two splits: object pairs across different scenes, and scene images paired with corresponding studio photographs.Both splits are generated using similar pipelines and are publicly available.
- Pipeline: The generation pipeline has three stages: description generation, image synthesis, and quality assessment.
- Description generation: The description stage creates 42 object categories and 4,696 unique objects, each with a brief description, eight scene descriptions, and one studio-photo description.
- Image synthesis: Image synthesis combines object and scene descriptions into prompts that preserve subject consistency while varying environments.Five random seeds generate diverse images at 1056×528 pixels before paired images are cropped to 512×512.
- Quality assessment: Each image undergoes five independent ChatGPT-4o vision evaluations covering composition, subject consistency, and image quality; only images passing all evaluations are retained.
A.2. Dataset Statistics
The dataset statistics describe large-scale paired-image production and rigorous filtering, while supplementary figures illustrate structured descriptions, prompt construction, and successful versus failed generations.
- Dataset statistics: 211,320 subject-consistent image pairs are generated from 4,696 object instances, and 111,767 pairs pass quality control for the final dataset.The filtered pairs correspond to 223,534 high-quality training images.
- Generation examples: The dataset generation examples show structured object descriptions and prompt templates combining object information with distinct scene descriptions.
- Training-data comparison: Models trained on Subjects200K generate novel views while preserving identity, whereas data augmentation tends to copy inputs directly.
B.1. Effect of training data
This section compares traditional augmentation with Subjects200K for subject-driven generation. Subjects200K provides synthesized, identity-consistent image pairs for training.
- Subject-driven generation uses a reference subject image and text description to produce novel images while preserving key characteristics.
- The comparison evaluates traditional augmentation against training with Subjects200K.Augmentation includes cropping, rotation, scaling, and adjustments to contrast, saturation, and color.
B.2. Evaluation for subject-driven generation
The evaluation framework assesses subject-driven generation through criteria covering subject preservation and requested modifications. User studies further compare identity consistency, text-image alignment, and visual coherence.
- Evaluation criteria: Five criteria assess identity preservation, material quality, color fidelity, natural appearance, and modification accuracy.
- User studies: 375 valid user-study responses evaluate identity consistency, text-image alignment, and visual coherence.Visual coherence concerns the relationship between subjects and backgrounds.
- User studies: The user-study results report superior performance for OminiControl across all three evaluation dimensions.
B.3. Additional generation results
This section presents additional generation results on the DreamBooth dataset and other subject-driven generation tasks. The supplied table caption describes quantitative evaluation criteria and indicates that higher percentages are better.
- Figure S7 presents additional generation results on the DreamBooth dataset.
- Figure S8 demonstrates OminiControl on other subject-driven generation tasks.
- Table S1 reports quantitative evaluation results in percentages, with higher values indicating better performance.
- Figure S6 provides more results on the DreamBooth dataset.