Source-linked AI summary
Less-to-More Generalization: Unlocking More Controllability by In-Context Generation
Shaojin Wu, Mengqi Huang, Wenxu Wu, Yufeng Cheng, Fei Ding, Qian He
TL;DR
Subject-driven generation remains constrained by the difficulty of scaling consistent paired data and controllability from single- to multi-subject settings. The paper introduces a progressive in-context synthesis pipeline and UNO, a multi-image conditioned model with progressive cross-modal alignment and universal rotary position embedding. Experiments show high subject consistency and controllability for both single-subject and multi-subject generation.
Problem
Customized generation lacks scalable, high-quality paired data covering diverse subject variations, while existing methods center mainly on single-subject generation.
Method
The paper combines progressive single- to multi-subject in-context synthesis with UNO, iteratively trained from a text-to-image model using progressive cross-modal alignment and UnoPE.
Results
UNO achieves high-quality subject similarity and text controllability in both single-subject and multi-subject customization.
Takeaways & Limitations
Model-data co-evolution enables enhanced controllability while supporting stable and scalable customized generation.
Takeaways & Limitations
The dataset contains limited editing and stylization data, while the paper primarily focuses on subject-driven generation.
Abstract
from arXiv · showhide
Although subject-driven generation has been extensively explored in image generation due to its wide applications, it still has challenges in data scalability and subject expansibility. For the first challenge, moving from curating single-subject datasets to multiple-subject ones and scaling them is particularly difficult. For the second, most recent methods center on single-subject generation, making it hard to apply when dealing with multi-subject scenarios. In this study, we propose a highly-consistent data synthesis pipeline to tackle this challenge. This pipeline harnesses the intrinsic in-context generation capabilities of diffusion transformers and generates high-consistency multi-subject paired data. Additionally, we introduce UNO, which consists of progressive cross-modal alignment and universal rotary position embedding. It is a multi-image conditioned subject-to-image model iteratively trained from a text-to-image model. Extensive experiments show that our method can achieve high consistency while ensuring controllability in both single-subject and multi-subject driven generation.
1. Introduction
Customized generation is limited by scarce paired data and the difficulty of scaling controllability from single- to multi-subject settings. The paper proposes model-data co-evolution, progressive synthetic data curation, and UNO to address these constraints.
- Motivation: Real paired datasets covering diverse subject poses, locations, and sizes are impractical to acquire, creating a data bottleneck for customized generation.The desired data must encompass multi-perspective subject variations.
- Motivation: The proposed model-data co-evolution paradigm uses less-controllable models to synthesize better customization data for progressively more-controllable variants.The paradigm is designed to support persistent co-evolution between enhanced models and enriched data.
- Technical contributions: The synthetic data framework progresses from single-subject to multi-subject in-context generation and applies multi-stage filtering to produce high-fidelity, high-resolution paired data.The framework is intended to systematically harness knowledge distillation from less-controllable models.
- Technical contributions: UNO is a universal customization architecture that scales multi-condition control through minimal modification of Diffusion Transformers.Its design targets hierarchical controllability adaptation across varying degrees of controllability.
- Experimental contribution: UNO achieves the highest DINO and CLIP-I scores on DreamBench and multi-subject driven generation benchmarks.These results indicate strong subject similarity and text controllability in the evaluated tasks.
2. Related Work
Subject-driven generation methods have progressed from per-subject optimization to reference-image injection and multi-reference support. However, existing approaches still rely heavily on paired images, which are difficult to collect, especially for multi-subject settings.
- Subject-driven generation: Few-data methods such as DreamBooth, textual inversion, and LoRA require per-subject optimization, causing substantial computational and inference-time overhead.These methods introduce subject-driven capability through fine-tuning or textual inversion.
- Subject-driven generation: Image-encoder methods such as IP-Adapter and BLIP Diffusion enable subject-driven generation without fine-tuning for each new concept.They encode reference images and inject the resulting information into the diffusion model.
- Subject-driven generation: DiT-based methods exploit inherent image-reference capability, while later work improves facial identity, joint image-text control, and multiple-reference support.The related approaches include IC LoRA, OmniControl, and subsequent reference-image injection methods.
- Remaining challenge: Despite these advances, existing approaches heavily depend on paired images that are hard to collect, particularly for multi-subject generation.This dependence remains a central dataset challenge for subject-driven generation.
3. Methodology
The methodology combines in-context synthetic data curation with iterative training of UNO, a multi-image conditioned subject-to-image model. It progresses from single-subject to multi-subject generation while using filtering and positional design to improve consistency and controllability.
- Synthetic Data Curation Framework: The framework uses DiT-based in-context generation to create high-resolution, subject-consistent data progressing from single-subject to multi-subject pairs.The pipeline is designed to address the scarcity of comprehensive real paired datasets and supports synthetic data curation at broader subject scope.
- Evaluation: Higher-quality synthesized pairs significantly improve subject similarity on DreamBench, while qualitative comparisons report stronger detail preservation, attribute editing, and multi-subject consistency.The reported comparisons include higher DINO and CLIP-I scores for high-quality data and improved adherence to text editing instructions.
- Synthetic Data Curation Framework: Generated image pairs are filtered first with DINOv2 consistency scores and then with VLM scores covering appearance, details, and attributes.The VLM produces automatically generated evaluation dimensions and a final consistency score for each pair.
- Synthetic Data Curation Framework: The curated dataset trains an S2I model that, together with an open-vocabulary detector, generates multi-subject-consistent data from cropped subjects and corresponding prompts.The trained S2I model receives extracted subject crops and prompts to produce new multi-subject results.
- Customization Model Framework (UNO): UNO is trained progressively: Stage I uses single-subject conditions, followed by Stage II training with multi-image conditions for multi-subject generation.This gradual complexity reduces disruption from introducing reference-image tokens and unlocks more controllability from a pretrained T2I model.
- Customization Model Framework (UNO): UnoPE adjusts position indices for reference-image tokens so multi-image conditioning focuses on text-derived layout while mitigating attribute confusion and spatial over-referencing.The design reuses the original positional format while shifting reference-token positions relative to the noisy image tokens.
4. Experiments
UNO is evaluated on single- and multi-subject generation through qualitative comparisons, automatic metrics, user studies, and ablations. Results indicate strong subject consistency, text controllability, and benefits from progressive alignment and UnoPE.
- Qualitative Analyses: UNO nearly preserves fine subject details while editing attributes in single-image-conditioned generation.The qualitative comparison highlights details such as numbers on a clock dial and reduced copy-paste behavior.
- Qualitative Analyses: UNO preserves all reference images while following text editing instructions in multi-image-conditioned generation.Other methods either lose subject consistency or fail to follow the input text instructions.
- Quantitative Evaluations: 0.760 DINO and 0.835 CLIP-I are UNO's highest reported DreamBench scores in zero-shot single-subject generation.UNO also reports a leading CLIP-I score of 0.304 in the compared settings and achieves the highest DINO and CLIP-I scores in the multi-image scenario.
- Quantitative Evaluations: 30 evaluators ranked UNO across 300 image combinations and five dimensions covering text fidelity, subject similarity, composition, and visual appeal.The study reports that UNO excels in subject similarity and text fidelity while also performing strongly on the other dimensions.
- Ablation Study: Direct multi-reference training without progressive cross-modal alignment significantly reduces DINO, CLIP-I, and subject similarity.Progressive alignment also raises the model's upper limit in single-image-conditioned scenarios.
- Ablation Study: Removing UnoPE significantly lowers DINO and CLIP-I, while generated images follow text descriptions but scarcely reference input images.Among tested position-index offsets, UnoPE achieves the best results on both DreamBench and the multi-subject benchmark.
5. Conclusion
The conclusion presents UNO as a diffusion-transformer customization architecture trained progressively from single-subject to multi-subject data. It reports high similarity and controllability across both single- and multi-subject customization while reducing copy-paste behavior.
- 5. Conclusion: UNO combines progressive cross-modal alignment with universal rotary position embedding to unlock multi-condition contextual generation in diffusion transformers.Its training first uses single-image inputs and then continues with multiple-subject data pairs.
- 5. Conclusion: The progressive synthesis pipeline evolves from single-subject to multi-subject in-context generation and produces high-quality synthetic data.The conclusion links this pipeline to reduced copy-paste behavior.
- 5. Conclusion: UNO achieves high-quality similarity and controllability in both single-subject and multiple-subject customization.The conclusion states that the method also effectively reduces the copy-paste phenomenon.
Supplementary Material
The supplementary material details an in-context data-generation pipeline that constructs diverse subjects and scenes before generating and filtering subject-consistent image pairs.
- Supplementary Material: The pipeline builds a taxonomy tree, generates subject-consistent image pairs with a pretrained T2I model, filters synthesized data with CoT, and trains an S2I model for multi-subject data.These stages are presented as the supplementary description of the in-context data-generation process.
- Supplementary Material: The taxonomy tree uses 365 Object365 classes and an LLM to generate diverse subject instances and scenes.The LLM instructions generate instances according to asset categories.
F.2. Single-Subject In-Context Data Generation
Single-subject in-context data generation uses FLUX.1's in-context capability to create subject-consistent image pairs, then filters failures such as inconsistency and missing subjects.
- F.2. Single-Subject In-Context Data Generation: A pretrained T2I model generates subject-consistent image-pair data using its in-context generation ability.DINOv2 similarity and a vision-language model are used to filter low-quality pairs with poor subject consistency.
- F.2. Single-Subject In-Context Data Generation: A taxonomy tree and diptych text template guide FLUX.1 to generate image pairs at 1024×1024, 1024×768, and 768×1024 resolutions.The three resolutions are produced using FLUX.1's multi-resolution generation capability.
- F.2. Single-Subject In-Context Data Generation: Synthetic image pairs can exhibit subject inconsistency and missing subjects, so the diptych is split into reference and target images for filtering.The split uses the Hough Transform and follows the diptych template.
F.3. Multi-Subject In-Context Data Generation
The pipeline converts single-subject paired data into high-quality multi-subject in-context data by synthesizing consistent reference images and filtering the results. This process can also produce examples with more than two reference images, supporting broader multi-subject generalization.
- Cropping the second subject from a target image can create severe copy-paste artifacts.
- A single-subject S2I model generates a new reference image that preserves the cropped subject while placing it in a different scene.
- The resulting multi-subject pairs retain subject consistency after applying a similar filtering pipeline to the synthesized reference image.
- Some sampled pairs contain more than two reference images because T2I generation and OVD introduce randomness, enabling generalization to more-subject generation.
G. Analysis on LoRA Rank
The LoRA-rank ablation finds sustained performance gains as rank increases, with slower improvement beginning at rank 128; UNO ultimately uses rank 512 after considering performance and resource consumption.
- Increasing the LoRA rank from 4 to 512 produces sustained performance gains, but improvement slows once the rank reaches 128.
- UNO uses a LoRA rank of 512 after jointly considering performance and resource consumption.
H.1. Qualitative Results on Multi-Subject Driven Generation
UNO shows strong qualitative performance in multi-subject generation and generalizes across diverse multi-image conditional scenarios, including settings with little or no specialized training data.
- UNO generates images with strong multi-subject consistency while following editing instructions for subjects and backgrounds.
- UNO demonstrates generalization across identity preservation, virtual try-on, and stylized generation despite minimal exposure to such data during training.
- UNO handles complex multi-subject applications such as logo design and virtual-real integration.
- Virtual Try-on: UNO performs well in virtual try-on without specialized training on virtual try-on datasets, indicating object-relationship understanding beyond copy-paste operation.
I. Limitation and Discussion
The paper’s scope is centered on subject-driven generation, while its synthetic data currently covers limited editing and stylization content. The authors identify expanding data types as a future direction for broader task coverage.
- The paper primarily focuses on subject-driven generation, limiting its coverage beyond that task scope.
- The current dataset contains limited editing and stylization data.
- The types of synthetic data may restrict UNO’s abilities despite its generalization capabilities.
- The authors plan to expand the data types to unlock UNO’s potential and cover a broader range of tasks.