Source-linked AI summary
DreamOmni2: Multimodal Instruction-based Editing and Generation
Bin Xia, Bohao Peng, Yuechen Zhang, Junjia Huang, Jiyang Liu, Jingyao Li, Haoru Tan, Sitong Wu, Chengyao Wang, Yitong Wang, Xinglong Wu, Bei Yu, Jiaya Jia
TL;DR
Existing editing and subject-driven generation methods struggle with detailed reference-conditioned edits, abstract concepts, and multi-image inputs. DreamOmni2 introduces multimodal tasks, synthetic data construction, and a multi-image framework; experiments show impressive performance across editing and generation.
Problem
Language-only editing misses detailed visual intent, while subject-driven generation focuses mainly on concrete objects and lacks multimodal support for abstract concepts.
Method
DreamOmni2 combines a three-stage data synthesis pipeline with index and shifted position encoding and joint VLM-generation training.
Results
DreamOmni2 shows impressive performance across multimodal editing and generation, outperforming Nano Banana on generation evaluations and approaching GPT-4o.
Takeaways & Limitations
The proposed tasks, dataset, framework, and benchmark extend unified image creation toward concrete and abstract multimodal instructions.
Abstract
from arXiv · showhide
Recent advancements in instruction-based image editing and subject-driven generation have garnered significant attention, yet both tasks still face limitations in meeting practical user needs. Instruction-based editing relies solely on language instructions, which often fail to capture specific editing details, making reference images necessary. Meanwhile, subject-driven generation is limited to combining concrete objects or people, overlooking broader, abstract concepts. To address these challenges, we propose two novel tasks: multimodal instruction-based editing and generation. These tasks support both text and image instructions and extend the scope to include both concrete and abstract concepts, greatly enhancing their practical applications. We introduce DreamOmni2, tackling two primary challenges: data creation and model framework design. Our data synthesis pipeline consists of three steps: (1) using a feature mixing method to create extraction data for both abstract and concrete concepts, (2) generating multimodal instruction-based editing training data using the editing and extraction models, and (3) further applying the extraction model to create training data for multimodal instruction-based editing. For the framework, to handle multi-image input, we propose an index encoding and position encoding shift scheme, which helps the model distinguish images and avoid pixel confusion. Additionally, we introduce joint training with the VLM and our generation/editing model to better process complex instructions. In addition, we have proposed comprehensive benchmarks for these two new tasks to drive their development. Experiments show that DreamOmni2 has achieved impressive results. Models and codes will be released.
1 INTRODUCTION
DreamOmni2 introduces multimodal instruction-based editing and generation to address limitations of language-only editing and concrete-object-only generation. It combines a three-stage data pipeline with a multi-image framework and joint VLM training.
- DreamOmni2 proposes multimodal editing and generation guided by text and reference images containing concrete objects or abstract attributes.
- A three-stage pipeline creates training data by mixing features, generating multimodal editing pairs, and extracting references for generation.
- Index encoding, shifted position encoding, and joint VLM-generation training address multiple image inputs and complex user instructions.
- DreamOmni2 introduces a real-image benchmark for both tasks, and experiments demonstrate effectiveness in real-world scenarios.
2 RELATED WORK
Prior work studies language-based editing and subject-driven generation, but DreamOmni2 extends both toward multimodal guidance and abstract attributes. Its training-data construction uses feature mixing to create paired examples.
- Language-based editing often cannot express detailed real-world changes, motivating reference images for concrete objects and abstract attributes.
- Subject-driven generation methods commonly require multiple subject images and per-subject fine-tuning, limiting user-friendliness.
- DreamOmni2’s feature-mixing scheme creates paired training data for concrete objects and abstract attributes using the base model’s text-to-image capabilities.
3 METHODOLOGY
DreamOmni2 addresses missing multimodal training data and multi-image processing through a synthetic data pipeline, a multi-reference framework, joint VLM training, and a benchmark.
- 3.1 SYNTHETIC DATA: Existing pipelines cannot readily create reference-conditioned editing data or references for abstract attributes and occluded concrete objects.
- 3.1 SYNTHETIC DATA: The three-stage pipeline mixes features, generates editing data from target images, and extracts references to form generation tuples.
- 3.1 SYNTHETIC DATA: Feature mixing generates paired source and target images without halving resolution or introducing the dividing-line content blending associated with diptych generation.
- 3.1 SYNTHETIC DATA: The dataset combines real and synthetic targets, concrete objects and abstract attributes, and one to five reference images.
- 3.2 FRAMEWORK AND TRAINING: Index and shifted position encodings distinguish multiple reference images, while joint VLM training converts irregular instructions into the model’s structured format.
- 3.3 BENCHMARK: The DreamOmni2 benchmark uses real images to evaluate abstract and concrete reference generation and editing, addressing the absence of an existing benchmark.
4 EXPERIMENTS
DreamOmni2 is evaluated on multimodal instruction-based editing and generation, with ablations testing joint training and multi-image encoding. It outperforms open-source competitors across both tasks and approaches or exceeds selected commercial models.
- Multimodal Instruction-based Image Editing: DreamOmni2 produces more accurate and consistent multimodal edits than competitive methods and closed-source GPT-4o and Nano Banana in qualitative comparisons.The editing cases cover concrete objects and abstract attributes.
- Multimodal Instruction-based Image Editing: VLM evaluations show DreamOmni2 significantly outperforming open-source models and approaching commercial-model performance, while GPT-4o and Nano Banana introduce unintended inconsistencies.GPT-4o also causes edited images to appear yellowed, and these issues are difficult for VLMs to detect accurately.
- Multimodal Instruction-based Image Generation: DreamOmni2 significantly outperforms open-source models in multimodal instruction-based generation and achieves results comparable to GPT-4o while outperforming Nano Banana.The comparison covers human evaluations and assessments by Doubao 1.6 and Gemini 2.5.
- Multimodal Instruction-based Image Generation: DreamOmni2 significantly outperforms open-source models in generation accuracy and object consistency, including on concrete-object cases where those models are specialized.It also performs better on abstract attributes, which open-source models struggle to generate.
- Joint Training: Joint training of the VLM with the generation/editing model outperforms separate training schemes, improving understanding of complex real-world instructions.The ablation compares the base model, basic instruction training, VLM-assisted training, and joint training.
- Index and Position Encoding: Index encoding identifies referenced images, while position encoding shifts prevent copy-and-paste effects and improve multi-image generation and editing.DreamOmni2 incorporates both mechanisms for multiple reference images.
5 CONCLUSION
DreamOmni2 addresses the limitations of language-only editing and concrete-object-only generation through multimodal tasks covering concrete and abstract concepts. Its data pipeline and framework support multi-image inputs and complex instructions, with experiments showing strong performance.
- 5 CONCLUSION: DreamOmni2 introduces multimodal instruction-based editing and generation tasks guided by concrete objects and abstract attributes.The tasks combine text and image instructions to address limitations of language-only editing and concrete-object-focused generation.
- 5 CONCLUSION: A three-stage synthesis pipeline creates extraction data, multimodal editing data, and multimodal generation data for training.Feature mixing supports extraction of shared objects or attributes before editing and generation data are synthesized.
- 5 CONCLUSION: Index encoding and position encoding shifts distinguish multiple images and avoid copy-and-paste effects, while joint VLM training improves complex-instruction understanding.These framework components address multi-image input and irregular real-world instructions.
- 5 CONCLUSION: Extensive experiments report impressive performance for DreamOmni2 on the proposed tasks.
A.1 DREAMOMNI2 BENCHMARK
The DreamOmni2 benchmark evaluates multimodal instruction-based editing and generation across varied reference-image counts, attributes, and concrete objects. It contains 205 editing cases and 114 generation cases.
- A.1 DREAMOMNI2 BENCHMARK: The benchmark contains 205 multimodal instruction-based editing test cases and 114 instruction-based generation test cases.
- A.1 DREAMOMNI2 BENCHMARK: Test cases use one to five input reference images and cover diverse local and global attributes as well as concrete objects.Examples are visualized separately for generation and editing.
A.2 MORE MULTIMODAL INSTRUCTION-BASED EDITING CASES
Additional visual cases demonstrate DreamOmni2 on multimodal instruction-based editing.
- A.2 MORE MULTIMODAL INSTRUCTION-BASED EDITING CASES: Figures 8–21 provide additional visual cases for DreamOmni2 on multimodal instruction-based editing.
A.3 MORE MULTIMODAL INSTRUCTION-BASED GENERATION CASES
Figures 22–30 present additional visual cases of DreamOmni2 for multimodal instruction-based generation.
- Figures 22–30 provide additional multimodal instruction-based generation cases for DreamOmni2.