PaperlayerView exact revision on arXiv

Source-linked AI summary

DreamOmni3: Scribble-based Editing and Generation

Bin Xia, Bohao Peng, Jiyang Liu, Sitong Wu, Jingyao Li, Junjia Huang, Xu Zhao, Yitong Wang, Ruihang Chu, Bei Yu, Jiaya Jia

arXiv:2512.22525v2cs.CV

TL;DR

Unified generation and editing models rely heavily on text, which can miss intended edit locations and fine visual details. DreamOmni3 adds scribble-based editing and generation, supported by synthetic data and a joint-input framework. The paper reports strong scribble-generation evaluations and establishes a benchmark using real-world images.

  • Problem

    Text prompts can fail to capture users’ intended edit locations and fine-grained visual details, while some objects or positions are difficult to describe in language.

    Source: §1 · p. 3

  • Method

    DreamOmni3 synthesizes scribble-based editing and generation data and feeds both original and scribbled source images using shared position and index encodings.

    Source: §1 · p. 3

  • Results

    On scribble-based generation, DreamOmni3 outperforms Nano Banana in human, Doubao, and Gemini evaluations, with performance similar to GPT-4o.

    Source: §4 · p. 11

  • Takeaways & Limitations

    The defined scribble-based tasks can combine text and image instructions and provide clear targets for optimization and future research.

    Source: §1 · p. 3

Abstract

from arXiv · show

Recently unified generation and editing models have achieved remarkable success with their impressive performance. These models rely mainly on text prompts for instruction-based editing and generation, but language often fails to capture users intended edit locations and fine-grained visual details. To this end, we propose two tasks: scribble-based editing and generation, that enables more flexible creation on graphical user interface (GUI) combining user textual, images, and freehand sketches. We introduce DreamOmni3, tackling two challenges: data creation and framework design. Our data synthesis pipeline includes two parts: scribble-based editing and generation. For scribble-based editing, we define four tasks: scribble and instruction-based editing, scribble and multimodal instruction-based editing, image fusion, and doodle editing. Based on DreamOmni2 dataset, we extract editable regions and overlay hand-drawn boxes, circles, doodles or cropped image to construct training data. For scribble-based generation, we define three tasks: scribble and instruction-based generation, scribble and multimodal instruction-based generation, and doodle generation, following similar data creation pipelines. For the framework, instead of using binary masks, which struggle with complex edits involving multiple scribbles, images, and instructions, we propose a joint input scheme that feeds both the original and scribbled source images into the model, using different colors to distinguish regions and simplify processing. By applying the same index and position encodings to both images, the model can precisely localize scribbled regions while maintaining accurate editing. Finally, we establish comprehensive benchmarks for these tasks to promote further research. Experimental results demonstrate that DreamOmni3 achieves outstanding performance, and models and code will be publicly released.

1 Introduction

DreamOmni3 addresses the difficulty of specifying edit locations and visual details through language by adding scribble-based editing and generation to unified creation models. It pairs synthetic training data with a joint-input framework and establishes a real-world benchmark for these tasks.

1 Introduction

  • Language can be insufficient for specifying edit locations, distinguishing identical or unnamed objects, and communicating creative changes, motivating models that interpret manual scribbles.The paper frames scribbles as an additional way to express editing and generation intentions.Source: §1 · p. 3
  • DreamOmni3 defines four scribble-based editing tasks and three generation tasks, including multimodal-instruction variants, image fusion, and doodle editing and generation.The task set integrates scribble-based creation with an existing unified editing and generation framework.Source: §1 · p. 3
  • The data synthesis pipeline builds editing and generation training examples from DreamOmni2 data using located objects, drawn symbols, cropped reference objects, and sketches.For generation, the source image is a blank canvas; editing examples use the source image, with reference images omitted for instruction-only variants.Source: §1 · p. 3
  • The framework feeds both original and scribbled source images, sharing index and position encodings to align their pixels while preserving scribble information.The paper motivates scribbles over binary masks as simpler for multiple edit areas and compatible with RGB-input models.Source: §1 · p. 3
  • DreamOmni3 introduces joint training of the generation/editing model and VLM to recognize and explain diverse scribbles, and builds a real-world-image benchmark.The benchmark is intended to evaluate generalization and real-world performance.Source: §1 · pp. 3–4

2 Related Work

Prior work covers mask-guided generation and several mask-based editing approaches, while DreamOmni3 frames scribble-based creation within a unified editing and generation system. Its related-work discussion emphasizes rougher region marking, contextual editing, and multimodal image fusion.

2.1 Mask-based Generation

  • Mask-guided generation ranges from multi-region text-to-image methods to subject-driven generation, but existing frameworks can require complex inputs and inference.DreamOmni3 presents a unified design in which users combine painted regions with image and language instructions.Source: §2.1 · p. 4

2.2 Mask-based Editing

  • Mask-based editing includes inpainting, auxiliary edge or low-resolution guidance, and object insertion using reference-image IDs.The related-work section also distinguishes training-free manipulation from learning-based editing.Source: §2.2 · p. 4
  • DreamOmni3’s instruction-based data supports contextual changes such as lighting or shadows, while users can roughly circle a region instead of precisely masking it.The paper contrasts this with traditional inpainting datasets and notes that MagicQuill still requires precise manual masking.Source: §2.2 · p. 4
  • DreamOmni3 unifies editing and generation despite previously inconsistent input formats, and supports multimodal image fusion beyond a single reference image.The paper describes support for multiple images, text, and scribbles, as well as direct sketching on tablets.Source: §2.2 · p. 5

3 Methodology

DreamOmni3 builds scribble-based editing and generation data and pairs it with a framework for interpreting scribbles alongside images and text. It also introduces a benchmark and evaluation standards for these tasks.

3.1 Synthetic Data

  • DreamOmni3 extends DreamOmni2’s multimodal instruction data with scribbles to address the shortage of training data for scribble-based editing and generation.Source: §3.1 · pp. 5, 7
  • Editing data covers four tasks: multimodal and instruction-based editing, doodle editing, and image fusion, with shapes, doodles, or cropped objects marking edit regions.The pipeline uses located regions, augmented scribble symbols, and reference-image crops to construct examples.Source: §3.1 · p. 7
  • Generation data covers multimodal and instruction-based generation plus doodle generation, placing scribbles on a white canvas to indicate where referenced or described content should appear.Source: §3.1 · p. 7

3.2 Framework and Training

  • The framework uses scribbled RGB inputs instead of binary masks, allowing colored scribbles to specify complex relationships while avoiding separate mask inputs for each region.Source: §3.2 · p. 8
  • Joint training adapts a VLM to interpret complex text, image, and scribble instructions and output a predefined annotation format for the generation and editing models.Source: §3.2 · p. 8
  • For source images containing scribbles, joint input supplies both the original and scribbled image with shared index and position encodings; it is omitted when pixel-level preservation is unnecessary.Source: §3.2 · pp. 8–9

3.3 Benchmark and Evaluation

  • The DreamOmni3 benchmark uses real images and manually drawn scribbles from different users across four editing and three generation tasks, including abstract-property and concrete-object edits.Source: §3.3 · p. 9
  • Evaluation standards assess instruction following, consistency of appearance, objects and attributes, visual artifacts, and alignment with doodle regions; VLM assessments align closely with human evaluations.Source: §3.3 · p. 9

4 Experiments

DreamOmni3 performs strongly across scribble-based editing and generation evaluations, while ablations identify benefits from joint inputs, shared encodings, and VLM assistance. Fine-grained evaluations also reveal distinct challenges across task types.

Evaluation on Scribble-based Editing

  • DreamOmni3 achieves the best human-evaluation results for scribble-based editing, outperforms open-source models in VLM evaluations, and approaches commercial models on CLIP-I and DINO consistency scores.Qualitative comparisons also report more precise edits with better consistency.Source: §4 · pp. 9, 11

Evaluation on Scribble-based Generation

  • For generation, DreamOmni3 outperforms Nano Banana in human and automated evaluations, performs similarly to GPT-4o, and surpasses open-source models in visual comparisons.Open-source outputs often retain scribbles or miss instructions; Nano Banana sometimes produces collage-like artifacts.Source: §4 · p. 11, Figure 6 · p. 12

Agreement of VLM-based and human evaluations

  • VLM-based evaluations show strong agreement with human judgments across scribble-based tasks, supporting their use for comparisons, training monitoring, and as a reinforcement-learning reward model.Agreement is measured with Cohen’s κ.Source: §4 · p. 13, Table 3 · p. 13

Joint Input and Joint Training

  • Joint input significantly improves editing more than generation, while the dataset and trained VLM improve scribble capabilities and intent understanding.The authors use joint input for editing when the source contains scribbles, but not for other reference images or scribble-based generation.Source: §4 · p. 13

Index and Position Encoding

  • Using the same index and position encodings for both source inputs yields the best performance, supporting pixel-level alignment and preserving the original training format for reference images.The authors link alignment to more precise scribble localization and editing.Source: §4 · p. 14

Fine-Grained Evaluation on Scribble-Based Editing

  • Editing challenges vary by task: multimodal instruction-based editing has the lowest success rate, while doodle naturalness, instruction responsiveness, reference consistency, and image fusion remain distinct bottlenecks.The reported bottlenecks are naturalness for doodle editing, instruction response for scribble-and-instruction editing, reference appearance consistency for multimodal editing, and collage-image fusion quality.Source: §4 · p. 14

Fine-Grained Evaluation on Scribble-Based Generation

  • Generation challenges likewise vary: multimodal instruction-based generation has the lowest success rate, while doodle naturalness, instruction response, and reference appearance consistency remain difficulties.The multimodal task’s reference-consistency issue concerns the appearance of people and objects.Source: §4 · pp. 14–15

5 Conclusion

The conclusion presents DreamOmni3 as a framework for scribble-based editing and generation, pairing synthesized training data with joint inputs and VLM/model training. The appendix overview lists additional training, evaluation, benchmark, and visual-result materials.

5 Conclusion

  • DreamOmni3 defines scribble-based editing and generation tasks and builds datasets from DreamOmni2 using Referseg-based region extraction and scribble, text, and image instructions.The framework pairs original and scribbled images and applies shared index and position encodings to preserve detail and correspondence.Source: §5 · p. 15
  • The conclusion describes color-distinguished scribbles for multiple masks, joint input to retain covered image details, and joint VLM/model training to interpret user instructions.It also reports a benchmark and evaluation framework combining scribble, text, and image instructions.Source: §5 · p. 15

A The Overview of the Appendix

  • The appendix overview lists further training details, evaluation prompts and details, benchmark information, and visual results for editing and generation.The visual results are assigned to separate editing and generation appendix sections.Source: A The Overview of the Appendix · p. 20

B Training of DreamOmni3

DreamOmni3 training follows a two-phase VLM and Kontext scheme, with separate fine-tuning and LoRA training budgets for the editing and generation models.

B Training of DreamOmni3

  • Training first teaches the VLM to translate user instructions into predefined standard instructions, then trains Kontext to generate target images from images and those instructions.The scheme follows DreamOmni2’s joint VLM and Kontext training approach.Source: B Training of DreamOmni3 · p. 20
  • Qwen2.5-VL 7B is fine-tuned at 1 × 10−5 for approximately 10 A100 hours, while editing and generation LoRA use batch size 16 and learning rate 5 × 10−6 for about 400 A100 hours.Source: B Training of DreamOmni3 · p. 20

C Model Evaluation

The evaluation appendix supplies system prompts for Doubao and Gemini across the scribble-based editing and generation tasks, and specifies the human-evaluation success rule.

C Model Evaluation

  • The appendix provides Doubao and Gemini system prompts for multimodal and instruction-based editing, doodle editing, and image fusion.Source: C Model Evaluation · p. 20
  • It also provides prompts for multimodal and instruction-based generation and doodle generation.Source: C Model Evaluation · p. 20
  • Human reviewers assess each editing case against the stated criteria; a case succeeds when at least three of five evaluators approve it.Source: C Model Evaluation · p. 20

D DreamOmni3 Benchmark

The DreamOmni3 benchmark evaluates scribble-based editing and generation across multiple task types and reference-image configurations. It combines language, images, and scribbles across varied design objects and attributes.

D DreamOmni3 Benchmark

  • The benchmark contains 136 editing and 88 generation test cases, with reference-image inputs ranging from one to four.Source: D DreamOmni3 Benchmark · p. 20
  • It covers four editing types and three generation types, including instruction-based, multimodal instruction-based, and doodle tasks, plus image fusion for editing.Source: D DreamOmni3 Benchmark · p. 20
  • Its tasks integrate language, images, and scribbles across concrete objects and abstract attributes to evaluate different scenarios.The benchmark will be released.Source: D DreamOmni3 Benchmark · p. 20

E More Scribble-based Editing Cases

This section presents additional visual examples of scribble-based editing and documents a VLM assessment prompt for multimodal instruction-based editing.

E More Scribble-based Editing Cases

F More Scribble-based Generation Cases

The section provides additional scribble-based generation examples and benchmark illustrations, alongside VLM assessment prompts for generation tasks.

F More Scribble-based Generation Cases

Loading 2512.22525v2…