PaperlayerView exact revision on arXiv

Source-linked AI summary

UniReal: Universal Image Generation and Editing via Learning Real-world Dynamics

Xi Chen, Zhifei Zhang, He Zhang, Yuqian Zhou, Soo Ye Kim, Qing Liu, Yijun Li, Jianming Zhang, Nanxuan Zhao, Yilin Wang, Hui Ding, Zhe Lin, Hengshuang Zhao

arXiv:2412.07774v2cs.CV

TL;DR

Image-generation and editing methods vary by task despite shared needs to preserve input consistency while producing controlled visual variation. UniReal treats input and output images as discontinuous video frames and uses video-derived supervision in a unified framework. The paper reports state-of-the-art performance across multiple tasks and capabilities in modeling real-world dynamics and generalizing to new tasks.

  • Problem

    Image-generation and editing methods are increasingly specialized by task, limiting generalizable learning across fields and increasing the work of designing methods and collecting domain-specific data.

    Source: §1 · p. 2

  • Method

    UniReal treats varying numbers of input and output images as pseudo-frames in a video-generation framework, using image-index and hierarchical prompts alongside scalable video supervision.

    Source: §1 · p. 2

  • Results

    UniReal achieves state-of-the-art performance in multiple image-generation and editing tasks and shows promising capabilities in modeling real-world dynamics and generalizing to new tasks.

    Source: §5 · p. 8

  • Takeaways & Limitations

    After learning versatile tasks, UniReal exhibits emergent abilities for novel applications without training data.

    Source: §1 · p. 2

    Stability decreases and computation becomes intensive when the number of input and output images exceeds five; typically, 3–4 input images suffice for most applications.

    Source: §5 · p. 8

Abstract

from arXiv · show

We introduce UniReal, a unified framework designed to address various image generation and editing tasks. Existing solutions often vary by tasks, yet share fundamental principles: preserving consistency between inputs and outputs while capturing visual variations. Inspired by recent video generation models that effectively balance consistency and variation across frames, we propose a unifying approach that treats image-level tasks as discontinuous video generation. Specifically, we treat varying numbers of input and output images as frames, enabling seamless support for tasks such as image generation, editing, customization, composition, etc. Although designed for image-level tasks, we leverage videos as a scalable source for universal supervision. UniReal learns world dynamics from large-scale videos, demonstrating advanced capability in handling shadows, reflections, pose variation, and object interaction, while also exhibiting emergent capability for novel applications.

1. Introduction

UniReal addresses the specialization of image-generation and editing methods by treating diverse tasks as discontinuous video-frame generation. It combines pseudo-frame inputs, structured prompts, and video-derived supervision to preserve consistency while modeling visual variation.

1. Introduction

  • UniReal is motivated by task-specific methods and data that limit cross-task generalization and increase the work of designing methods and collecting domain-specific datasets..Source: §1 · p. 2
  • UniReal reframes image generation and editing as discontinuous frame generation, adapting video models’ balance of consistency and variation to image-level tasks.It uses one diffusion transformer with full attention across frames.Source: §1 · p. 2
  • Varying numbers of input and output images act as pseudo-frames, with image-index embeddings and hierarchical prompts coordinating their roles and relationships.Inputs are distinguished as target images, references, or condition maps; prompts combine context-level and image-level guidance with a base prompt.Source: §1 · p. 2
  • Video frame pairs provide scalable supervision for editing, customization, and composition, capturing changes such as adding, removing, or altering objects and structure.The paper uses large-scale frame pairs with captions as editing data and builds an automatic pipeline for customization and composition data.Source: §1 · p. 2
  • UniReal is reported to preserve input consistency and details while modeling lighting, reflections, and object interactions, with emergent abilities on novel applications without training data..Source: §1 · p. 2

2. Related Work

Prior image-generation and editing approaches use varied task-specific solutions, while universal models have explored different ways to combine text and image processing. UniReal’s pipeline encodes images and prompts into a shared transformer input for generation and editing.

Diffusion-based image editing

  • Image-editing work spans prompt-based attribute edits, instructive editing, mask-based editing, customization, and application-specific tasks, but settings and solutions vary despite shared consistency and variation requirements.Universal image-generation and editing methods include instruction-based, task-embedding, and transformer-based approaches, with different modality representations and attention schemes.Source: §2 · p. 2
  • UniReal’s pipeline encodes image latents and noise as visual tokens, adds image-role and index embeddings, joins them with encoded text, and sends the sequence to a transformer before decoding outputs..Source: Figure 2 · p. 3

3. Method

UniReal uses a video-generation framework to unify image tasks and draws supervision from scalable video data. Its design combines image–text associations and hierarchical prompts with video-derived datasets and staged multi-task training.

3. Method

  • UniReal processes input and output images as video frames and uses scalable video data as universal supervision for image generation and editing tasks..Source: §3 · p. 3

3.1. Model Design

  • The diffusion transformer combines image, noise, and text tokens under full attention, while image-index embeddings associate prompt references such as IMG1 and RES1 with corresponding images.Hierarchical prompts combine base instructions with context prompts for task and data attributes, and image prompts distinguish canvas, asset, and control roles.Source: §3.1 · p. 3

3.2. Dataset Construction

  • Video-derived data pairs frames with captions for editing, and uses entity captions, tracked masks, depth, and edge maps to support customization, insertion, inpainting, segmentation, and control tasks.A subset of 200K samples receives more precise instructions; the training mix also includes open-source and in-house task-specific datasets, while video data are described as easier to scale.Source: §3.2 · p. 4
  • Context prompts label video samples as static or dynamic and distinguish synthetic from realistic styles, reference-object data, and perception tasks.The paper uses optical flow and pixel MSE to label static/dynamic scenarios.Source: §3.2 · p. 4

3.3. Training schemes

  • Training uses a 5B-parameter transformer, pretrained at about 256×256 before multi-task training and progressive increases to 512 and 1024 resolution.The paper specifies a 1e-5 learning rate with warm-up at each stage and flow-matching loss; Table 1 describes the mix of existing and newly constructed video-based datasets.Source: §3.3 · p. 4, Table 1 · p. 4
  • Figure 4 presents qualitative instructive-editing comparisons with OmniGen [60], UltraEdit [72], MGIE [13], InstructPix2Pix [1], and CosXL [52], selecting each model’s best result from multiple generations.Table 2 reports comparisons on EMU Edit [48] and MagicBrush [69] test sets, listing task-specific models and concurrent universal models in separate blocks.Source: Figure 4 · p. 5, Table 2 · p. 5

4. Experiments

UniReal is evaluated on image editing, customization, and object insertion, with analyses of its prompt and training-data choices and examples of trained and zero-shot applications. The experiments report strong task-specific results alongside broader capabilities and scope limits.

4.1. Comparisons with Existing Works

  • UniReal achieves the best CLIPdir and CLIPout on EMU Edit and MagicBrush, the best CLIPim on MagicBrush, and the best CLIP-T on DreamBench with competitive DINO and CLIP-I scores.The DreamBench comparison includes oracle, tuning, and zero-shot methods.Source: §4.1 · pp. 5–6, Table 3 · p. 5
  • Qualitative comparisons highlight fine-detail preservation and challenging object changes, while insertion can adapt an object's status without masks and preserve background pixels.Customization examples include logos, drastic pose changes, and interactions; insertion examples include placing a dog in water and adjusting an object's view.Source: §4.1 · p. 6, Figure 5 · p. 6, Figure 7 · p. 7
  • In user studies, UniReal shows advantages over other methods in detail preservation, alignment, and image quality across customization, editing, and object-insertion evaluations.Ten annotators rated the three aspects; the study used different sample sets for each task.Source: §4.1 · pp. 6–7, Figure 6 · p. 6

4.2. Analysis for the Core Components

  • Hierarchical image and context prompts let the same inputs and text prompt yield different targets, reducing ambiguity across mixed tasks and data sources.At inference, prompts can be expanded automatically and revised by users.Source: §4.2 · p. 7, Figure 8 · p. 7
  • Video-only training supports object addition and removal, color editing, and customization, but multiple-input handling is not stable and subtasks such as stylization and precise instruction following remain difficult.The authors retain task-specific data to provide standard learning examples for these gaps.Source: §4.2 · p. 7, Figure 9 · p. 7
  • The full multi-task model outperforms task-specific models; the authors note that task data may lack coverage or generalization, overfit, or contain artifacts from synthetic editing data.They attribute a role to realistic video data in retaining generation quality and say different tasks can complement gaps in one another's data.Source: Table 4 · p. 7, §4.2 · p. 8

4.3. More Applications

  • Shown applications range from generation, controllable generation, reference-guided inpainting, and perception outputs to zero-shot combinations such as multi-object insertion, pose or color edits, feature transfer, and object manipulation.The reported unseen-task examples include adding objects behind masked foreground elements, moving objects, and resizing them.Source: §4.3 · p. 8, Figure 10 · p. 8

5. Conclusion

UniReal unifies image generation and editing by treating images as frames and using video data to learn consistency and variation. It reports strong performance across multiple tasks, while stability and compute become limiting beyond five images.

5. Conclusion

  • UniReal treats images as individual frames in a video-generation framework and uses video data as universal supervision for consistency and variation.Source: §5 · p. 8
  • UniReal achieves state-of-the-art performance across multiple image generation and editing tasks and shows capabilities in modeling real-world dynamics and generalizing to new tasks.Source: §5 · p. 8
  • When the number of input or output images exceeds five, stability decreases and computation becomes intensive; typically, three to four input images suffice.Supporting tasks with many images would require training data containing more images and more efficient architectures.Source: §5 · p. 8

Related papers

Loading 2412.07774v2…