PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniGen2: Towards Instruction-Aligned Multimodal Generation

Chenyuan Wu, Pengfei Zheng, Ruiran Yan, Shitao Xiao, Xin Luo, Yueze Wang, Wanli Li, Xiyan Jiang, Yexin Liu, Junjie Zhou, Ze Liu, Ziyi Xia, Chaofan Li, Haoge Deng, Jiahao Wang, Kun Luo, Bo Zhang, Defu Lian, Xinlong Wang, Zhongyuan Wang, Tiejun Huang, Zheng Liu

arXiv:2506.18871v4cs.CVcs.AIcs.CL

TL;DR

Existing benchmarks and open-source models provide limited coverage for versatile instruction-aligned multimodal generation and consistent in-context evaluation. OmniGen2 addresses this with a VLM-conditioned, instruction-aligned generative model, dedicated data pipelines, and the OmniContext benchmark; it reports competitive multi-task performance and a GenEval score of 0.95.

  • Problem

    Existing open-source models can be specialized or lose plasticity, while existing in-context benchmarks lack multiple-image support and diverse tasks.

  • Method

    OmniGen2 combines a VLM-conditioned diffusion architecture, progressive multi-task reinforcement learning, dedicated data pipelines, and the OmniContext benchmark.

  • Results

    OmniGen2 demonstrates semantic consistency, versatile capabilities, and generation quality across standard benchmarks and OmniContext, achieving a GenEval overall score of 0.95.

  • Takeaways & Limitations

    The results support instruction alignment as a crucial step toward general multimodal systems and provide a standardized tool for measuring in-context generation progress.

  • Takeaways & Limitations

    OmniGen2 is sensitive to ambiguous multi-image instructions and improves when prompts explicitly identify each object’s source image.

Abstract

from arXiv · show

In this work, we introduce OmniGen2, a versatile and open-source generative model designed to provide a unified solution for diverse generation tasks, including text-to-image, image editing, and in-context generation. Unlike OmniGen v1, OmniGen2 features two distinct decoding pathways for text and image modalities, utilizing unshared parameters and a decoupled image tokenizer. This design enables OmniGen2 to build upon existing multimodal understanding models without the need to re-adapt VAE inputs, thereby preserving the original text generation capabilities. To facilitate the training of OmniGen2, we developed comprehensive data construction pipelines, encompassing image editing and in-context generation data. Additionally, we introduce a reflection mechanism tailored for image generation tasks and curate a dedicated reflection dataset based on OmniGen2. Despite its relatively modest parameter size, OmniGen2 achieves competitive results on multiple task benchmarks, including text-to-image and image editing. To further evaluate in-context generation, also referred to as subject-driven tasks, we introduce a new benchmark named OmniContext. OmniGen2 achieves state-of-the-art performance among open-source models in terms of consistency. We will release our models, training code, datasets, and data construction pipeline to support future research in this field. Project Page: https://vectorspacelab.github.io/OmniGen2; GitHub Link: https://github.com/VectorSpaceLab/OmniGen2

1. Introduction

OmniGen2 targets a simple, versatile foundation for instruction-aligned multimodal generation across text-to-image, image editing, and in-context generation. It combines a VLM-conditioned diffusion architecture, progressive multi-task reinforcement learning, and new benchmark and data pipelines.

  • Motivation: Existing open-source generation models often lack the versatility or plasticity needed for broad multimodal instruction alignment.Some models are task-specialized, while others are over-optimized for particular aesthetic preferences.
  • Architecture: OmniGen2 conditions a diffusion transformer on variable-length VLM hidden states to leverage multimodal semantics and world knowledge.Omni-RoPE is introduced to improve spatial consistency and cross-image localization, while avoiding fixed-length query bottlenecks.
  • Instruction Alignment: Progressive reinforcement learning uses GRPO across sequential stages, with task-specific rewards and an organized training sequence to promote inter-task transfer.The stages target instruction alignment for specific tasks.
  • Evaluation: The authors introduce OmniContext, an eight-category benchmark for evaluating consistency across individuals, objects, and scenes in in-context generation.The benchmark addresses the lack of established public evaluation for this task.
  • Contributions: The work contributes an instruction-aligned OmniGen2 model, an end-to-end alignment pipeline, and a standardized benchmark for in-context image generation.The stated goals include improving instruction following, context consistency, and generation quality across diverse scenarios.

2. Dataset Construction

OmniGen2’s dataset construction targets the need for high-quality, large-scale, diverse data for general multimodal generation. It combines curated multimodal and text-to-image data with dedicated pipelines for complex editing and in-context tasks.

  • Foundational Knowledge and General Capabilities: OmniGen2 curates approximately 140M open-source image-text pairs and adds 10M proprietary images annotated by Qwen2.5-VL-72B.The foundation dataset covers multimodal understanding and text-to-image generation.
  • Advanced Capabilities for Editing and In-Context Tasks: Dedicated construction pipelines address data scarcity for detailed image editing and consistent in-context generation.The pipelines supplement existing datasets and leverage scalable data creation to fill gaps in complex task domains.

3. Method

OmniGen2 combines decoupled text and image generation pathways with Omni-RoPE and progressive multi-task alignment. Its training and positional-design choices target versatile instruction following, spatial correspondence, and contextual consistency.

  • Decoupled Architecture: OmniGen2 uses separate autoregressive and diffusion transformers to support multimodal understanding and high-fidelity image synthesis.The autoregressive transformer is initialized from Qwen2.5-VL-3B, while the diffusion transformer is dedicated to image synthesis.
  • Decoupled Architecture: The VLM processes multimodal context first, then passes hidden states to the diffusion decoder, while VAE features preserve fine-grained visual details.A learned <|img|> token triggers image generation; the diffusion decoder uses VLM semantics alongside low-level image features.
  • Omni-RoPE: Omni-RoPE separates image identity from local 2D layout, allowing tokens to distinguish images while preserving spatial alignment across editing operations.Each image receives an instance identity, while coordinates are computed locally within that image; text reduces naturally to a 1D positional index.
  • Omni-RoPE: Omni-RoPE converges faster and reaches lower reconstruction loss than the compared RoPE designs in the toy reconstruction task.The experiment reconstructs the k-th image from randomly sampled inputs using loss < 0.014 as its high-fidelity target; image-index embedding further improves final fidelity.
  • Training and Alignment: The foundation model uses pre-training followed by supervised fine-tuning, then progressive reinforcement learning for multi-task instruction alignment.The curriculum covers editing, text-to-image, and in-context generation tasks, with task-specific rewards and Flow-GRPO training.
  • Training and Alignment: The alignment curriculum uses learned rewards for editing and in-context generation, verifiable GenEval rewards for text-to-image, and excludes high reward-hacking-risk or nonsynergistic tasks.Aesthetic HPSv3 rewards and specialized OCR tasks are omitted; the three stages use EditScore, GenEval, and Qwen2.5-VL-72B rewards respectively.

4. OmniContext Benchmark

OmniContext is a benchmark for reference-based in-context image generation designed to address limited task diversity, multi-image support, and weak evaluation metrics. It uses diverse curated images, structured task categories, and GPT-4.1-based scoring.

  • Motivation: OmniContext addresses benchmark gaps by supporting multiple input images and diverse reference-based generation tasks beyond limited object and prompt collections.The benchmark responds to limitations of DreamBench and simplistic CLIP-I evaluation for multi-subject assessment.
  • Dataset Construction: The benchmark contains manually collected high-quality images spanning Character, Object, and Scene categories.Sources include personal photos, open-source images, animation stills, and AI-generated images.
  • Task Design: OmniContext defines SINGLE, MULTIPLE, and SCENE task categories, with 50 examples per subtask.SINGLE uses one context image, MULTIPLE combines multiple subjects, and SCENE conditions on environmental context.
  • Dataset Construction: Image–prompt pairs combine MLLM filtering, expert image selection, and GPT-4o prompt generation refined for semantic and syntactic variety.Experts select images based on clarity, aesthetics, and diversity after automated low-quality filtering.
  • Evaluation: GPT-4.1 scores outputs on Prompt Following, Subject Consistency, and an Overall Score defined as their geometric mean.The evaluator reports 0–10 scores with rationales for each assessment.

5. Experiments

OmniGen2 is evaluated across multimodal understanding, text-to-image, image editing, and in-context generation, alongside ablations of its multi-task reinforcement-learning strategy. It achieves strong benchmark results, while ablations show that task selection, reward design, and training order materially affect performance.

  • Multimodal Understanding: OmniGen2 achieves multimodal understanding scores of 79.1 on MMBench, 53.1 on MMMU, and 61.8 on MM-Vet.These results support its ability to interpret complex visual and textual instructions.
  • Text-to-Image Generation: 0.95 overall GenEval and 0.47 overall OneIG-Bench demonstrate strong and competitive text-to-image performance.The GenEval score exceeds UniWorld-V1 at 0.84 and BAGEL at 0.88, while OneIG-Bench trails only large-scale models identified by the authors.
  • Image Editing: OmniGen2 balances edit accuracy and preservation, attaining Emu-Edit CLIP-Out 0.311, CLIP-I 0.896, DINO 0.876, and GEdit-Bench overall 7.21.It also achieves GEdit-Bench SC 7.58 and PQ 7.94, and surpasses some strong open-source models on ImgEdit-Bench.
  • In-context Generation: 7.95 overall on OmniContext establishes a strong in-context-generation baseline and surpasses Qwen-Image-Edit-2509.The benchmark contains eight subtasks and evaluates subject consistency across redesigned contexts.
  • Ablation Study: Ablations show that limited skill overlap can cause negative interference, whereas aligned tasks can improve both GenEval and GEdit performance.OCR-only training lowers GEdit Overall from 6.28 to 6.13, while Edit & GenEval reaches GenEval 0.95 versus 0.94 and GEdit Overall 7.19 versus 7.01.
  • Ablation Study: Reward choice and curriculum order are consequential: HPSv3 inflates PQ to 8.22 while collapsing SC and IC, and alternate ordering lowers GEdit Overall from 7.21 to 7.06.The authors also report that editing-first ordering performs better than T2I-first ordering.

6. Related Works

Related work spans diffusion-based image synthesis and controllable or instruction-guided editing, unified multi-task generation, and reinforcement-learning methods for alignment. The supplied passages position OmniGen2 within these developments and identify OmniContext as its proposed in-context-generation benchmark.

  • Multimodal Generation: Diffusion models support high-fidelity image synthesis, while ControlNet and T2I-Adapter improve controllability and several methods support instruction-guided editing.The passage names Stable Diffusion, DALL·E, Imagen, ControlNet, T2I-Adapter, StyleShot, InstructPix2Pix, and EMU-Edit.
  • In-context Generation Evaluation: OmniContext is introduced as a benchmark for comprehensive evaluation of in-context image generation, with eight subtasks and an overall comparison of existing models.The table caption identifies Character + Object as one abbreviated category.
  • Multimodal Generation: Unified image-generation models extend prior work by integrating multiple tasks into a single model.The passage specifically cites OmniGen and UniReal among related unified approaches.
  • Reinforcement Learning for Alignment: Reinforcement-learning approaches formulate diffusion sampling as sequential decision-making and optimize alignment with policy updates or reward optimization.The supplied text mentions DDPO, DPOK, ReFL, AlignProp, and GRPO-based methods, but the passage is truncated before completing the final example.

7. Conclusion

OmniGen2 combines a flexible multimodal architecture with systematic instruction alignment and demonstrates semantic consistency, versatile capabilities, and generation quality across standard and OmniContext benchmarks. The authors conclude that instruction alignment may be a crucial step toward general multimodal systems.

  • Conclusion: OmniGen2 combines a robust, flexible base model with a multi-task reinforcement-learning alignment scheme for diverse multimodal generation tasks.The conclusion emphasizes a simple, efficient, and flexible architecture.
  • Conclusion: Experiments on standard benchmarks and OmniContext demonstrate semantic consistency, versatile capabilities, and superior generation quality.The conclusion reports consistent benefits from instruction alignment across various tasks.
  • Conclusion: The authors suggest that instruction alignment may be a crucial step toward realizing general multimodal systems.This is presented as a supported implication of the reported results, not as a universal guarantee.

8. More Qualitative Results

OmniGen2 demonstrates versatile, high-fidelity generation and editing across text-to-image, image editing, and in-context tasks. The qualitative results also expose limitations involving language, input quality, body-shape edits, ambiguous multi-image instructions, and imperfect object reproduction.

  • OmniGen2 generalizes across text-to-image, image editing, in-context generation, and complex scenarios with consistency and fidelity.
  • Text-to-image generation: The model produces high-fidelity text-to-image outputs across varied concepts, lighting, composition, and aspect ratios.
  • Image editing: OmniGen2 handles localized object edits, semantic changes, global transformations, and background modifications while preserving subjects and unmodified regions.
  • In-context generation: In-context generation combines multiple subjects or places objects and people into new scenes while preserving their identities.
  • Limitations: The model performs worse with Chinese prompts and low-quality images, struggles with human body-shape modifications, and is sensitive to ambiguous multi-image instructions.
  • Limitations: In-context generation can fail to reproduce objects perfectly; increasing image guidance scale partially alleviates but does not fully solve the issue.

9. Other Experimental Details

The paper supplements OmniGen2 with architectural analysis and scalable data-construction pipelines for in-context generation, in-context editing, and diverse image editing. These pipelines use video and image sources to create semantically consistent examples and precise instructions.

  • Omni-RoPE analysis: Omni-RoPE variants reduce toy reconstruction loss faster than prior positional encodings, while image index embeddings improve late-stage fidelity and stability.
  • In-context generation: In-context generation requires reproducing visual concepts from reference images without additional fine-tuning, but high-quality task-specific datasets remain scarce.
  • In-context data: Video frames provide consistent subjects across varying poses, viewpoints, and illumination for constructing in-context training pairs.
  • In-context editing: In-context editing extracts elements from a context image and uses them to edit a target image.
  • Data-construction motivation: The pipeline addresses limitations of inpainting-based datasets, including low image quality, inaccurate instructions, and limited coverage of motion or expression changes.
  • Video editing data: Video-derived editing pairs are filtered by scene continuity, viewpoint consistency, and local visual differences before an MLLM generates editing instructions.

9.5. Interleave Data

OmniGen2’s interleaved-data and reflection procedures extend training beyond static image pairs. They create continuous multimodal sequences and teach the model to inspect, correct, and stop generation, while retaining known reflection limitations.

  • Interleave data: The interleaved-data pipeline builds sequences from intra-scene and inter-scene video frames and captions changes between consecutive frames.
  • Interleave data: 0.8 million interleaved video samples are used to pretrain processing of continuous multimodal sequences.
  • Reflection data: Reflection data interleave instructions, generated images, and step-by-step analyses of unmet requirements with proposed corrective actions.
  • Reflection data: An MLLM evaluates generated images, identifies instruction or quality failures, and suggests modifications for reflection training.
  • Reflection results and limitations: Online reinforcement learning for stronger reflection capabilities is left for future exploration.
  • Reflection results and limitations: Reflection fine-tuning enables successful correction and appropriate termination, but over-reflection and failed revision remain challenges linked to limited data and 3B-scale perception.

9.7. Training pipeline details

The training and evaluation details combine staged curriculum learning with OmniContext and out-of-distribution testing. OmniGen2 achieves strong text-to-image results efficiently, while the full RL curriculum transfers across tasks and reward settings.

  • Training pipeline: The staged pipeline progresses from general pre-training to instruction alignment with increasing task complexity and resolution.
  • OmniContext evaluation: OmniContext reports Prompt Following, Subject Consistency, and Overall scores separately for SINGLE, MULTIPLE, and SCENE tasks.
  • OOD evaluation: Out-of-distribution evaluation uses Emu-Edit and OneIG-Bench to measure transferability beyond the training rewards.
  • RL curriculum: The full Edit → GenEval →IC curriculum consistently outperforms the base model and alternative orders across all OOD metrics.
  • GenEval results: 0.95 overall on GenEval surpasses UniWorld-V1 (0.84) and BAGEL (0.88), using 4B trainable parameters, 15M T2I pairs, and 50k RL prompts.
Loading 2506.18871v4…