PaperlayerView exact revision on arXiv

Source-linked AI summary

Skywork UniPic 2.0: Building Kontext Model with Online RL for Unified Multimodal Model

Hongyang Wei, Baixin Xu, Hongbo Liu, Size Wu, Jie Liu, Yi Peng, Peiyu Wang, Zexiang Liu, Jingwen He, Yidan Xietian, Chuanxin Tang, Zidong Wang, Yichen Wei, Liang Hu, Boyi Jiang, Wei Li, Ying He, Yang Liu, Xuchen Song, Yangguang Li, Yahui Zhou

arXiv:2509.04548v2cs.CV

TL;DR

Unified image generation and editing models face high computational costs and challenges with instruction adherence and editing consistency. UniPic 2.0 combines a compact generation-and-editing model, staged online reinforcement learning, and a connector-based multimodal framework. The paper reports strong benchmark performance and faster inference for its lightweight model.

  • Problem

    Unified generation and editing models can be computationally costly and face challenges in precise instruction following, editing consistency, and joint reinforcement learning without task conflict.

    Source: §1 · pp. 1–2

  • Method

    The approach modifies SD3.5-Medium for generation and reference-image editing, applies editing-first then generation online reinforcement learning, and connects the Kontext model to a frozen multimodal language model.

    Source: §1 · p. 2, §3.1 · p. 5

  • Results

    With 2B generation parameters, UniPic2-SD3.5M-Kontext surpasses much larger models, while UniPic 2.0 reports state-of-the-art benchmark performance in instruction following, editing consistency, and generation stability.

    Source: §5 · pp. 12–13

  • Takeaways & Limitations

    The paper presents a practical, scalable, and reproducible paradigm for efficient, deployable multimodal intelligence.

    Source: §5 · p. 13

    The model struggles with complex text rendering, precise semantic details, object extraction, complex text modification, and precise attribute conversion in editing.

    Source: §4.4 · p. 12

Abstract

from arXiv · show

Recent advances in multimodal models have demonstrated impressive capabilities in unified image generation and editing. However, many prominent open-source models prioritize scaling model parameters over optimizing training strategies, limiting their efficiency and performance. In this work, we present UniPic2-SD3.5M-Kontext, a 2B-parameter DiT model based on SD3.5-Medium, which achieves state-of-the-art image generation and editing while extending seamlessly into a unified multimodal framework. Our approach begins with architectural modifications to SD3.5-Medium and large-scale pre-training on high-quality data, enabling joint text-to-image generation and editing capabilities. To enhance instruction following and editing consistency, we propose a novel Progressive Dual-Task Reinforcement strategy (PDTR), which effectively strengthens both tasks in a staged manner. We empirically validate that the reinforcement phases for different tasks are mutually beneficial and do not induce negative interference. After pre-training and reinforcement strategies, UniPic2-SD3.5M-Kontext demonstrates stronger image generation and editing capabilities than models with significantly larger generation parameters-including BAGEL (7B) and Flux-Kontext (12B). Furthermore, following the MetaQuery, we connect the UniPic2-SD3.5M-Kontext and Qwen2.5-VL-7B via a connector and perform joint training to launch a unified multimodal model UniPic2-Metaquery. UniPic2-Metaquery integrates understanding, generation, and editing, achieving top-tier performance across diverse tasks with a simple and scalable training paradigm. This consistently validates the effectiveness and generalizability of our proposed training paradigm, which we formalize as Skywork UniPic 2.0.

1 Introduction

The introduction presents a compact unified image generation and editing model, then describes staged reinforcement learning and a connector-based multimodal framework. It reports strong results across generation and editing tasks.

  • UniPic2-SD3.5M-Kontext builds on a compact 2B-parameter SD3.5-Medium module, architectural enhancements, and high-fidelity pretraining for shared generation and editing.Progressive-resolution training and balanced sampling across aspect ratios and resolutions provide native dynamic-resolution support during training and inference.Source: §1 · pp. 1–3
  • PDTR applies online reinforcement learning first to editing and then to text-to-image generation, improving both tasks without negative interference.The approach uses GRPO; editing rewards include Skywork-EditReward and GPT-4.1, while generation uses GenEval and detectors.Source: §1 · p. 2
  • UniPic2-SD3.5M-Kontext outperforms most unified frameworks on GenEval instruction-following accuracy and establishes new records in editing quality.The evaluation is presented as evidence of generation fidelity, instruction comprehension, and editing consistency.Source: §1 · p. 3
  • Skywork UniPic 2.0 combines UniPic2-SD3.5M-Kontext and UniPic2-MetaQuery, and publicly releases its models and source code to support reproducibility.The paper states that lightweight models can outperform substantially larger counterparts in generation fidelity, instruction compliance, and inference efficiency.Source: §1 · p. 3
  • UniPic2-Metaquery is a general and modular paradigm for unified multimodal modeling, enabling end-to-end integration of understanding, generation, and editing through a parameter-efficient connector-based training strategy, achieving SOTA performance and strong generalization across tasks.Source: §1 · p. 3

2 Related Work

The related work reviews image generation and understanding foundations, then contrasts approaches to unifying them. It highlights query-based connections between pretrained multimodal language models and diffusion generators.

2.1 Image Generation

  • Flow Matching replaces explicit diffusion simulation with an ODE-driven transformation from noise to data, reducing diffusion processes and permitting more direct probability transport paths.The section also notes the use of diffusion transformers as scalable image-generation backbones.Source: §2.1 · p. 3

2.2 Image Understanding

  • Multimodal large language models combine visual encoders and language models for visual reasoning, dialogue, and instruction following, but primarily output text rather than visual content.The section says visual encoders are typically pretrained through contrastive vision-language alignment.Source: §2.2 · p. 4

2.3 Unified Models

  • Unified models either build multimodal systems from scratch or connect pretrained language models and diffusion generators using hidden states or learnable queries.Native-from-scratch systems can struggle to accommodate both tasks because of differing granularity; MetaQuery and BLIP3-o use frozen multimodal language models.Source: §2.3 · p. 4

3 Method

UniPic 2.0 combines a pretrained diffusion generator and a frozen multimodal language model, then trains their connector and diffusion components for unified image understanding, generation, and editing. Its staged reinforcement strategy improves generation and editing without degrading the other task.

3 Method

  • UniPic 2.0 uses a frozen MLLM and diffusion transformer bridged by learnable queries and a connector, followed by GRPO post-training.The paper instantiates these components with Qwen2.5VL-7B and SD3.5-Medium.Source: §3 · p. 4

3.1 Pre-training

  • UniPic2-SD3.5M-Kontext extends SD3.5-Medium for joint generation and editing by conditioning on text instructions and reference-image VAE latents.Reference-image context tokens are concatenated with target-image noise tokens; training alternates generation and editing batches across common aspect ratios.Source: §3.1 · p. 5
  • A 24-layer, approximately 1-billion-parameter connector and learnable queries are trained on text-to-image data while the MLLM and DiT remain frozen.The authors report that the prompted DiT performs better for text-to-image generation than with its original text encoders, such as T5.Source: §3.1 · p. 5
  • UniPic2-MetaQuery jointly fine-tunes the learnable queries, connector, and DiT while freezing the MLLM for image understanding, generation, and editing.For editing, the MLLM receives text instructions and reference images, while the DiT also receives reference-image VAE latents.Source: §3.1 · p. 5

3.2 Post-training

  • PDTR decouples task optimization into sequential stages: image-editing reinforcement first, followed by text-to-image reinforcement.The first stage uses GRPO and editing rewards; the second uses verifiable rewards, including GenEval compositionality metrics and automated detection.Source: §3.2 · p. 6
  • Reinforcing either task improves the other, and applying text-to-image reinforcement after editing does not degrade editing performance.The authors describe this as non-adversarial co-evolution within a shared model architecture.Source: §3.2 · p. 6
  • Skywork-EditReward is trained on 333k generated editing results scored by GPT-4.1 and is used to provide editing-quality rewards during reinforcement learning.The authors report that it captures quality differences traditional metrics cannot quantify and improves editing naturalness and consistency.Source: §3.2 · p. 6

4 Experiments

Experiments report strong generation and editing results for the 2B-parameter UniPic2-SD3.5M-Kontext, alongside multimodal capabilities from UniPic2-Metaquery. Ablations support progressive reinforcement and coordinated image conditioning, while failure cases remain on complex text and precise instructions.

4.1 Setup

  • Pretraining used 5M image-editing and 6M text-to-image samples; post-training kept hyperparameters fixed across reinforcement stages.The reported pretraining ran for 200K steps for UniPic2-SD3.5M-Kontext.Source: §4.1 · pp. 6, 8

4.2 Main results.

  • GenEval scores were 0.89 for UniPic2-SD3.5M-Kontext and 0.90 for UniPic2-Metaquery; Kontext exceeded FLUX.1-dev (0.67), SD3.5-Medium (0.65), and BAGEL (0.88).The comparison table covers image understanding, generation, and editing.Source: Table 1 · p. 8, §4.2 · p. 9
  • With a 2B-parameter generation module, UniPic2-SD3.5M-Kontext scored (6.59, 4.00) on GEdit and ImgEdit, exceeding 12B Flux-Kontext (6.26, 3.52) and 7B BAGEL (6.52, 3.20).UniPic2-Metaquery scored 6.87 on GEdit-En, compared with Step1X-Edit at 6.97.Source: Table 1 · p. 8, §4.2 · p. 9
  • UniPic2-Metaquery demonstrates multimodal understanding across recognition, scene description, structured object grounding, and OCR tasks.The reported examples include JSON-formatted bounding boxes and transcription of stylized text.Source: §4.2 · p. 9, Figure 5 · p. 11

4.3 Ablation Studies

  • Reference images passed to both the MLLM and DiT produced the best image-conditioning result, while unlocking both the connector and DiT produced the best UniPic2-MetaQuery result.The image-editing indicator for conditioning strategies was GEdit-EN.Source: §4.3 · pp. 9, 11, Table 2 · p. 9, Table 3 · p. 9
  • Joint editing and T2I reinforcement achieved GenEval 0.89, GEditBench-EN 6.59, and ImgEdit 4.00, with the study reporting positive cross-task synergy.Skywork-EditReward outperformed online GPT-4.1 evaluation across benchmarks and avoided costly API calls.Source: §4.3 · pp. 11–12, Table 4 · p. 12

4.4 Failure Cases

  • Figure 6 highlights failures in complex text rendering, precise object extraction, and detailed instruction following, with data scaling suggested as a possible improvement.Examples include distorted book-cover text and blurred artifacts when extracting an object.Source: §4.4 · p. 12

5 Conclusions

Skywork UniPic 2.0 combines pretrained diffusion and multimodal language models for unified understanding, generation, and editing. The paper reports strong benchmark performance and efficiency, while Figure 6 highlights persistent difficulties with lengthy, intricate prompts.

5 Conclusions

  • Skywork UniPic 2.0 integrates mature multimodal language models and pretrained diffusion generators through a parameter-efficient connector-based design.The framework is built on SD3.5-Medium and Qwen2.5-VL architectures.Source: §5 · p. 12
  • Figure 6 identifies failures on lengthy prompts with intricate semantics, complicated text, and detailed object-count requirements.Source: Figure 6 · p. 13
  • Skywork UniPic 2.0 reports state-of-the-art benchmark performance in instruction following, editing consistency, and generation stability while maintaining efficiency.The conclusion characterizes the paradigm as practical, scalable, and reproducible.Source: §5 · p. 13
Loading 2509.04548v2…