PaperlayerView exact revision on arXiv

Source-linked AI summary

RealUnify: Do Unified Models Truly Benefit from Unification? A Comprehensive Benchmark

Yang Shi, Yuhao Dong, Yue Ding, Yuran Wang, Xuanyu Zhu, Sheng Zhou, Wenting Liu, Haochen Tian, Rundong Wang, Huanqian Wang, Zuyan Liu, Bohan Zeng, Ruizhe Chen, Qixun Wang, Zhuoran Zhang, Xinlong Chen, Chengzhuo Tong, Bozhou Li, Qiang Liu, Haotian Wang, Wenjing Yang, Yuanxing Zhang, Pengfei Wan, Yi-Fan Zhang, Ziwei Liu

arXiv:2509.24897v2cs.AI

TL;DR

Existing benchmarks provide limited evidence about whether visual understanding and generation mutually enhance one another. RealUnify tests that bidirectional synergy across reasoning-guided generation and generation-supported understanding, using direct and stepwise evaluation; results indicate current unified models still struggle to achieve effective synergy.

  • Problem

    Existing benchmarks mostly assess understanding and generation separately and do not rigorously test whether unified models achieve bidirectional capability synergy.

    Source: §1 · p. 2

  • Method

    RealUnify evaluates understanding-enhanced generation and generation-enhanced understanding with direct end-to-end and diagnostic stepwise protocols across 1,000 instances, 10 categories, and 32 subtasks.

    Source: §1 · p. 2

  • Results

    Current unified models still struggle to synergize understanding and generation, despite their unified architectures.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Architectural unification alone is insufficient to achieve capability synergy, and the paper calls for more advanced training schemes and stronger inductive biases.

    Source: §1 · p. 3

Abstract

from arXiv · show

The integration of visual understanding and generation into unified multimodal models represents a significant stride toward general-purpose AI. However, a fundamental question remains unanswered by existing benchmarks: does this architectural unification actually enable synergetic interaction between the constituent capabilities? Existing evaluation paradigms, which primarily assess understanding and generation in isolation, are insufficient for determining whether a unified model can leverage its understanding to enhance its generation, or use generative simulation to facilitate deeper comprehension. To address this critical gap, we introduce RealUnify, a benchmark specifically designed to evaluate bidirectional capability synergy. RealUnify comprises 1,000 meticulously human-annotated instances spanning 10 categories and 32 subtasks. It is structured around two core axes: 1) Understanding Enhances Generation, which requires reasoning (e.g., commonsense, logic) to guide image generation, and 2) Generation Enhances Understanding, which necessitates mental simulation or reconstruction (e.g., of transformed or disordered visual inputs) to solve reasoning tasks. A key contribution is our dual-evaluation protocol, which combines direct end-to-end assessment with a diagnostic stepwise evaluation that decomposes tasks into distinct understanding and generation phases. This protocol allows us to precisely discern whether performance bottlenecks stem from deficiencies in core abilities or from a failure to integrate them. Through large-scale evaluations of 12 leading unified models and 6 specialized baselines, we find that current unified models still struggle to achieve effective synergy, indicating that architectural unification alone is insufficient. These results highlight the need for new training strategies and inductive biases to fully unlock the potential of unified modeling.

1. Introduction

Existing benchmarks do not rigorously test whether understanding and generation mutually enhance one another. RealUnify evaluates that bidirectional synergy, and results show current unified models still struggle to achieve it.

1. Introduction

  • Existing evaluations largely test understanding and generation separately, leaving little rigorous evidence about whether unified models achieve bidirectional capability synergy.Some benchmarks combine both domains, while T2I-CoReBench [18] and WISE [24] examine understanding-enhanced generation without explicitly testing whether success depends on both capabilities.Source: §1 · p. 2
  • RealUnify evaluates whether understanding improves generation and whether generative reconstruction or visualization supports visual reasoning.Its instances require interplay between the capabilities, including reasoning before image generation and visual tracking of transformations to answer questions.Source: §1 · p. 2
  • RealUnify contains 1,000 human-crafted and validated instances across 10 categories and 32 subtasks, designed to require interplay between understanding and generation.Its dual protocol pairs realistic end-to-end direct evaluation with stepwise decomposition to diagnose weak component abilities versus weak integration.Source: §1 · p. 2
  • RealUnify shifts evaluation beyond image aesthetics and textual relevance toward solving complex, real-world tasks through capability synergy.The benchmark is presented as a way to distinguish genuine synergy from functional coexistence.Source: §1 · p. 2
  • Despite unified architectures, current models struggle to synergize understanding and generation; stepwise UEG improves BAGEL [8] from 32.7% to 47.7%, while an oracle specialist combination reaches 72.7% on UEG.The oracle combines Gemini 2.5 Pro [7] for understanding and GPT-Image-1 for generation.Source: §1 · pp. 2–3

2. Related Work

Prior unified-model research has moved from enabling both functions in one model toward testing whether unification yields cross-task benefits. Existing benchmarks assess multiple capabilities, but provide limited evidence about reciprocal synergy.

Unified Models

  • Research interest has shifted from functional integration toward whether unification yields cross-task benefits or emergent abilities.Liquid [39] reports reciprocal cross-task benefits, UniFluid [9] strengthens such gains with training strategies, and BAGEL [8] reports complex compositional behaviors.Source: §2 · p. 3

Benchmarks for Unified Models

  • MME-Unify [42] jointly assesses comprehension, generation, and mixed-modality tasks, while UniEval [19] evaluates without auxiliary models or human annotations.These benchmarks still fall short of measuring whether integration produces performance gains.Source: §2 · p. 3
  • Text-to-image benchmarks including MMMG [23], T2I-CoReBench [18], and WISE [24] provide limited evidence of reciprocal understanding-generation benefits because they emphasize T2I tasks.Source: §2 · p. 3

3. RealUnify

RealUnify tests two directions of capability transfer through UEG and GEU tasks, with direct and stepwise evaluation. Its task suite covers reasoning-guided image generation, generation-supported understanding, and 32 subtasks.

3. RealUnify

  • RealUnify evaluates whether understanding enhances generation (UEG) and whether generative ability supports understanding (GEU).The tracks assess cross-capability transfer in complex tasks.Source: §3 · p. 3

3.1. Understanding Enhances Generation (UEG)

  • UEG tasks require interpreting prompts before generating images, spanning world knowledge, commonsense, and mathematical reasoning.World knowledge covers seven domains; mathematical tasks include calculation, probability, proportional reasoning, and constraint-based equation solving.Source: §3.1 · pp. 3–4
  • Other UEG tasks test reasoning over logical constraints, scientific principles, and code execution to produce images matching the inferred instruction.Scientific reasoning spans physics, chemistry, biology, and geography.Source: §3.1 · p. 4

3.2. Generation Enhances Understanding (GEU)

  • GEU tasks test whether generation supports understanding through reconstructing shuffled images, tracking visual transformations, focusing on salient regions, and navigating mazes or maps.Map navigation includes finding shortest routes or following paths under constraints.Source: §3.2 · pp. 4–5

3.3. Benchmark Construction

  • UEG prompts are curated by experts and independently cross-checked; GEU samples combine automated generation, expert annotation, and selected external sources.Attentional Focusing draws from BLINK [10] and HR-Bench [33], while map images come from Google Search API.Source: §3.3 · p. 5
  • Direct evaluation tests tasks end to end, using text-to-image for UEG and image-to-text multiple-choice questions for GEU; generated images are polled with verification questions.Gemini 2.5 Pro judges the polling-based image evaluation.Source: §3.3 · p. 5, Figure 3 · p. 5
  • Stepwise evaluation decomposes each task into two stages: understanding then generation for UEG, and generation then understanding for GEU.This decomposition supports fine-grained analysis of bottlenecks and whether unification brings performance gains.Source: §3.3 · p. 5
  • The benchmark contains 1,000 questions—600 UEG and 400 GEU—covering 10 categories and 32 subtasks.Source: §3.3 · p. 5, Figure 4 · p. 6

4. Experiments

RealUnify evaluations find that current unified models struggle to integrate understanding and generation end to end, despite gains under stepwise decomposition and some competitive specialist comparisons. Analysis experiments show both the promise of staged capability use and the remaining limits of unified models.

4.1. Evaluation on RealUnify

  • The evaluation covers 12 unified models and measures the performance gap against three high-performing specialist models each in visual understanding and image generation.The unified set includes 11 state-of-the-art open-source models and the proprietary Gemini-2.5-Flash-Image.Source: §4.1 · p. 6
  • Stepwise decomposition improves every model on UEG, with BAGEL gaining +15, but reduces every model’s GEU performance.The authors interpret the UEG gains as evidence that models retain needed knowledge but fail to use it inherently; GEU degradation suggests they struggle to apply generation to problem-solving.Source: §4.1 · p. 6
  • Under direct evaluation, the best open-source model scores 37.5% on UEG versus 63.0% for proprietary Nano Banana, while unified models perform poorly on both UEG and GEU.For GEU, open-source models show stronger understanding than proprietary models, but the paper says all models perform poorly.Source: §4.1 · p. 6, Table 2 · p. 7

4.2. Analysis Experiments

  • Unified models perform competitively on UEG, sometimes surpassing state-of-the-art image-generation specialists, while open-source models still trail proprietary counterparts on challenging generation tasks.On GEU, open-source unified models show strong understanding abilities, outperforming certain proprietary specialist models.Source: Table 3 · p. 7, §4.2 · pp. 7–8

4.1. Evaluation on RealUnify

  • Gemini 2.5 Pro agrees more closely with human image-evaluation experts than Qwen2.5-VL, supporting its use as the judge.The study compares both judges against averaged manual evaluations by four human experts.Source: §4.1 · pp. 6–7, Table 4 · p. 7

4.2. Analysis Experiments

  • A stepwise combination of Gemini 2.5 Pro and GPT-Image-1 scores 72.7 on UEG, while its GEU performance degrades substantially.The authors treat this combination as an approximation of current unified models’ upper bound and note that adapting generation for practical problem-solving remains challenging.Source: Table 5 · p. 7, §4.2 · p. 8
  • Bagel’s stepwise examples show reasoning identifying a lightsaber for generation and partial reconstruction guiding a GEU answer.The examples illustrate cases where staged integration solves tasks that neither capability alone could address.Source: Figure 5 · p. 8, §4.2 · p. 8
  • Bagel still underperforms with stepwise execution, but improves when given ground-truth intermediate results, pointing to limits in its individual capabilities.The authors suggest these capability limits may also hinder effective integration in real-world problem-solving.Source: Figure 6 · p. 8, §4.2 · p. 8

5. Conclusion

RealUnify evaluates whether unified models achieve synergy between understanding and generation, and the reported experiments find that current models remain far from genuine synergy.

  • RealUnify evaluates understanding-enhances-generation and generation-enhances-understanding across 10 categories, finding that current unified models remain far from genuine synergy.Models can accomplish tasks under stepwise decomposition, but their end-to-end performance indicates an absence of true synergy.Source: §5 · p. 8

Supplementary Material

Supplementary material provides examples, benchmark-construction details, implementation information, and generation failure modes.

  • Supplementary sections provide subtask visualizations and task-distribution and annotation details, implementation information for reproducibility, and common generation failure modes.Source: Supplementary Material · p. 11

A. Benchmark Details

The benchmark appendix presents representative examples across its UEG and GEU tasks.

A.1. Representative Examples from RealUnify

  • Two representative examples are provided for each task, with Figures 7–10 covering UEG and GEU examples.Source: §A.1 · p. 11

A.2. Task Distribution

RealUnify evaluates each task in both direct and stepwise settings, with stepwise evaluation separating visual understanding from generation.

A.2. Task Distribution

  • Each task is evaluated in both direct and stepwise settings.The stepwise setting decomposes the task into visual understanding and generation.Source: §A.2 · p. 11
  • Stepwise evaluation separates visual understanding from generation.Source: §A.2 · p. 11
  • Table 6 presents the distribution of task instances across RealUnify categories.Source: §A.2 · p. 11

A.3. Dataset Annotation and Verification

UEG prompts were designed by AI experts and reviewed for correctness, objectivity, and answer uniqueness; GEU samples used automated construction and human verification.

A.3. Dataset Annotation and Verification

  • Ten AI-specialist undergraduate and doctoral students designed the UEG prompts and question lists.Source: §A.3 · p. 11
  • Three independent doctoral-student reviewers checked each UEG sample for correctness, objectivity, and answer uniqueness.Source: §A.3 · p. 11
  • Mental Reconstruction, Mental Tracking, and Cognitive Navigation use automated construction pipelines followed by manual filtering and verification.Cognitive Navigation maps came from the Google Search API; samples were retained only with approval from all three reviewers.Source: §A.3 · p. 11
  • Attentional Focusing samples came from BLINK [10] and HR-Bench [33], with an additional round of human verification.Source: §A.3 · p. 11

B.1. Evaluation Setup

The evaluation covers 12 unified models and uses distinct prompts and procedures for UEG, GEU, and their stepwise evaluations.

B.1. Evaluation Setup

  • The benchmark evaluation includes 12 unified models: 11 open-source systems and proprietary Gemini 2.5 Flash Image, also called Nano Banana.Open-source models use official default or recommended inference settings.Source: §B.1 · p. 11

B.2. Evaluation Prompt

  • UEG image outputs are assessed by Gemini 2.5 Pro using a polling-based method and a specified evaluation prompt.Source: §B.1 · p. 11, §B.2 · p. 11
  • GEU tasks use multiple-choice questions, for which the benchmark supplies a dedicated evaluation prompt.Source: §B.2 · p. 11
  • Stepwise UEG first asks models to refine the original prompt, while stepwise GEU performs image generation or editing before visual understanding.GEU prompts are specified separately for Mental Reconstruction, Mental Tracking, Attentional Focusing, and Cognitive Navigation.Source: §B.2 · p. 11

C. Common Failure Modes of Unified Models in Generation Tasks

Unified models show recurring image-generation errors, including attribute and quantity mistakes, weak fidelity, spatial errors, and violations of detail or physical plausibility.

C. Common Failure Modes of Unified Models in Generation Tasks

Related papers

Loading 2509.24897v2…