PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantifying the Gap between Understanding and Generation within Unified Multimodal Models

Chenlong Wang, Yuhang Chen, Zhihan Hu, Dongping Chen, Wenhu Chen, Sarah Wiegreffe, Tianyi Zhou

arXiv:2602.02140v1cs.CL

TL;DR

Unified multimodal models increasingly perform both understanding and generation, but whether these capabilities are cognitively aligned remains unclear. The paper introduces GAPEVAL, a symmetric bidirectional benchmark, and supplements it with knowledge-manipulation experiments. Across diverse models, it finds a persistent gap and disjoint knowledge representations, indicating that current unification is largely functional rather than deeply integrated.

  • Problem

    Whether UMMs genuinely align understanding and generation, including implicit knowledge transfer across modalities, remains unclear.

  • Method

    The paper introduces GAPEVAL, whose semantically matched questions support text or image answers, and conducts single-sided knowledge injection and editing experiments.

  • Results

    Across diverse UMMs, experiments reveal a persistent understanding-generation gap, while single-sided fine-tuning often fails to transfer knowledge across modalities.

  • Takeaways & Limitations

    Current UMMs achieve engineering-level or functional coupling, but their embedded knowledge remains decoupled across modalities.

Abstract

from arXiv · show

Recent advances in unified multimodal models (UMM) have demonstrated remarkable progress in both understanding and generation tasks. However, whether these two capabilities are genuinely aligned and integrated within a single model remains unclear. To investigate this question, we introduce GapEval, a bidirectional benchmark designed to quantify the gap between understanding and generation capabilities, and quantitatively measure the cognitive coherence of the two "unified" directions. Each question can be answered in both modalities (image and text), enabling a symmetric evaluation of a model's bidirectional inference capability and cross-modal consistency. Experiments reveal a persistent gap between the two directions across a wide range of UMMs with different architectures, suggesting that current models achieve only surface-level unification rather than deep cognitive convergence of the two. To further explore the underlying mechanism, we conduct an empirical study from the perspective of knowledge manipulation to illustrate the underlying limitations. Our findings indicate that knowledge within UMMs often remains disjoint. The capability emergence and knowledge across modalities are unsynchronized, paving the way for further exploration.

1. Introduction

Unified multimodal models aim to combine understanding and generation, but existing evaluations rarely test whether the two capabilities share knowledge or reinforce one another. GAPEVAL addresses this gap, while experiments show persistent cross-modal misalignment and only surface-level unification.

  • UMMs combine textual responses to text or image inputs with image generation from textual inputs, creating a potential synergy between understanding and generation.
  • Most existing benchmarks evaluate understanding or generation separately, while recent unified frameworks still do not genuinely quantify the gap between the two capabilities.
  • GAPEVAL uses bidirectional questions answerable in image or text, with a Gap Score based on multidimensional item response theory to quantify the performance disparity.
  • Across diverse UMMs, models often answer correctly in one modality but fail to use the same underlying knowledge after switching modalities.
  • Knowledge transfer across modalities remains unclear because prior work emphasizes explicit reasoning rather than implicit reinforcement of one capability by the other.
  • Single-sided knowledge manipulation reveals pronounced misalignment: fine-tuning one capability often has little effect on the other, while injection can introduce cross-modal inconsistencies.

2. GAPEVAL: Quantifying the Gap between Understanding and Generation

GAPEVAL is a bidirectional benchmark that evaluates the same underlying questions through text and image responses across several capability categories. Its two-stage evaluation uses judged correctness and MIRT-based gap measurement to account for item difficulty and cross-task consistency.

  • 2.1. Motivation & Overviw: Each GAPEVAL question supports text- and image-based answers with shared semantic intent, enabling evaluation of identical underlying knowledge across modalities.
  • Instruction Following: Instruction Following tests explicit and implicit image edits to assess whether textual descriptions and visual executions remain aligned.
  • Numerical Perception: Numerical Perception couples counting across modalities through transformations such as swapping the quantities of two object categories.
  • World Knowledge: World Knowledge evaluates whether models map visual evidence to factual entities across animals, plants, landmarks, instruments, literature, and culture.
  • Reasoning: Reasoning requires textual and visual answers across image selection, knowledge selection, real-world reasoning, and logical reasoning tasks.
  • 2.3. Metric Design: The evaluation first assigns binary correctness labels using GPT-5-mini, then applies MIRT to estimate model ability and item difficulty while encouraging consistency between tasks.

3. Evaluation Results on GAPEVAL

GAPEVAL results show that unified multimodal models often lack balanced cross-modal performance, with understanding and generation developing unevenly. The gap can widen at intermediate capability levels, while stronger models may reduce it without making performance and unification equivalent.

  • Critical Performance Gap: Many UMM samples are answered correctly in only one modality, indicating limited cross-modal consistency.OmniGen2 performs markedly worse on generation than its FLUX.1-dev diffusion backbone, especially on World Knowledge and Instruction Following.
  • Evaluation Setup: The evaluation covers 9 UMMs and 6 non-UMMs on GAPEVAL, comparing success, understanding, generation, failure, and gap measures.Success denotes correct answers in both modalities; understanding and generation denote text- and image-answer accuracy, respectively.
  • Critical Performance Gap: Unified architectures can partially transfer reasoning ability from understanding to generation, but often at the cost of balanced overall performance.Understanding-only models consistently outperform UMMs on comprehension-oriented tasks, whereas closed-source UMMs surpass generation-only baselines on generation tasks.
  • Performance & Unification: Higher overall performance does not necessarily imply stronger unification: OneCAT has a smaller gap than stronger Bagel, while UniWorld-V1 combines high accuracy with small gaps.The results indicate a decoupling between state-of-the-art performance and the degree of unification.
  • Transition in Unification Across Performance Levels: The modality gap initially widens and later narrows as capability increases across models.Low-capability models can have small gaps because both modalities fail, intermediate models improve understanding earlier than generation, and high-performance models gradually reduce the gap.

4. UNIFIED KNOWLEDGE: Empirical Analysis on the UMMs Gap

The study probes cross-modal knowledge consistency in UMMs through knowledge manipulation and training-trajectory analysis. Results show that modality-specific updates remain poorly synchronized, with understanding generally improving earlier than generation.

  • 4.1 Knowledge Formulation: Knowledge entities are represented as (subject, relation, object) tuples, with subjects and objects expressed as text or images for multimodal evaluation.The experiments focus on mappings between textual objects and corresponding visual representations.
  • 4.3 Experiment Setup: The experiments manipulate knowledge through injection or editing, using about 100 instances reformulated as Image-to-Text and Text-to-Image tasks with 20% held out.Three UMMs are selected, and GPT5-mini or CLIP evaluates the resulting understanding and generation behavior.
  • 4.4 Empirical Results: Fine-tuning one modality produces limited transfer to the other, revealing distinct knowledge representations and especially strong misalignment during knowledge editing.Understanding fine-tuning reaches 0.62 for Bagel and 0.56 for OmniGen2, while generation fine-tuning reaches 0.89 for Show-o in the reported examples.
  • 4.4 Empirical Results: Knowledge injection shows milder but still limited cross-modal influence, while modality-specific knowledge drift prevents coherent updates across the unified model.The findings characterize functionality as unified while embedded knowledge remains non-unified.
  • 4.4 Empirical Results: Unbalanced and slower convergence across modalities supports modality-specific learning dynamics and explains delayed performance alignment.Figure 6 compares training strategies in which improvement on one side does not affect the other side.
  • 4.4 Empirical Results: During knowledge editing, understanding rises sharply and saturates near 0.8 after a few thousand steps, whereas generation improves more slowly and requires a larger training budget.The generation curve eventually approaches understanding performance, indicating asymmetric convergence rather than permanently absent learning.

5. Related Works

UMMs seek unified cross-modal understanding and generation through diverse architectures, but prior benchmarks largely evaluate these capabilities separately or emphasize text-to-image generation.

  • Unified Multimodal Models: UMMs combine multimodal processing with cross-modal understanding and generation using architectures such as early fusion, autoregressive-plus-diffusion modeling, MoE, and MoT.These designs allocate or combine model components differently across text and visual generation.
  • Evaluation of UMMs: Existing benchmarks commonly target understanding-only or generation-only models, while T2I-CoReBench and WISE provide mainly text-to-image evaluation.This leaves limited insight into how the two capabilities integrate within UMMs.

6. Conclusion

The paper introduces GAPEVAL to measure the gap between understanding and generation, finding engineering-level unification and knowledge decoupling across the two capabilities.

  • 6. Conclusion: GAPEVAL is a bidirectional benchmark for quantifying the gap between understanding and generation in UMMs.The benchmark evaluates state-of-the-art models across diverse architectures before probing the gap through knowledge manipulation tuning.
  • 6. Conclusion: Knowledge manipulation experiments reveal decoupling between understanding and generation, motivating further work toward deeper UMM unification.The conclusion characterizes current systems as achieving engineering-level rather than deeper unification.

A. Detailed Metric Implementation (MIRT-MAP Version)

The metric jointly estimates text-understanding and image-generation abilities with a two-dimensional MIRT model under Bayesian MAP estimation, then normalizes and adjusts their capability gap using cross-modal success statistics.

  • Input statistics: The metric aggregates paired correctness outcomes into four count categories for text-only, image-only, joint-success, and joint-failure cases.These counts are collected per model before deriving marginal statistics.
  • Model specification: MIRT assigns each model a latent ability vector spanning text understanding and image generation.The model extends the Rasch model to two dimensions and jointly estimates both abilities.
  • Optimization: The joint log-likelihood and Gaussian prior form the total MAP objective optimized over model abilities, item parameters, and prior parameters.Gradient-based MAP estimation jointly updates the relevant parameters.
  • Bayesian coupling: A shared multivariate Gaussian prior couples the modality-specific abilities while allowing modality difficulty and cross-modality correlation to be learned adaptively.The covariance is parameterized through a Cholesky factor and kept positive definite during optimization.
  • Gap normalization: The normalized absolute gap is transformed for interpretability, then a logit-space reward–penalty adjustment uses co-success and co-failure rates.Positive λ_fail enlarges the gap under co-failure, whereas positive λ_succ shrinks it under co-success; the default setting is λ_fail = λ_succ = 2.

B. Reliability Analysis

Reliability analysis examines judge-model agreement and relates Gap Score to synergy performance, while the benchmark’s factual targets reduce preference bias relative to open-ended generation.

  • Judge reliability: GPT-Image-1 has the lowest Gap Score across the evaluated judge models.GPT5-mini and Gemini3-Flash produce consistent relative rankings, with Pearson correlation 0.9656 between their judgments.
  • Judge reliability: The benchmark’s semantic correctness targets are objective, making MLLM-as-a-Judge suitable and reducing model-specific preference bias.The evaluation focuses on definite properties such as specific objects and themes rather than open-ended generation quality.
  • Gap and synergy: Lower Gap Scores are strongly negatively correlated with higher performance on GIR-Bench synergy evaluations.The reported relationship connects narrower modality gaps with stronger synergy benchmark performance.
  • Gap and synergy: Synergy requires understanding to guide generation and generation to reflect understanding, whereas disjoint internal representations obstruct information transfer.The paper presents alignment as the structural prerequisite for such complementarity.
  • Benchmark coverage: The benchmark’s data cover world knowledge, numerical perception, instruction following, and reasoning through paired text-and-image tasks.Construction includes curated knowledge entities, controlled counting scenes, rule-based image edits, and complementary reasoning pipelines.

D.2. Empirical Study Data Collection

The empirical study constructs separate knowledge-injection and knowledge-editing datasets, filtering entities and prompts to support bidirectional analysis of knowledge manipulation.

  • Study design: The study builds two datasets targeting knowledge injection and knowledge editing to examine how UMMs handle manipulated knowledge.Injection introduces novel concepts, whereas editing alters existing conceptual associations.
  • Knowledge injection: Knowledge-injection candidates are drawn from approximately 2,000 visually distinctive, low-prominence entities across diverse domains.The collection targets entities likely absent from model pre-training data and gathers more than ten high-quality images per candidate.
  • Knowledge injection: The filtration protocol retains approximately 100 objects that all subject models fail to recognize visually and generate from text.Each candidate is tested with five randomly sampled images for identification and with its entity name for text-to-image generation.
  • Knowledge editing: Knowledge-editing data contain approximately 50 related object pairs selected for conceptual swapping, using common entities with strong prior model knowledge.Examples include boat↔car and camera↔microwave, with Qwen-Image generating multiple images for quality and consistency.
  • Study design: The strictly filtered, bidirectionally constructed datasets provide the foundation for analyzing knowledge manipulation mechanisms in UMMs.Their construction is intended to support cross-modal comparisons of manipulated knowledge.

E.1. Benchmark Evaluation Details

GAPEVAL evaluates UMMs with bidirectional text-and-image queries, repeated sampling, judge-based correctness assessment, and MIRT-derived Gap Scores, motivating knowledge-manipulation experiments.

  • Evaluation setup: The evaluation covers nine unified models, four understanding-only models, and two generation-only models.Each model is evaluated on bidirectional queries that can be answered through textual or visual modalities.
  • Evaluation setup: Ten independent sampling runs are performed for each output modality and question, with average accuracy and Gap Score reported as final results.This procedure is applied across the benchmark’s bidirectional queries.
  • Evaluation metrics: LLM-as-a-judge assesses response correctness using eight prompts spanning four benchmark categories and two modalities.The prompts are designed to support robust assessment across the benchmark’s evaluation dimensions.
  • Evaluation metrics: MIRT aggregates accuracy and related performance indicators to quantify the Gap Score between modalities.The detailed formulation is provided as a multidimensional item-response-theory metric.
  • Observed gap: UMMs often recall embedded knowledge in one modality but fail to apply it consistently when the modality changes.This disparity motivates testing whether knowledge is shared, isolated, or conflicting across capabilities.
  • Knowledge manipulation: Fine-tuning injects or edits knowledge in one modality and evaluates both modalities to test whether changes propagate across capabilities.Localized effects indicate separate knowledge representations, whereas cross-modal changes would support shared knowledge.
  • Knowledge manipulation: The knowledge-manipulation study evaluates Bagel, OmniGen2, and Show-o, using disjoint single-modality fine-tuning for OmniGen2 and Show-o.Bagel requires a different approach because single-modality fine-tuning causes complete capability loss in its official implementation.
  • Knowledge manipulation: Knowledge-editing generation is judged for semantic consistency, while knowledge-injection generation uses CLIP similarity to assess reproduction of newly injected visual features.The two metrics reflect the distinct goals of editing common objects and synthesizing novel objects.
Loading 2602.02140v1…