PaperlayerView exact revision on arXiv

Source-linked AI summary

UniG2U-Bench: Do Unified Models Advance Multimodal Understanding?

Zimo Wen, Boxiu Li, Wanbo Zhang, Junxiang Lei, Xiaoyu Chen, Yijia Fan, Qi Zhang, Yujiang Wang, Lili Qiu, Bo Li, Ziwei Liu, Caihua Shan, Yifan Yang, Yifei Shen

arXiv:2603.03241v1cs.CVcs.AI

TL;DR

It remains unclear when generation can improve multimodal understanding, and existing benchmarks do not systematically isolate this capability. UniG2U evaluates unified models against matched base VLMs across diverse G2U tasks, finding that gains are task- and architecture-dependent rather than universal.

  • Problem

    Existing benchmarks rarely evaluate whether generated visual intermediates intrinsically improve understanding beyond recognition, classification, or language-based reasoning.

  • Method

    UniG2U benchmarks unified models against matched base VLMs under shared protocols across seven cognitive regimes to isolate generation-to-understanding capability.

  • Results

    Unified models generally underperform base VLMs, while gains concentrate in spatial transformation, illusion-sensitive, and multi-round reasoning tasks; task and architecture patterns also correlate.

  • Takeaways & Limitations

    Generation improves understanding selectively, indicating that effective unified multimodal systems require stronger integration between generation and reasoning.

  • Takeaways & Limitations

    Fine-grained subtasks have modest sample sizes because genuinely synergistic tasks are scarce in existing datasets.

Abstract

from arXiv · show

Unified multimodal models have recently demonstrated strong generative capabilities, yet whether and when generation improves understanding remains unclear. Existing benchmarks lack a systematic exploration of the specific tasks where generation facilitates understanding. To this end, we introduce UniG2U-Bench, a comprehensive benchmark categorizing generation-to-understanding (G2U) evaluation into 7 regimes and 30 subtasks, requiring varying degrees of implicit or explicit visual transformations. Extensive evaluation of over 30 models reveals three core findings: 1) Unified models generally underperform their base Vision-Language Models (VLMs), and Generate-then-Answer (GtA) inference typically degrades performance relative to direct inference. 2) Consistent enhancements emerge in spatial intelligence, visual illusions, or multi-round reasoning subtasks, where enhanced spatial and shape perception, as well as multi-step intermediate image states, prove beneficial. 3) Tasks with similar reasoning structures and models sharing architectures exhibit correlated behaviors, suggesting that generation-understanding coupling induces class-consistent inductive biases over tasks, pretraining data, and model architectures. These findings highlight the necessity for more diverse training data and novel paradigms to fully unlock the potential of unified multimodal modeling.

1 Introduction

UniG2U introduces a benchmark designed to determine whether generative capacity actively improves multimodal understanding, rather than merely coexisting with it. Evaluations reveal that unified models usually underperform base VLMs, while generation helps selectively on transformation-intensive reasoning tasks and exhibits structured task- and model-level correlations.

  • Motivation and benchmark: UniG2U fills a benchmark gap by evaluating understanding through generation and pairing unified models with matched base VLMs and inference protocols.The testbed covers 3,000 curated instances across seven reasoning categories with standardized data, protocols, and scripts.
  • Evaluation design: Over 30 models are benchmarked across seven reasoning regimes using Direct and Generate-then-Answer inference modes.The study includes base VLMs, unified models, and agentic models across autoregressive, diffusion-based, and hybrid architectures.
  • Main findings: Unified models generally underperform corresponding base VLMs, while Generate-then-Answer typically performs worse than direct inference by propagating visual errors.This degradation is especially associated with structurally constrained domains.
  • Main findings: Generation consistently helps specific spatial, illusion-sensitive, and multi-round reasoning subtasks where visual transformations and intermediate image states support perception and reasoning.The reported mechanisms include enhanced spatial and shape perception and beneficial multi-step intermediate states.
  • Mechanistic analysis: G2U gains cluster by cognitive demand, and models sharing a base model correlate strongly, whereas architectural similarity alone yields much weaker behavioral correlation.These patterns motivate analysis of architectural trade-offs, GtA vulnerabilities, and intermediate alignment fidelity.

2 Related Works

Multimodal research has progressed from disjoint vision–language and generation pipelines toward unified models and benchmarks. UniG2U addresses the remaining gap by systematically evaluating generation-to-understanding across fine-grained regimes while isolating generation’s synergistic effects.

  • Unified multimodal models: Early multimodal systems separated vision–language understanding from image generation, whereas unified models combine both capabilities through shared representations and tighter coupling.Token-based early-fusion and hybrid autoregressive–diffusion/flow formulations support unified multimodal behavior.
  • Existing benchmarks: Existing benchmarks operationalize unification through capability coverage, mixed-modality outputs, or explicit bidirectional synergy protocols.MME-Unify audits understanding, generation, and mixed tasks, including text-option plus image-option selection and CLIP-style image judgments.
  • Benchmark gap: Existing benchmarks do not diagnose which generation-to-understanding tasks benefit intrinsically from intermediate visuals or how performance varies across fine-grained regimes and model archetypes.MME-U does not isolate synergy or define G2U task structure, while RealUnify and Uni-MMMU cover narrower task slices.
  • UniG2U: UniG2U evaluates 3,000 understanding tasks across 30 subtasks and 7 reasoning regimes using over 30 models, systematically quantifying generation’s synergistic effect on understanding.The benchmark curates G2U-centric regimes and pairs unified models with purely discriminative base VLMs to isolate G2U capability and analyze generation-understanding coupling.

3 Formulations and Definitions

This section formalizes unified multimodal models, Generation Helps Understanding (G2U), and the Direct versus Generate-then-Answer (GtA) protocols. It also defines a taxonomy distinguishing end-to-end, decoupled, and agentic systems, and isolates generation-related gains against matched base VLMs.

  • 3.1 Definitions: UMMs and G2U: Each task instance is represented as x = (V, T, Q), with visual context V, textual context T, query Q, and target answer y ∈ Y.
  • 3.1 Definitions: UMMs and G2U: A UMM combines multimodal understanding with a generative interface that produces an intermediate visual artifact G, such as an image, sketch, annotation, reconstruction, or edit.
  • 3.1 Definitions: UMMs and G2U: G2U gain is defined as ΔG2U(MUM; D) = Perf(MUM; D) − Perf(B(MUM); D), comparing each unified system with its matched purely discriminative base VLM.Capability pairing and matched computation budgets are intended to attribute the gain to generation-to-understanding synergy rather than model scale or empirical noise.
  • 3.2 Taxonomy of Unified Multimodal Models: Evaluated UMMs comprise three architectural categories: end-to-end unified models (E2E), decoupled unified systems, and agentic unified models (UM-Ag).
  • 3.2 Taxonomy of Unified Multimodal Models: E2E models jointly train generation and answering within one shared parameter space, whereas decoupled systems use separate generation and frozen answering modules connected through an intermediate artifact.
  • 3.2 Taxonomy of Unified Multimodal Models: UM-Ag systems unify separate models through control policies or tool-use protocols, and are analyzed separately because they generally lack strictly matched base VLMs.
  • 3.3 Direct vs. Generate-then-Answer (GtA) Inference: Direct inference answers without an external visual artifact, while GtA explicitly generates an intermediate visual rationale and re-consumes it before producing the final answer.GtA may use a single generated artifact or an agentic closed-loop rollout with multiple intermediate states.
  • 3.3 Direct vs. Generate-then-Answer (GtA) Inference: The framework separates intrinsic unified-training effects from explicit visual-reasoning externalization, while noting that inaccurate, implausible, or misaligned artifacts can mislead answering.

4 The Setup of UniG2U

UniG2U is a diagnostic benchmark designed to test when intermediate visual generation benefits multimodal understanding. It organizes approximately 3,000 samples into seven reasoning categories and 30 subtasks spanning generation-friendly and perception-heavy cognitive regimes.

  • Benchmark scope: UniG2U contains approximately 3,000 samples organized into 7 high-level categories and 30 fine-grained subtasks.The benchmark is designed for fine-grained analysis rather than evaluation of a single task family.
  • Design principles: The taxonomy prioritizes distinct cognitive structures and selects tasks with theoretical potential for Generation-to-Understanding synergy through visual externalization.Examples include geometric construction, physical simulation, spatial transformation, and fine-grained perceptual reconstruction.
  • Task taxonomy: The seven categories cover real-world applications, geometry, physics, puzzles and games, charts and tables, spatial intelligence, and perception reasoning.The categories range from structured intermediate-state reasoning to fine-grained visual perception.
  • Diagnostic purpose: The benchmark enables analysis of when intermediate visual generation is helpful, neutral, or harmful across generation-friendly and perception-heavy regimes.Its selection emphasizes settings where intermediate generation is likely to assist understanding, despite the scarcity of tasks with genuine G2U synergy.
  • Task taxonomy: Category sizes are 200 samples each for real-world applications, geometry, and physics; 537 for puzzles and games; 100 for charts and tables; 500 for spatial intelligence; and 1,263 for perception reasoning.This distribution supports comparisons across cognitive regimes from spatial manipulation to primarily perceptual recognition.
  • Evaluation setup: UniG2U also introduces RA and AL to evaluate the effectiveness of GtA-generated intermediate images using GPT-4o task-family rubrics on a 1-to-5 scale.These metrics are intended to mechanically characterize the success or failure of generated intermediates.

5 Results Analysis

Unified models generally incur an alignment tax in understanding, while gains emerge selectively in spatial, illusion, and transformation-intensive tasks. Direct inference usually avoids visual error propagation, whereas GtA helps when intermediate visuals are reliable structural scaffolds; task and model behaviors cluster by reasoning structure and inherited VLM base.

  • Overall Trends: Unified models generally underperform base VLMs because generation-understanding coupling introduces an intrinsic alignment tax and objective interference, even under Direct inference.The degradation is attributed to parameter-level coupling rather than low-quality generated intermediates, compromising fine-grained discriminative reasoning.
  • Category-Level Behaviors: Unified models show resilience or gains in Spatial Intelligence and Visual Illusions, where geometric transformations, state tracking, and visual ambiguity benefit from generative structural regularization.Generation appears beneficial when objectives are coherently aligned with spatial reasoning, but otherwise acts as an interfering auxiliary objective.
  • Overall Trend: GtA usually degrades logic-intensive performance because inaccurate, semantically misaligned, or poorly integrated generated visuals propagate errors, whereas Direct inference avoids committing to flawed artifacts.Across categories, most GtA points fall below the y = x line, and aggregate GtA accuracy is frequently lower than Direct.
  • Transformation-Intensive Regimes: GtA improves Maze navigation, Sliding puzzles, and Multi-Step Spatial Reasoning when explicit intermediate visual states provide a structurally accurate, semantically aligned workspace.The decisive factor is whether generation functions as a reliable scaffold for sequential spatial logic, not explicit generation alone.
  • Correlation Analysis: Perception-oriented and reasoning-oriented subtasks form internally correlated but typically negatively correlated clusters, while models sharing the exact same Base VLM correlate more strongly than models sharing only architecture.Structural reasoning subtasks such as jigsaw and maze correlate positively, and inherited foundational representations predominantly dictate behavioral consistency.
  • Alignment Fidelity: High alignment fidelity is necessary but insufficient: explicit externalization is redundant for faithful Perception drawings, yet poor alignment in structurally constrained domains causes severe error propagation.The result reinforces that G2U gains depend on alignment between generated visuals and the target reasoning structure.

6 Conclusion and Future Work

UniG2U is a large-scale diagnostic benchmark showing that generation-to-understanding gains depend strongly on task and architecture rather than consistently improving over base VLMs. Future work should strengthen generation–reasoning integration through alignment, verification, closed-loop refinement, and broader reasoning artifacts.

  • Conclusion: UniG2U evaluates generation-to-understanding across 3,000 curated instances spanning seven cognitive regimes.It is introduced as a large-scale diagnostic benchmark for studying when and how generation helps understanding in unified multimodal models.
  • Conclusion: Generation-to-understanding gains are highly task- and architecture-dependent, with improvements emerging primarily in spatial transformation and state-tracking tasks.Unified models do not consistently outperform their base VLMs.
  • Future Work: Future directions include representation-level alignment, reliability-aware or self-verifying intermediate generation, and closed-loop agentic refinement to mitigate error propagation.The passage also proposes extending beyond image-only intermediates to structured or multimodal reasoning artifacts.

A Performance Analysis of Unified Models with Frozen Backbones (Non-CoT Version)

Table 8 reports Direct (Non-CoT) performance for unified models with frozen backbones, with all results given as percentages and Overall averaging task-category accuracies. Although these models should theoretically match qwen2.5-vl-7b, empirical results show measurable deviations attributed to implementation-level factors.

  • Results: Direct (Non-CoT) results for frozen-backbone unified models are reported as percentages, with Overall defined as the average accuracy across all task categories.The table reports values as percentages (×100).
  • Expected equivalence: Frozen-backbone unified models should theoretically exactly match qwen2.5-vl-7b because the generative pathway is bypassed and the base VLM backbone remains frozen during training.This expectation follows from the stated model configuration.
  • Observed deviations: Empirical results nonetheless show measurable deviations from the base model, which are attributed to implementation-level factors.The passage introduces subtle weight differences as one such factor, but the provided text is truncated before listing all factors.

B Ablation Study for Models without Edit Training

For edit-untrained unified models, Generate-then-Answer introduces an out-of-distribution intermediate-generation setting that consistently harms performance. The ablations show that chaotic, irrelevant generations can mislead answering, confirming that G2U gains require explicit, training-aligned generation capabilities.

  • Motivation: Edit-untrained models face an out-of-distribution setting when Generate-then-Answer requires intermediate image editing absent from training.Including these results in the main comparison could introduce unfair bias.
  • Ablation Variants: The ablation tests OOD-Image Generation for UAE and X-Omni, and Prompt-Only Intermediate Generation for TokenFlow-XL.UAE and X-Omni generate from the original image, whereas TokenFlow-XL generates from the textual prompt alone before answering.
  • Results: 2.33%, 10.99%, and 4.65% overall performance drops occur for UAE, X-Omni, and TokenFlow-XL, respectively, versus Direct inference.The first two declines result from forced OOD-Image generation, while TokenFlow-XL uses Prompt-Only generation.
  • Interpretation: Intermediate images are often chaotic, structurally irrelevant, or information-deficient, actively misleading the subsequent answering stage.These qualitative failures support excluding edit-untrained Generate-then-Answer scores and indicate that G2U gains depend on explicit, training-aligned generation.

C Additional Result Analysis: Structured Subtask Gains

Unified models show clearer, repeatable G2U gains on structured, transformation-centric subtasks—especially spatial intelligence and illusion-related reasoning—while recognition-heavy tasks lack systematic improvement and may degrade.

  • Structured Subtask Gains: Spatial Intelligence and illusion-related perceptual reasoning show the most consistent positive offsets over base VLMs.These subtasks require transformation-aware reasoning, geometric consistency, or resolving visually ambiguous patterns.
  • Structured Subtask Gains: UniWorld-V1, JavisGPT, uni-video, and STAR-7B outperform their corresponding base models on the spatial Perception Reasoning subtask.Logo-related and shape-sensitive illusion categories also improve in certain unified configurations.
  • Recognition-Heavy Subtasks: Recognition-heavy subtasks such as icon_scene and in_scene show no systematic improvement and sometimes mildly degrade.This contrast indicates that G2U benefits emerge selectively in transformation-centric or structurally constrained regimes.

D Failure Case Taxonomy for Generation-Augmented Understanding

The appendix organizes generation-augmented reasoning failures into three categories based on how generated diagrams affect downstream performance. Across these categories, successful generation requires structural validity, semantic relevance, and reasoning utility.

  • Taxonomy: The taxonomy distinguishes capability failure, surface-relevant but non-utility generation, and irrelevant generation.These categories reflect whether diagrams are invalid, relevant but unhelpful, or misaligned with the task.
  • Category I: Capability Failure: Capability failure produces geometrically distorted, structurally inconsistent, or physically invalid diagrams that make downstream reasoning unreliable or impossible.Typical errors include violated geometric relations, distorted quantities, incorrect force directions, and missing constraints.
  • Category I: Capability Failure: Invalid diagrams can be treated as ground truth, causing incorrect intermediate computations or forcing solvers to ignore the intended benefit of visual externalization.Suggested mitigations include lightweight validation or self-checking before solver use, or routing uncertain cases to pure-text solutions.
  • Category II: Surface-Relevant but Non-Utility Generation: Surface-relevant generation produces on-topic diagrams that add no constraints or computational benefit, potentially wasting generation budget and distracting reasoning.The generator should instead produce minimal, goal-conditioned structures and skip generation when predicted utility is low.
  • Category III: Irrelevant Generation: Irrelevant generation reflects complete generation–task misalignment, such as drawing for factual questions or failing to preserve chart schemas.Such outputs can introduce spurious cues, reduce trust, and complicate debugging; direct answers or structured data preservation are preferable.

E Uni-MMMU Task Design

Uni-MMMU comprises three task types—Jigsaw, Maze, and Sliding Puzzle—that evaluate visual reasoning through interleaved image-text generation.

  • Uni-MMMU consists of three task types: Jigsaw, Maze, and Sliding Puzzle.
  • The three tasks evaluate different aspects of visual reasoning.
  • Each task uses interleaved image-text generation as part of its visual reasoning evaluation.

E.1 Jigsaw Completion Task … E.3.2 Visual CoT Generation Flow

This section defines jigsaw completion, maze solving, and sliding-puzzle tasks with structured textual outputs and visual chain-of-thought generation procedures. The generated images represent candidate completions or successive puzzle states under task-specific consistency constraints.

  • E.1.1 Task Description: Jigsaw completion selects which of two candidate patches correctly completes a reference image, reporting binary jigsaw_text_acc over 100 samples.The required output is <FINAL_ANSWER_JSON>{"choice": 0 or 1, "rationale": "..."}.
  • E.1.2 Visual CoT Generation Flow: Jigsaw visual CoT generates two 2×2 completions, placing each candidate in the bottom-right cell while preserving the other three reference cells pixel-identically.The model then analyzes both generated completions and returns a choice with a rationale.
  • E.2.1 Task Description: Maze solving finds legal up, down, left, or right moves from a blue starting dot to a green goal region without diagonals or wall crossing.Outputs are evaluated with maze_text_exact and maze_text_frame_acc on 100 samples.
  • E.2.2 Visual CoT Generation Flow: Maze visual CoT generates one image after each solution move, changing only the blue dot position while keeping all other maze elements identical.The number of generated images must equal the number of moves, excluding the initial state and disallowing collages, arrows, or annotations.
  • E.2.2 Visual CoT Generation Flow: Across maze and sliding-puzzle flows, each generated image encodes exactly one intermediate state, with the final answer emitted separately as a structured move list.Maze states preserve non-position elements, while sliding states preserve tile colors.
  • E.3.1 Task Description: The sliding-puzzle task solves a 3×3 configuration with eight colored tiles and one empty red space, naming moves by the direction the colored tile moves.It reports sliding_text_exact and sliding_text_frame_acc on 54 samples.
  • E.3.2 Visual CoT Generation Flow: Sliding-puzzle visual CoT generates an image after each move, changing tile positions while preserving tile colors identically before producing the final move sequence.The procedure excludes the initial state, requires one image per move, and forbids collages or multi-panel images.

F Experimental Results … AL addition

The supplied material specifies non-CoT and Visual CoT prompts across multimodal perception, reasoning, puzzle-solving, geometry, physics, chart, and spatial tasks. Visual CoT uses generated auxiliary visualizations or intermediate states before answering the original question, with strict output constraints for several tasks.

  • G Task Prompts (Non-CoT Version): Non-CoT prompts direct models to inspect appearance attributes, measurable properties, camera motion, and object motion before selecting an answer option.Appearance instructions emphasize color, shape, texture, orientation, count, and visual features; motion prompts analyze viewpoint or object changes across consecutive images.
  • G Task Prompts (Non-CoT Version): ICON, LOGO, and IN recognition prompts ask models to identify either the integrated shape or its background from task-specific candidate class lists, returning one class name.The three scene-recognition tasks share identical prompts apart from shape_string values, while shape-recognition prompts target the icon, logo, or shape.
  • G Task Prompts (Non-CoT Version): Puzzle Visual CoT requires unified models to generate intermediate image states for jigsaw completion, maze navigation, and sliding-puzzle moves while preserving strict visual constraints.Jigsaw prompts require exact candidate insertion and comparison; maze and sliding-puzzle prompts require one separate image per legal move followed by a single JSON move list.
  • H Task Prompts (Visual CoT Version): Visual CoT adds a two-stage format in which generation produces an auxiliary visualization and questioning uses it alongside the original input.The appendix explicitly defines [GEN_PROMPT] and [QUESTION] stages, with generated annotations tailored to the target reasoning or perception task.
  • H Task Prompts (Visual CoT Version): Visual CoT visualizations expose appearance, measurable attributes, camera or object motion, multi-step changes, and background elements forming icons, scenes, logos, or shapes.The prompts request annotations such as bounding boxes, measurement markers, motion arrows, trajectories, reasoning diagrams, and outlines of integrated visual forms.
  • H Task Prompts (Visual CoT Version): Reasoning Visual CoT prompts cover algorithmic, analogical, deductive, inductive, and spatial transformations through diagrams, inference chains, pattern annotations, and multi-view constructions.Answering generally requires using the original puzzle together with the generated diagram and selecting an option letter directly.
  • H Task Prompts (Visual CoT Version): Chart, geometry, solid-geometry, optics, and mechanics prompts generate task-specific auxiliary diagrams before solving with highlighted data, constructions, rays, or forces.The solid-geometry prompt additionally requires preserving the original diagram, applying relevant theorems, showing calculations, and stating the final numerical answer.

I.11 Notes on Robustness

The evaluation pipeline includes fallback handling for judge-response parsing and defines null-score behavior when retries are exhausted. It also supports multi-image tasks by concatenating original images before auxiliary images.

  • Regex extraction serves as a fallback when JSON parsing of the judge response fails.
  • Scores are returned as null if all retries fail.
  • Multi-image tasks concatenate all original images before auxiliary images.
Loading 2603.03241v1…