Source-linked AI summary
When Does Visual Generation Help Visual Understanding in Unified Multimodal Models?
Yubo Zhu, Zhehan Kan, Jingyi Yang, Miaolin Chen, Jinbo Xing, Kai Zhu, Zijian Wang, Sheng Zhong, Wei Tong
TL;DR
Unified multimodal models raise whether visual generation can improve visual understanding, but prior evaluations confound difficulty, paradigms, and generation–understanding interactions. VGAU-Diag addresses these issues with difficulty-stratified tasks, unified paradigm comparisons, and oracle-assisted protocols, finding that visual aids help easier instances more reliably while current UMMs often cannot use even faithful aids.
Problem
Prior evaluations provide mixed evidence because they confound task difficulty, reasoning paradigms, and the closed-loop interaction between visual generation and understanding.
Method
VGAU-Diag stratifies visual-planning samples by difficulty, evaluates multiple reasoning paradigms under one setting, and uses oracle-assisted protocols to diagnose generation–understanding bottlenecks.
Results
Generated visual aids help more reliably on easier instances but become unstable as reasoning complexity increases, while current UMMs often struggle to leverage faithful aids.
Takeaways & Limitations
Effective visual generation should target visual-understanding bottlenecks rather than add more visual reasoning steps, with targeted GtA more reliable than generic step-wise reasoning after correcting prior leakage.
Takeaways & Limitations
The evaluation is limited to static two-dimensional, procedurally generated visual-planning environments and may underrepresent natural-image diversity, ambiguity, and temporal or embodied interaction.
Abstract
from arXiv · showhide
Unified multimodal models (UMMs) can perform both understanding and generation, raising a central question: can visual generation improve understanding? Existing evaluations provide mixed evidence, but confound task difficulty, reasoning paradigms, and the closed-loop interaction between generation and understanding. We introduce VGAU-Diag, a fine-grained evaluation framework for vision generation-assisted understanding. It stratifies samples by difficulty, enables unified evaluation of multiple reasoning paradigms, and uses Oracle-Assisted Reference Protocols. Our analysis shows that generated visual aids help on easier instances but become unreliable as reasoning complexity increases. Oracle-assisted diagnosis further reveals that the main bottleneck often lies on the visual-understanding side rather than the visual-generation side, as current UMMs struggle to leverage even faithful visual aids. We also show that effective visual generation should target visual-understanding bottlenecks rather than add more reasoning steps, and identify a three-stage transition from task-irrelevant noise, to misleading plausible guidance, and finally to useful assistance. These findings would be useful to guide the development of better UMMs.
1 Introduction
The paper asks whether visual generation can improve visual understanding in unified multimodal models and introduces a framework to diagnose when and why it helps. It finds that assistance is easier to obtain on simple instances, while harder reasoning exposes visual-understanding bottlenecks and a transition from noise to misleading guidance to useful assistance.
- Motivation and framework: Existing evaluations confound sample difficulty, reasoning paradigm, and the entangled contributions of visual generation and visual understanding.These confounds make it difficult to determine whether gains reflect genuine capability improvement or failures in generation versus understanding.
- Motivation and framework: VGAU-Diag evaluates visual generation-assisted understanding by stratifying difficulty, comparing three reasoning paradigms, and using oracle-assisted diagnosis.The framework supports VCoT, GtA, and VR under a common setting and decouples visual generation from visual understanding.
- Key findings: Visual generation yields observable gains on easier instances, but its benefits become much less stable as reasoning complexity increases.Current unified models can exploit generated aids for relatively simple visual states but struggle to convert them into reliable gains on complex states.
- Key findings: Oracle-assisted analysis identifies visual understanding as the primary bottleneck because current UMMs often fail to leverage faithful visual aids.This contrasts with the view that unreliable visual generation is usually the dominant bottleneck.
- Key findings: Generation-assisted understanding progresses from task-irrelevant interference, through misleading plausible guidance, to stable assistance when both capabilities reach sufficient levels.The benefit does not increase monotonically with stronger visual generation.
- Key findings: Targeted GtA is more reliable than generic step-wise visual reasoning after correcting optimal-step prior leakage in Self-VCoT.The result supports generating task-relevant visual intermediates to address visual-understanding bottlenecks rather than simply adding reasoning steps.
2 Related Work
Prior work uses visual generation to support visual understanding during training through visual supervision and during inference through external visual aids.
- Visual Generation for Visual Understanding: Visual generation can improve visual understanding during training through visual supervision that enhances visual perception.
- Visual Generation for Visual Understanding: During inference, external visual aids can assist visual grounding in vision-language models.
- Unified Multimodal Models: These approaches motivate studying visual generation-assisted understanding in unified multimodal models that both understand and generate visuals.
3 Evaluation Framework
The evaluation framework tests visual generation-assisted understanding with controlled tasks, multiple paradigms, and oracle references. It uses executable visual-planning solutions and solver-based evaluation to separate generation, understanding, and paradigm effects.
- Task Design: The framework evaluates UMMs across six verifiable visual-planning task families and three difficulty levels.The task families are Maze, Sokoban, Path, Onet, Parking, and Klotski.
- Task Design: Models generate complete executable solutions rather than selecting multiple-choice options, with 100 instances per task family split into easy, medium, and hard levels.This design supports more reliable task-level evaluation than predefined-option selection.
- Evaluation Protocols: Closed-loop protocols evaluate Self-GtA, Self-VCoT, and Self-VR, covering auxiliary visual generation, step-wise visual interaction, and direct visual solution encoding.Self-GtA adds grids and coordinates to clarify spatial layouts, while Self-VCoT generates successive visual states and Self-VR directly encodes predicted solutions visually.
- Evaluation Protocols: Oracle-assisted protocols replace model-generated visuals with externally constructed aids to diagnose whether failures arise from generation fidelity, visual understanding, or paradigm effectiveness.The protocols include Text-CoT, Oracle-VCoT, and Oracle-GtA, with controlled comparisons of visual-aid designs.
- Evaluation Metrics: Solver-based task-level evaluation reports optimal and feasible solution accuracy instead of relying only on exact-match or step-wise accuracy.This avoids rejecting valid alternative solutions and avoids rewarding action overlap without task success.
4 Experiment
The experiments show that visual generation helps UMMs mainly on easier instances, while harder reasoning exposes visual-understanding bottlenecks and unstable generation-assisted behavior. Oracle comparisons, fidelity diagnostics, and corrected protocols indicate that targeted, faithful visual aids are more useful than additional reasoning steps.
- Main results: Difficulty-aware evaluation reveals observable gains on easy instances, but visual-generation benefits become unstable or disappear as reasoning complexity increases.The evaluation stratifies samples into easy, medium, and hard levels using optimal solution length.
- Main results: Oracle-GtA results indicate that visual understanding, rather than visual generation, is the primary bottleneck for most open-source UMMs.Across 48 task-level comparisons from 8 open-source UMMs and 6 tasks, only 12 cases fall below the Oracle-GtA reference.
- Self-generation analysis: The controlled diagnosis separates understanding scale from generation fidelity and finds that stronger understanding generally improves performance, whereas fidelity improvements are non-monotonic.Performance can drop when task-irrelevant noise is replaced by plausible but incorrect aids, then improve substantially with high-fidelity task-aligned visuals.
- Self-generation analysis: Generation-assisted understanding transitions from interference, through misleading plausible guidance, to assistance as visual aids progress from irrelevant noise to incorrect and then faithful task-aligned content.Incorrect aids can encode a false task state, while faithful aids provide task-relevant information and support stable gains.
- Protocol correction: Self-VCoT can appear effective with task-irrelevant images because its protocol uses the answer-derived optimal step count, making the task easier.The protocol forces the generated answer to match the reference solution length, so the measured gain need not come from generated visual states.
- Protocol correction: After removing the optimal-step prior, Showo2’s Self-VCoT accuracy on Parking drops from 56.6% to 0.0%, while LLaDA2.0-Uni’s drops from 40.0% to 0.0%.Self-GtA remains more reliable because it uses a single auxiliary visual aid to target the visual-understanding bottleneck.
5 Conclusion
VGAU-Diag evaluates whether visual generation improves visual understanding in unified multimodal models and finds that assistance is more reliable on easier instances than on complex reasoning tasks.
- VGAU-Diag stratifies samples by difficulty, unifies multiple generation-assisted reasoning paradigms, and uses oracle-assisted protocols.The framework evaluates how visual generation interacts with understanding rather than treating the process as an undifferentiated closed loop.
- Generated visual aids help more reliably on easier instances but become unstable as reasoning complexity increases.The reported pattern indicates that current models can exploit visual aids in simpler settings but struggle to use them reliably for harder reasoning.
- Oracle-assisted diagnosis identifies visual understanding, rather than visual generation, as the main bottleneck in many cases.Current unified models struggle to use even faithful visual aids.
- Effective visual generation should target visual-understanding bottlenecks rather than add more reasoning steps.The paper also identifies a transition from task-irrelevant interference, through misleading plausible guidance, to effective assistance.
6 Limitations
The framework is limited to static two-dimensional planning environments and procedurally generated, structured visual states, which do not fully represent dynamic, embodied, three-dimensional, or natural-image settings.
- The evaluation does not fully capture temporal dynamics, embodied interaction, or continuous three-dimensional scenes.The controlled tasks focus on spatial state parsing, planning horizons, and intermediate visual reasoning.
- Procedurally generated tasks and structured visual states may reduce visual diversity and underrepresent noise, ambiguity, and appearance variation in natural images.The design supports controllability, unique solutions, and reliable automatic evaluation.
- Future work can extend the evaluation to video-based planning, interactive environments, embodied decision-making, and more realistic visual inputs.The proposed extensions retain the goal of preserving verifiable solutions.
A.1.1 Maze
The maze task asks models to navigate a 10 × 10 grid from a blue start marker to a green goal using legal four-directional moves, with optional state-image generation after each move.
- Task Definition: The model must move the agent from the blue start marker to the green goal marker in a 10 × 10 maze.Black cells are walls, off-white cells are walkable, and the goal cell is passable.
- Task Definition: The only legal actions are up, down, left, and right, each moving the agent by exactly one grid cell.The final action sequence is returned as a JSON array wrapped in <ANSWER_JSON>...</ANSWER_JSON>.
- Task Definition: In the step-by-step setting, the model generates one state image after each move while preserving the maze layout and moving only the blue agent marker.
- Construction: The dataset contains easy mazes with path lengths 1–6, medium mazes with lengths 7–17, and hard mazes with lengths 18–30.Samples are verified using BFS and retained when their unique shortest path falls into the desired difficulty bucket.
- Evaluation: Evaluation replays the predicted moves on the ground-truth walkable-cell set and reports legal goal reaching with any length or with BFS-optimal length.Invalid predictions include unknown actions, leaving the grid, or entering a wall.
A.1.2 Sokoban
The Sokoban task asks models to solve a 10 × 10 single-box puzzle by moving the player and pushing the box onto a green target, with legality determined by walls and push constraints.
- Task Definition: The model controls a player to push one wooden box onto a green target cell in a 10 × 10 Sokoban puzzle.Brick cells are walls, sand cells are walkable, and the image marks the player, box, and target.
- Task Definition: The only legal actions are up, down, left, and right, and pushing is legal only when the box can move one cell in that direction.
- Construction: The dataset procedurally generates 100 puzzles with exactly one player, one movable box, and one target, then uses a programmatic solver to obtain shortest valid solutions.Unsolvable puzzles and puzzles outside the intended solution-length range are excluded.
- Evaluation: Evaluation replays predicted actions from the annotated player and box positions and rejects wall collisions or illegal pushes.It reports reach_any for legal solutions of any length and reach_optimal when the predicted length matches the annotated optimum.
A.1.3 Path
The Path task asks models to identify labeled points on a valid simple route through a 10 × 10 map. Instances are procedurally generated and evaluated by exact label-sequence matching across difficulty levels.
- Task Definition: Models identify which labeled points lie on the valid simple path from the orange start to the blue destination.Movement is orthogonal, walls are impassable, and the path cannot revisit or backtrack.
- Construction: 100 procedurally generated instances use fixed visual styling, sampled valid paths, and five labeled walkable points.The construction records which labels lie on each sampled path.
- Evaluation: Predictions are correct only when the extracted label sequence exactly matches the ground truth in both content and order.Outputs are parsed from <answer>...</answer> and normalized to uppercase letters A–E.
- Evaluation: Accuracy is reported separately for easy, medium, and hard subsets, as well as for the full dataset.
A.1.4 Parking
The benchmark defines several visual puzzle tasks and compares text-based reasoning with generated annotated visual aids. It uses procedurally constructed, solvable instances with replay-based validity checks and difficulty analyses grounded in solution structure.
- A.1.4 Parking: Parking requires sliding oriented vehicles on an 8 × 8 board until the horizontal red car reaches the right-boundary exit.Vehicles cannot overlap or leave the board, and step-by-step settings additionally generate an updated state image after each move.
- Construction: 100 parking-exit puzzles are generated with controlled difficulty, solvability filtering, and BFS-computed shortest solutions.Instances vary in vehicle count, orientations, and lengths, and are discarded when unsolvable or outside the target step range.
- Evaluation: Predicted parking moves are replayed on the annotated board, with reach_solved requiring a legal trajectory that reaches the exit and reach_optimal additionally requiring the annotated shortest length.Invalid moves include unknown vehicles, orientation violations, overlaps, and leaving the board.
- Task Definitions: Onet, Klotski, and related puzzles require complete legal action sequences under board-specific movement, matching, or block-sliding constraints.Onet removes connectable identical fruit pairs, while Klotski moves a 2 × 2 target block toward its goal.
- Difficulty Analysis: Maze difficulty splits are supported by structural statistics whose reported metrics increase consistently across difficulty levels.The analysis includes walkable cells, branch nodes, corridor nodes, and ground-truth solution-path statistics.
- Reasoning Paradigms: Text-CoT prompts models to reason about puzzle states and planned moves, whereas visual assistance prompts them to generate annotated images before producing solutions.The visual prompts add red grid boundaries and coordinate labels while preserving task-relevant layouts and markers across Maze, Sokoban, Path, and Parking.