Source-linked AI summary
Uni-MMMU: A Massive Multi-discipline Multimodal Unified Benchmark
Kai Zou, Ziqi Huang, Yuhao Dong, Shulin Tian, Dian Zheng, Hongbo Liu, Jingwen He, Bin Liu, Yu Qiao, Ziwei Liu
TL;DR
Existing benchmarks rarely test the logical interaction between visual understanding and generation. Uni-MMMU introduces eight bidirectionally coupled, reasoning-centric tasks with reproducible evaluation of intermediate and final textual and visual outputs. Evaluation reveals that synergy is strongest under strong logical dependencies, while generation remains a major bottleneck in unified models.
Problem
Existing evaluations often isolate understanding and generation or omit tasks that enforce their logical dependency, despite the importance of their interaction in multi-step reasoning.
Method
Uni-MMMU is an eight-task benchmark spanning two bidirectional paradigms, with deterministic ground-truth paths and reproducible dual-level scoring of textual and visual outputs.
Results
Synergy is strongest in tasks with strong logical dependencies, where intermediate modal information improves final accuracy, while unified models remain understanding-skewed with generation as a bottleneck.
Takeaways & Limitations
Uni-MMMU provides a fine-grained foundation for diagnosing when generation and understanding reinforce each other and where unified models fail.
Takeaways & Limitations
The benchmark uses static images and does not cover open-ended creativity, subjective judgment, nuanced commonsense reasoning, video, or longer-term temporal interactions.
Abstract
from arXiv · showhide
Unified multimodal models aim to jointly enable visual understanding and generation, yet current benchmarks rarely examine their true integration. Existing evaluations either treat the two abilities in isolation or overlook tasks that inherently couple them. To address this gap, we present Uni-MMMU, a comprehensive and discipline-aware benchmark that systematically unfolds the bidirectional synergy between generation and understanding across eight reasoning-centric domains, including science, coding, mathematics, and puzzles. Each task is bidirectionally coupled, demanding models to (i) leverage conceptual understanding to guide precise visual synthesis, or (ii) utilize generation as a cognitive scaffold for analytical reasoning. Uni-MMMU incorporates verifiable intermediate reasoning steps, unique ground truths, and a reproducible scoring protocol for both textual and visual outputs. Through extensive evaluation of state-of-the-art unified, generation-only, and understanding-only models, we reveal substantial performance disparities and cross-modal dependencies, offering new insights into when and how these abilities reinforce one another, and establishing a reliable foundation for advancing unified models.
1 Introduction
Uni-MMMU addresses the lack of benchmarks that enforce logical interaction between generation and understanding. It introduces bidirectionally coupled tasks, reproducible dual-level evaluation, and evidence that generation remains a bottleneck in unified models.
- Existing benchmarks often isolate generation and understanding or assess only superficial cross-modal properties, without enforcing logical dependency between them.This omission limits evaluation of the interaction required for multi-step problem-solving.
- Its deterministic evaluation protocol scores unique final answers and intermediate reasoning steps using programmatic parsers, perceptual metrics, and LLM-as-a-Judge.The resulting pipeline is automated, reproducible, objective, and interpretable.
- Intermediate modal information improves final accuracy on strongly dependent tasks, with oracle intermediates producing substantial gains over end-to-end approaches.Even imperfect model-generated intermediates can improve final accuracy.
- Current unified models are skewed toward understanding, while generation bottlenecks include imprecise image editing, schematic synthesis, and fine-grained spatial reasoning.
- Uni-MMMU introduces eight bidirectionally coupled tasks spanning rigorous disciplines, where generation aids understanding or understanding guides generation.The benchmark targets mutual reinforcement between the two capabilities.
2 Related Works
Prior unified-model research and benchmarks increasingly combine multimodal capabilities, but evaluation still often separates understanding from generation. Uni-MMMU responds by coupling both abilities with multi-turn tasks and dual assessment of processes and results.
- Unified-model architectures have progressed from combining modules to systematic multimodal integration, including mixed experts, native unification, reinforcement learning, and video generation.
- Multimodal benchmarks have expanded from isolated perception or semantic fidelity toward thinking-with-images and understanding-driven generation tasks.
- Existing unified benchmarks still largely assess understanding and generation in isolation, leaving their interaction insufficiently tested.
- Uni-MMMU combines multimodal understanding, generation, complex multi-turn tasks, and dual evaluation of intermediate processes and final results.This design supports fine-grained error attribution.
3 The Uni-MMMU
Uni-MMMU organizes eight disciplines into two complementary paradigms: generated visual states support reasoning, while conceptual understanding guides image synthesis. Its evaluation separately and jointly scores textual reasoning and visual outputs.
- The benchmark contains two paradigms: generation supports understanding in spatial reasoning tasks, while understanding guides generation in science and code rendering.Figure 1 groups Maze, Sliding, Geometry, and Jigsaw under the first paradigm, and Physics, Chemistry, Biology, and Code Rendering under the second.
- Generation aids Understanding: Generation-aids-understanding tasks use visual outputs as iterative working memory, planning canvases, and validation checkpoints for multi-step reasoning.Maze, Sliding Puzzle, Geometry, and Jigsaw require models to generate or update visual states before reaching textual decisions or solutions.
- Understanding aids Generation: Understanding-aids-generation tasks test whether models can translate physical, chemical, biological, or programmatic concepts into faithful visual outcomes.Science tasks use deterministic observable changes, while Code Rendering maps SVG specifications to descriptions and rendered images.
- Evaluation: The benchmark uses deterministic parsers, perceptual similarity metrics, and model-based judging to evaluate textual and visual outputs across task-specific protocols.Its shared philosophy scores intermediate visual steps separately from final textual outcomes.
- Evaluation: Text and image channels are scored separately and jointly, preventing strong rendering from compensating for poor reasoning or vice versa.This design isolates failures in perception, generation, reasoning, and their integration.
4 Experiments
Uni-MMMU evaluates unified and specialized models on coupled generation-understanding tasks, finding cross-modal dependencies, evaluator agreement, and recurring failures in visual reasoning and instruction adherence.
- 4.2 Uni-MMMU Evaluation Results: GPT achieves the highest Jigsaw image-generation and textual-understanding scores, while stronger generation by GPT and nano-banana supports better Maze and Sliding path planning.Bagel performs lowest in both Jigsaw categories, whereas Bagel leads the open-source models on Geometry’s final score.
- 4.2 Uni-MMMU Evaluation Results: Understanding ability correlates with generation quality on Science, while Code shows that strong understanding alone does not ensure accurate graphical output.GPT leads Science generation, FLUX-Kontext has low image accuracy, and Bagel’s strong understanding accompanies low Code image accuracy.
- 4.1 Experiment Setup: The LLM-based evaluator reaches Cohen’s Kappa values of 0.6 to 0.8 against human annotators and Gemini-2.5-pro, indicating substantial agreement.Textual evaluation consistency is reported as even higher.
- 4.2 Uni-MMMU Evaluation Results: Intermediate generative states improve reasoning most when tasks impose strict logical dependencies, with oracle intermediates providing consistent, sizeable gains across tasks.The strongest effects occur for Maze solution accuracy and Science image correctness.
- 4.4 Insights and Analysis: Qualitative failures include topology and semantic violations, background or style drift, fragile spatial perception, and instruction-following errors that corrupt downstream reasoning.Examples include distorted Maze wall-path topology, copied Jigsaw references, and SVG semantic misreadings or rasterization errors in Code Rendering.
5 Conclusion
Uni-MMMU is a comprehensive benchmark for bidirectionally coupled understanding and generation across eight reasoning-centric tasks. It evaluates intermediate processes and final outcomes, revealing substantial room for improvement in instruction adherence and spatial reasoning while supporting reproducible analysis.
- 5 Conclusion: Uni-MMMU evaluates eight reasoning-centric tasks where generation aids understanding or understanding aids generation.
- 5 Conclusion: Its evaluation pipeline scores intermediate processes and final outcomes for fine-grained analysis of unified-model behavior.
- 5 Conclusion: Evaluation of state-of-the-art models reveals substantial room for improvement, especially in precise instruction adherence and spatial reasoning.
- 5 Conclusion: The authors release code, datasets, evaluation tools, and judge configurations to support transparency and reproducibility.
7 Limitations
Uni-MMMU’s limitations concern task scope, data realism, and evaluator reliability. The benchmark emphasizes deterministic reasoning tasks and static images, uses procedurally generated or LLM-curated data, and relies partly on model-based judges.
- Uni-MMMU focuses on reasoning-centric disciplines with deterministic, verifiable solutions, limiting coverage of open-ended creativity, subjective judgment, and nuanced commonsense reasoning.
- The benchmark uses static images, leaving video and longer-term temporal interactions unevaluated.
- Procedural generation ensures unique solutions and objective parsing but may reduce the complexity, noise, and visual diversity of real-world imagery.
- The science curation pipeline combines LLM generation with manual review, which may introduce subtle biases from models or human curators.
- Several tasks rely on VLM judges whose biases or knowledge gaps may affect evaluation accuracy despite substantial agreement with human annotators.
A.1.1 Maze Navigation
Maze Navigation requires models to parse a 6 × 6 maze, produce the unique shortest path through alternating text and image outputs, and undergo automated visual and textual scoring.
- Maze Navigation gives a 6 × 6 maze image and requires the unique shortest path as sequential state images followed by the complete textual move sequence.
- The inference loop generates the next textual move, uses it to prompt the corresponding state image, and repeats until the final action sequence.
- The parser isolates the maze, discretizes it into a 6 × 6 grid, and converts the result into a character-based matrix for evaluation.
- Image evaluation includes whole-sample and step-level accuracy, while text evaluation measures exact sequence accuracy and position-wise move accuracy.
A.1.2 Sliding Puzzle
Sliding Puzzle tests optimal visual state-space planning by requiring models to solve a 3 × 3 8-puzzle through intermediate generated states and a final move list.
- The task assesses visual state parsing, optimal state-space search, and execution through sequential generative actions.
- Each instance is procedurally generated from the solved state, and BFS verifies solvability and uniqueness of the optimal path.
- Models receive multimodal few-shot demonstrations containing states, intermediate solution images, and the final JSON answer, with inference adapted to model architecture.
A.1.3 Geometry
Geometry couples visual generation with mathematical reasoning: models draw instructed auxiliary lines, then use the generated diagram to produce a calculation or proof.
- Geometry requires models to generate specified auxiliary lines on a problem figure and then solve the problem using the generated diagram as a visual aid.
- The dataset contains 140 Geo-Laux problems with figures, English construction instructions, ground-truth auxiliary-line images, and step-by-step solutions.
- The benchmark includes both calculation and proof questions, testing whether generated visual constructs scaffold complex logical deduction.
- A VLM judge scores auxiliary-line correctness, while an LLM judge requires both rigorous reasoning and a correct final conclusion or numerical result.
A.1.4 Jigsaw
The benchmark couples visual generation with understanding across puzzles, science, and code by requiring models to generate intermediate or final visuals and reason over them. These tasks use structured outputs and task-specific metrics to evaluate both fidelity and analytical correctness.
- Jigsaw: Jigsaw requires generating two candidate-completed panels before selecting and explaining which patch correctly completes the image.The task evaluates generation and understanding sequentially, with image similarity and candidate-selection accuracy as separate measures.
- Natural sciences: Science tasks require explaining a physical outcome from an initial image and condition, then generating an image that depicts the predicted final state.Evaluation separately measures scientific reasoning, final-state plausibility, and image correctness.
- Code Rendering: Code Rendering requires parsing SVG source, summarizing its intended scene, and rendering a faithful image without an external interpreter.The corpus spans simple primitives, multiple shapes, and complex overlapping or control-flow-based graphics.
- Code Rendering: SVG evaluation measures textual semantic consistency, shape and color correctness, and spatial layout accuracy.The image rubric covers object types, polygon side counts, colors, positions, alignment, spacing, layering, and rotation.
C Quantitative Failure-Mode Analysis
Per-task statistics show that unified models most often fail through visual-generation errors, including invalid outputs, reference copying, semantic misreading, and instruction-following lapses.
- Jigsaw: 100% of OmniGen2 jigsaw cases copied the 2×2 reference panel, while nano-banana produced irrelevant images in 41.6% of cases.
- Code Rendering: Qwen-Image-Edit rasterized the text-only Render Summary in 55% of cases, compared with 100% for Ovis-U1.
- Maze: Invalid maze images occurred in 15% of nano-banana outputs, 84% of GPT outputs, and 100% of Bagel outputs.
- Sliding Puzzle & Geometry: Sliding Puzzle and Geometry failures primarily involved background or style drift and instruction-following lapses that accounted for most score degradation.
- Visual-generation fidelity is identified as the primary bottleneck across the analyzed failure modes.