Source-linked AI summary
ViGoR-Bench: How Far Are Visual Generative Models From Zero-Shot Visual Reasoners?
Haonan Han, Jiancheng Huang, Xiaopeng Sun, Junyan He, Rui Yang, Jie Hu, Xiaojiang Peng, Lin Ma, Xiaoming Wei, Xiu Li
TL;DR
Generative models often achieve high visual fidelity while failing physical, causal, and complex spatial reasoning, and existing evaluations overlook these deficits. ViGoR-Bench unifies cross-modal coverage with process-and-result scoring, evidence-grounded judging, and fine-grained diagnostics; experiments on over 20 models reveal substantial reasoning weaknesses.
Problem
Current fidelity metrics and fragmented benchmarks do not adequately evaluate physical, causal, and complex spatial reasoning or the generative process.
Method
ViGoR-Bench evaluates Image-to-Image, Sequential Image-to-Image, and Image-to-Video generation through dual-track process-and-result scoring, evidence-grounded judging, and cognitive diagnostics.
Results
Experiments on over 20 leading models show significant reasoning deficits, including video models whose Process Visual Quality can be high while Result Reasoning Success remains low.
Takeaways & Limitations
ViGoR-Bench serves as a critical stress test for visual reasoning and shifts evaluation toward genuine reasoning capabilities rather than visual fidelity alone.
Abstract
from arXiv · showhide
Beneath the stunning visual fidelity of modern AIGC models lies a "logical desert", where systems fail tasks that require physical, causal, or complex spatial reasoning. Current evaluations largely rely on superficial metrics or fragmented benchmarks, creating a ``performance mirage'' that overlooks the generative process. To address this, we introduce ViGoR Vision-G}nerative Reasoning-centric Benchmark), a unified framework designed to dismantle this mirage. ViGoR distinguishes itself through four key innovations: 1) holistic cross-modal coverage bridging Image-to-Image and Video tasks; 2) a dual-track mechanism evaluating both intermediate processes and final results; 3) an evidence-grounded automated judge ensuring high human alignment; and 4) granular diagnostic analysis that decomposes performance into fine-grained cognitive dimensions. Experiments on over 20 leading models reveal that even state-of-the-art systems harbor significant reasoning deficits, establishing ViGoR as a critical ``stress test'' for the next generation of intelligent vision models. The demo have been available at https://vincenthancoder.github.io/ViGoR-Bench/
1. Introduction
Modern generative models can produce visually sophisticated content while failing physical, causal, and complex spatial reasoning that conventional fidelity metrics overlook. ViGoR-Bench addresses this evaluation gap with unified cross-modal, process-and-result, evidence-grounded, and diagnostic assessment, revealing substantial reasoning deficits in leading models.
- Visual fidelity can coexist with failures on physical and causal reasoning, which CLIPScore and FID do not reliably distinguish.These metrics prioritize semantic alignment and statistical fidelity over structural integrity.
- Existing reasoning benchmarks cover knowledge, physical causality, and temporal logic, but the evaluation landscape remains fragmented.
- ViGoR-Bench bridges Image-to-Image, Sequential Image-to-Image, and Image-to-Video tasks across 20 reasoning dimensions.
- Its Process plus Result pipeline evaluates intermediate states and final outputs for physical-law and causal consistency.
- An evidence-grounded multi-agent judge targets human-aligned evaluation, while granular diagnostics identify specific reasoning gaps beyond aggregate scores.
- Experiments on over 20 leading models show that even visually strong systems retain significant world-knowledge reasoning deficits.
- Proprietary models lead open-source counterparts, while explicit Chain-of-Thought improves interpretability without guaranteeing final-accuracy gains.
- Reward-driven reinforcement learning shows greater potential than supervised fine-tuning for visual reasoning, and challenging out-of-distribution training improves simpler in-distribution generalization.
2. Related Work
Related work has progressed from visual synthesis and editing toward unified multimodal and video generation, while evaluation is shifting from perceptual quality to cognitive reasoning. Existing approaches span spatial control, attention manipulation, multimodal architectures, and reasoning-focused benchmarks, but coverage remains specialized across domains and modalities.
- Text-to-image and editing systems use spatial control, attention manipulation, and instruction-following paradigms to guide generation.
- Unified vision models combine understanding and generation, supporting multimodal reasoning through sequence-to-sequence, early-fusion, and unified-tokenization architectures.
- Video generation models extend synthesis to temporal sequences requiring reasoning about dynamics, physics, and causality.
- Traditional generative benchmarks emphasize perceptual quality or compositional aspects through metrics and task-specific evaluations.
- Reasoning-centric benchmarks increasingly test multidisciplinary knowledge, logical consistency, and temporal logic in image and video domains.
3. ViGoR-Bench
ViGoR-Bench combines diverse reasoning-focused data construction with dual-track evaluation of intermediate processes and final generative outputs. Its VLM-as-a-Judge pipeline uses ground-truth references to assess reasoning quality and output validity, with human-alignment analysis supporting its evaluation reliability.
- Benchmark Scope: ViGoR-Bench covers Physical, Knowledge, and Symbolic Reasoning through tasks spanning embodied scenarios, world knowledge, and precise logical manipulation.Its task suite includes 20 distinct subtasks organized across the three domains.
- Data Construction: The data engine combines generative synthesis, real-world acquisition, and algorithmic construction, with human review or verification used to establish reliable references.The construction strategies are tailored to each reasoning domain and produce image-ground-truth pairs or validated textual answers where appropriate.
- Evaluation Protocol: The dual-track protocol evaluates both the logical coherence of intermediate reasoning sequences and the validity of final outputs using Gemini-2.5-Pro as a VLM-as-a-Judge.Process metrics score dynamic outputs on a continuous 0–100 scale, whereas result metrics provide binary pass/fail assessments of final solutions.
- Evaluation Protocol: Process evaluation scores background consistency, rule obedience, visual quality, and reasoning alignment across intermediate frames or video sequences.The evaluator receives the input image, editing prompt, generated sequence, visual and textual ground truths, and a process-specific scoring template.
- Evaluation Protocol: Result evaluation checks background consistency, rule obedience, visual quality, and reasoning success for the final generated image or final reasoning frame.These metrics target whether the final output preserves irrelevant structure, follows instructions, remains plausible, and completes the task.
- Reliability Analysis: With ground-truth references, Gemini-2.5-Pro reaches 73.3% Process accuracy and 78.6% Result accuracy, while its variance is comparable to or below human inter-annotator variance.Ground-truth references substantially reduce mean absolute error and variance, supporting stable large-scale automated evaluation.
4. Experiment
Experiments evaluate open-source and proprietary models across process and final-result metrics, revealing gaps between visual coherence, reasoning, and execution. Post-training and problem complexity analyses further show that reinforcement learning and harder training examples can improve reasoning performance.
- Experimental Setup: Models are evaluated in four groups using official checkpoints and default inference settings for zero-shot comparison.
- Main Results: Proprietary models lead open-source counterparts, with Nano Banana Pro strongest across most metrics and only top-tier models succeeding reliably in complex physical and symbolic reasoning.
- Main Results: Explicit CoT improves intermediate-process interpretability and logical-chain completeness but does not guarantee higher final accuracy.
- Problem Complexity: Maze Navigation and Jigsaw Puzzle reasoning success declines monotonically with grid size, whereas Sudoku follows an inverted-U pattern.The paper hypothesizes that training-data distribution biases explain Sudoku’s intermediate-dimension peak.
- Post-training: Qwen-Image-Edit-2511-RL trained on 8 × 8 mazes reaches 97.0% Reasoning Success and a 99.0 Average Score, outperforming the state-of-the-art proprietary model.The setup uses SFT followed by GRPO-based RL on synthetic maze datasets.
- Post-training: Training on higher-complexity 8 × 8 mazes yields the best overall generalization to simpler benchmark tasks.The paper attributes this to learning underlying reasoning rules rather than surface patterns.
5. Conclusion
ViGoR-Bench combines comprehensive coverage with dual-track evaluation to assess visual generation reasoning and quantify current model limitations. Results particularly expose degradation on high-complexity puzzle tasks.
- ViGoR-Bench couples comprehensive visual-generation coverage with rigorous dual-track evaluation and validates its automated pipeline against human experts.
- Experiments quantify state-of-the-art limitations, particularly performance degradation on high-complexity puzzle tasks.
Impact Statement
ViGoR-Bench aims to shift generative AI evaluation from visual fidelity toward genuine reasoning capabilities. The intended impact is more reliable, safer, and transparent AI development.
- ViGoR-Bench shifts attention from visual fidelity toward genuine reasoning capabilities and helps researchers identify and address logical deficits.
- Better understanding of physical laws and causal reasoning could reduce nonsensical or harmful content in critical applications.The paper names education, scientific simulation, and engineering design as examples.
A. Dataset Statistics and Qualitative Examples
ViGoR-Bench contains 918 samples spanning symbolic, knowledge, and physical reasoning, with examples that pair instructions and intermediate visual states. These examples cover multi-step tasks such as Sudoku, navigation, and mathematical function construction.
- Dataset Statistics: The benchmark contains 918 samples across Symbolic Reasoning, Knowledge Reasoning, and Physical Reasoning categories.
- Qualitative Examples: Each example pairs an instruction with intermediate visual states, emphasizing process-aware reasoning rather than final outcomes alone.
- Qualitative Examples: Representative tasks include Sudoku completion, path planning and navigation, and mathematical function construction.
- Qualitative Examples: Post-RL visual trajectories show improved solving capability over SFT, particularly in high-dimensional mazes.
B. Capability Profiling.
ViGoR profiling shows a consistent gap between strong visual-generation quality and weaker rule compliance and reasoning outcomes across symbolic, physical, and knowledge tasks. The gap is especially pronounced for multi-step, constraint-heavy, puzzle-oriented, and factually grounded reasoning.
- Symbolic reasoning: Symbolic reasoning is strongest on Algebraic Calculation and Block Building, but weaker on Jigsaw Puzzle, Function Plotting, and Maze Navigation.The lower-performing tasks require multi-step symbolic manipulation and structural reasoning.
- Symbolic reasoning: High Background Consistency and Visual Quality do not reliably translate into correct symbolic reasoning outcomes.Rule Obey drops on tasks with strict symbolic constraints, while Reasoning Accuracy and Reasoning Success degrade on puzzle-oriented tasks.
- Physical reasoning: Physical reasoning retains strong Visual Quality and Background Consistency but shows pronounced gaps in Rule Obey and Reasoning Accuracy.The largest gaps occur in Measurement & Verification, Object Assembly, and Situational Decision Making.
- Physical reasoning: Physical-task results likewise show that visually plausible intermediate states do not necessarily produce correct reasoning outcomes.The profiling identifies precise rule compliance and reasoning reliability as improvement areas.
- Knowledge reasoning: Knowledge reasoning preserves visual structure and semantic context well, with strong Background Consistency and Visual Quality across most domains.The profiling covers Biology, Physics, Chemistry, Geography, History, Sports, and Common Sense.
- Knowledge reasoning: Knowledge tasks show lower and more variable Rule Obey and degraded result-level reasoning, especially in History, Geography, and Sports.These domains require precise factual grounding and temporal or causal reasoning, while Reasoning Accuracy and Reasoning Success remain limited across multiple sub-tasks.
C. Evaluation Templates
ViGoR-Bench uses standardized templates to evaluate knowledge, physical, and symbolic reasoning across binary and chain-of-thought settings. Its profiling figures organize model comparisons by reasoning domain, sub-task, and process-versus-result metrics.
- Evaluation protocol: Evaluation templates standardize task context, input image ordering, reference information, and explicit criteria across reasoning types.The protocol is intended to support consistent and reproducible assessment across domains.
- Benchmark materials: ViGoR-Bench includes samples, benchmark statistics, and qualitative comparisons of Qwen-Image-Edit-2511 after SFT and RL.These materials are presented in Figures 7–9.
- Capability profiling: Figures 10–12 compare five models across seven sub-tasks for symbolic, physical, and knowledge reasoning.Each figure reports four Process Metrics and four Result Metrics in its top and bottom rows.
- Knowledge reasoning templates: Knowledge reasoning is covered by binary and chain-of-thought templates.These are listed as Tables 5 and 6.
- Physical reasoning templates: Physical reasoning is covered by chain-of-thought and binary templates.These are listed as Tables 7 and 8.
- Symbolic reasoning templates: Symbolic reasoning templates cover Sudoku, Jigsaw Puzzle, Function Plotting, Algebraic Calculation, Block Building, Klotski Puzzle, and Maze Navigation.The tables provide both chain-of-thought and binary variants where listed, with Maze Navigation also represented in both formats.