Source-linked AI summary
VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models
Weiye Xu, Jiahao Wang, Weiyun Wang, Zhe Chen, Wengang Zhou, Aijun Yang, Lewei Lu, Houqiang Li, Xiaohua Wang, Xizhou Zhu, Wenhai Wang, Jifeng Dai, Jinguo Zhu
TL;DR
Existing multimodal reasoning evaluations can conflate visual reasoning with text-based reasoning, motivating benchmarks that isolate vision-centric capabilities. The paper introduces VisuLogic, evaluates language-only and multimodal models, and finds that leading models remain far below human performance; it also identifies RL as a promising improvement direction.
Problem
Existing multimodal benchmarks may assess language-driven reasoning or basic perception rather than core vision-centric reasoning, leaving accurate evaluation important.
Method
VisuLogic provides 1,000 vision-centric reasoning tasks across six categories, curated through expert annotation and human review, with language-only models evaluated using generated image descriptions.
Results
Most evaluated MLLMs scored below 30%, while humans reached 51.4% and text-only models supplied with descriptions barely exceeded the 24.9% random baseline.
Takeaways & Limitations
VisuLogic exposes substantial gaps in visual logical reasoning, while reinforcement learning is identified as a promising approach for improving MLLM visual reasoning.
Takeaways & Limitations
The discussed reasoning approaches are based only on pure-text corpora, motivating chain-of-thought techniques tailored to multimodal data.
Abstract
from arXiv · showhide
Visual reasoning is a core component of human intelligence and a critical capability for advanced multimodal models. Yet current reasoning evaluations of multimodal large language models (MLLMs) often rely on text descriptions and allow language-based reasoning shortcuts, failing to measure genuine vision-centric reasoning. To address this, we introduce VisuLogic: a benchmark of 1,000 human-verified problems across six categories (e.g., quantitative shifts, spatial relations, attribute comparisons). These various types of questions can be evaluated to assess the visual reasoning capabilities of MLLMs from multiple perspectives. We evaluate leading MLLMs on this benchmark and analyze their results to identify common failure modes. Most models score below 30% accuracy-only slightly above the 25% random baseline and far below the 51.4% achieved by humans-revealing significant gaps in visual reasoning. Furthermore, we provide a supplementary training dataset and a reinforcement-learning baseline to support further progress.
1 Introduction
Existing multimodal evaluations can conflate language-based inference with visual reasoning, motivating VisuLogic’s vision-centric benchmark. Its evaluation finds that leading MLLMs remain far below human performance, while reinforcement learning improves a baseline model.
- Motivation: Existing benchmarks often assess basic perception or diagram-based problem solving, allowing language-driven results to be mistaken for visual reasoning.This motivates evaluations that isolate vision-centric reasoning rather than text-based inference.
- Benchmark: VisuLogic spans six reasoning categories and is designed to make critical visual details difficult to replace with textual descriptions.Examples include quantitative shifts, symmetry, and rotation.
- Results: 26.3%, 28.1%, 28.0% and 27.7% accuracy were achieved by GPT-4o, Doubao-1.5-Vision-Pro, Gemini-2.0-Pro-Exp and InternVL3-78B, versus 51.4% for humans.The reported model results remain close to the 24.9% random-chance baseline.
- Results: 25.5% to 31.1%: reinforcement-learning fine-tuning improved the baseline model’s accuracy on VisuLogic.The resulting accuracy exceeded those of the compared open-source and closed-source counterparts.
- Contributions: The benchmark includes comprehensive evaluations and comparative studies of MLLMs under different settings.The authors also release training code and data to facilitate future research.
2 Related Work
Multimodal benchmarks have expanded from basic visual perception toward specialized domains, including text extraction and diagram-based reasoning. This progression motivates more targeted evaluation of model capabilities.
- Multimodal Benchmarks: Early multimodal benchmarks focused on visual question answering, image captioning, and referring expression comprehension.These tasks primarily covered foundational visual perception capabilities.
- Multimodal Benchmarks: Later benchmarks assess specialized domains such as textual content extraction through OCRBench, ChartQA, and DocVQA.The cited benchmarks broaden capability coverage beyond early perception tasks.
- Multimodal Benchmarks: Recent reasoning-oriented work uses mathematical problem-solving benchmarks with diagrams, including MathVista, MathVerse, and MathVision.The paper contrasts these approaches with its focus on visual reasoning.
3 VisuLogic
VisuLogic is curated through collection, quality control, and taxonomy stages into a 1,000-question benchmark spanning six visual-reasoning categories. A separate validated training set supports subsequent investigation and model improvement.
- Data Curation Pipeline: The data-curation pipeline comprises data collection, quality control, and detailed taxonomy, followed by construction statistics and an auxiliary training dataset.The training questions are analogous to benchmark questions and intended to support future research.
- Data Collection: Raw questions are fetched from licensed public online resources, cleaned of noise and markup, and structurally standardized.Playwright and custom parsing scripts are used during fetching and extraction.
- Quality Control: Reliability checks verify image existence and formatting, remove text- and image-level duplicates, and apply human review.Perceptual hashing identifies visually similar images during duplicate removal.
- Data Taxonomy: The taxonomy contains Quantitative, Spatial, Positional, Attribute, Stylistic, and Other categories based on expert annotation and human review.The categories cover changes in counts, 3D reconstruction, transformations, intrinsic shape properties, and related visual competencies.
- Benchmark Statistics: 1,000 single-choice questions are distributed across Quantitative Reasoning (35.3%), Spatial Reasoning (23.1%), Positional Reasoning (13.6%), Attribute Reasoning (8.2%), Stylistic Reasoning (9.0%), and Other (10.8%).Answer options are also reported as A (23.1%), B (26.7%), C (25.2%), and D (25.0%).
- Supplementary Training Dataset: 4,296 question–answer pairs form an auxiliary training set drawn from the same domains and validated to prevent benchmark overlap.Its category proportions are Quantitative Reasoning (30.7%), Spatial Reasoning (25.5%), Positional Reasoning (13.0%), Attribute Reasoning (8.8%), Stylistic Reasoning (9.9%), and Other (12.1%).
4 Experiments
VisuLogic evaluates language-only and multimodal models on visual reasoning tasks, revealing low accuracy, limited CoT gains, and stronger improvements from hints and reinforcement learning. Error analyses show that models struggle with spatial, stylistic, positional, and relational reasoning despite recognizing basic visual content.
- Experimental Setup: The evaluation covers human and random references, 28 language-only or multimodal models, and a caption-based protocol for language-only systems.Language-only models receive GPT-4o-generated image descriptions before answering VisuLogic questions.
- Overall Results: 28.0% was the best language-only accuracy, while textual reasoning remained insufficient for VisuLogic’s visual logical reasoning problems.Qwen2.5-72B-Instruct achieved 28.0%; GPT-4 and Deepseek-R1 achieved 23.6% and 26.6%.
- Overall Results: 28.1% was the highest multimodal score, leaving Doubao-1.5-Vision-Pro-32k 23.3 points below human performance.GPT-4o and Gemini-2.0-Pro reached 26.3% and 28.0%, respectively.
- Prompting Effects: CoT prompting produced minimal gains, with GPT-4o-mini improving by 1.2 points and all other models improving by less than 1.0 point.The paper speculates that current CoT training relies on pure-text corpora.
- Prompting Effects: Hint prompts raised three models by more than 8 points to above 35%, but models still failed to form coherent and reliable reasoning chains.The authors argue that future work must improve the reliability and correctness of multimodal reasoning procedures.
- Reinforcement Learning: 31.1% was achieved by InternVL2.5-38B-RL, exceeding its non-RL counterpart by 5.6% and establishing a new VisuLogic state-of-the-art.Qwen2.5-VL-7B-Instruct-RL reached 28.0%, a 2.0-point boost over its counterpart; qualitative analysis associates RL with stepwise reasoning and hypothesis revision.
- Error Analysis: Models struggled most with spatial, stylistic, and positional reasoning, while qualitative cases showed failures to infer evolving inter-element relations beyond static recognition.Stylistic error rates exceeded 75%, and text-only systems often omitted counts, shapes, and progression patterns needed for deduction.
5 Conclusion
VisuLogic is a 1,000-question benchmark spanning six visual reasoning categories, used to evaluate advanced LLMs and MLLMs. Results show substantial gaps relative to human performance, while reinforcement learning appears promising for improving MLLM visual reasoning.
- 5 Conclusion: VisuLogic contains 1,000 vision-centric reasoning tasks across six distinct categories.The paper also releases evaluation code, training scripts, and datasets.
- 5 Conclusion: Advanced LLMs and MLLMs evaluated on VisuLogic fall short of human performance.The authors describe these results as indicating substantial opportunities for advancement in visual logical reasoning.
- 5 Conclusion: Reinforcement learning is identified as a promising approach for enhancing MLLMs’ visual reasoning capabilities.
B.1 Statistical analysis
VisuLogic questions are typically short, while its images span broad width and height ranges with constrained overall scale. The appendix also provides additional category-specific examples.
- B.1 Statistical analysis: VisuLogic question text is mostly concentrated around 40 tokens.The length is calculated using Llama-3.1’s and InternVL2.5’s tokenizers.
- B.1 Statistical analysis: Image widths range from 200 to 700 pixels, averaging 592.3 pixels, while heights range from 90 to 825 pixels, averaging 327.9 pixels.Image sizes are limited to within the same order of magnitude.
- B.1 Statistical analysis: Additional examples cover questions from different VisuLogic categories.
C.1 Evaluation of LLMs
The evaluation compares language-model captioning and model-generated solutions for VisuLogic. GPT-4o struggles to describe benchmark details accurately, and most models fail on accurate solutions, sometimes producing correct answers with flawed methods.
- C.1 Evaluation of LLMs: GPT-4o-generated captions are used to support comparative LLM evaluation of VisuLogic images.The captions are prompted to describe fine-grained scenes, objects, relationships, and text without directly answering the question.
- C.1 Evaluation of LLMs: GPT-4o encounters difficulties accurately describing details in VisuLogic images across six categories.
- C.1 Evaluation of LLMs: For most questions, almost all evaluated models fail to provide accurate solutions.Some models produce the correct final answer while retaining methodological errors.
C.4 RL Experiments
The supplementary RL experiments compare supervised fine-tuning and reinforcement-learning settings using visual reasoning prompts, rule-based rewards, and multiple implementation configurations. The provided materials include prompt examples, category examples, reward definitions, and training hyperparameters.
- C.4 RL Experiments: Comparative SFT experiments use the same dataset as the RL experiments with direct-answer responses.The instruction combines questions with non-CoT prompts.
- C.4 RL Experiments: RLOO is used as the reinforcement-learning algorithm because it is critic-model-free, computationally inexpensive, and more robust to noise and KL constraints.
- C.4 RL Experiments: The appendix provides additional examples of benchmark questions across Quantitative, Spatial, Positional, Attribute, Stylistic, and Other categories.
- C.4 RL Experiments: Additional materials describe image-caption token-length distributions for LLM evaluation settings.
- C.4 RL Experiments: The reward system combines format rewards with accuracy rewards based on output structure and answer correctness.Format rules require thinking and final answers to appear within designated tags, after which the extracted answer is judged.
C.5 RL models Evaluation Details
The evaluation uses training-aligned prompts that require step-by-step reasoning and a structured answer format.
- Training-aligned prompts ask models to reason step by step before returning an answer in <think> and <answer> fields.These prompts are used during VisuLogic evaluation to investigate latent reasoning abilities after applying format rewards in RL experiments.
C.6 Effectiveness of RL Experiments
The RL experiments compare baseline and RL-optimized outputs across Qwen2.5-VL-7B and InternVL-2.5-38B. These comparisons indicate that RL training enables fundamental-level analysis of reasoning tasks embedded in graphical representations.
- RL training enables models to perform fundamental-level analysis of reasoning tasks embedded in graphical representations.The qualitative comparisons cover baseline and RL-optimized outputs.
- Figures 20–22 compare model outputs before and after RL training for Qwen2.5-VL-7B.
- Figures 23–25 compare model outputs before and after RL training for InternVL-2.5-38B.