Source-linked AI summary
LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts
Yijia Xiao, Edward Sun, Tianyu Liu, Wei Wang
TL;DR
MLLMs have made progress across visual and language tasks, but systematic evaluation of general visual logical reasoning remains limited. LogicVista addresses this gap with a 448-question benchmark spanning five reasoning categories and nine capabilities, using annotations for both multiple-choice and open-ended evaluation. Evaluation of eight MLLMs provides detailed performance breakdowns across reasoning skills and visual capabilities, while the benchmark remains focused on abstract rather than real-world blended scenarios.
Problem
General visual logical reasoning in MLLMs lacks comprehensive evaluation beyond mathematical reasoning, despite its relevance to problem-solving tasks.
Method
LogicVista combines 448 manually collected and annotated multiple-choice questions across five reasoning categories and nine capabilities with multiple-choice and open-ended evaluation.
Results
Eight MLLMs are evaluated across five logical reasoning categories, with detailed breakdowns by reasoning skill and capability rather than a single overall score.
Takeaways & Limitations
LogicVista provides a platform for examining the current state of logical reasoning in multimodal LLMs across diverse visual reasoning tasks.
Takeaways & Limitations
Future work should examine AI agents performing tasks that blend abstract reasoning with real-world scenarios.
Abstract
from arXiv · showhide
We propose LogicVista, an evaluation benchmark that assesses the integrated logical reasoning capabilities of multimodal large language models (MLLMs) in Visual contexts. Recent advancements in MLLMs have demonstrated various fascinating abilities, from crafting poetry based on an image to performing mathematical reasoning. However, there is still a lack of systematic evaluation of MLLMs' proficiency in logical reasoning tasks, which are essential for activities like navigation and puzzle-solving. Thus we evaluate general logical cognition abilities across 5 logical reasoning tasks encompassing 9 different capabilities, using a sample of 448 multiple-choice questions. Each question is annotated with the correct answer and the human-written reasoning behind the selection, enabling both open-ended and multiple-choice evaluation. A total of 8 MLLMs are comprehensively evaluated using LogicVista. Code and Data Available at https://github.com/Yijia-Xiao/LogicVista.
1 Introduction
LogicVista addresses the limited evaluation of integrated visual logical reasoning in MLLMs by covering diverse reasoning tasks and capabilities with structured annotations and multiple evaluation strategies.
- Existing MLLM benchmarks largely emphasize perception, recognition, text understanding, or mathematics, leaving general visual logical reasoning insufficiently evaluated.
- LogicVista covers five reasoning categories: inductive, deductive, numerical, spatial, and mechanical reasoning.The categories contain 107, 93, 95, 79, and 74 samples, respectively.
- The benchmark spans nine capabilities, including diagrams, OCR, patterns, graphs, tables, 3D shapes, puzzles, sequences, and physics.Capability sample counts range from 45 for 3D shapes to 330 for diagrams.
- Manual annotations provide correct choices, reasoning explanations, and support for both multiple-choice and open-ended evaluation.The LLM evaluator enables quantitative analysis, while annotated reasoning supports checking individual reasoning steps.
- Questions are presented without surrounding real-life context, requiring models to use the specified reasoning abilities rather than contextual cues.The benchmark includes examples such as inferring an object’s top view from another perspective.
- Eight open- and closed-source MLLMs are evaluated on 448 tasks across five reasoning categories with performance broken down by skill and capability.This breakdown provides more detail than a single overall score.
2 Related Works
Related work develops multimodal generalist systems and benchmarks, but existing evaluations often emphasize broad vision-language performance or reasoning in context-rich settings. LogicVista instead targets integrated vision-language reasoning with detailed skill-level analysis and leakage-conscious data collection.
- Vision-language research has expanded from multimodal understanding and generation to generalist systems that combine LLMs with visual and other sensory inputs.
- Existing benchmarks evaluate capabilities such as scene-text understanding, commonsense reasoning, mathematics, instruction following, and external knowledge incorporation.
- Some reasoning benchmarks use real-life contexts that may allow models to rely on contextual cues rather than direct reasoning.
- LogicVista focuses on integrated vision-language capabilities and aims to provide deeper insight than model rankings alone.
3 Data annotation and organization
LogicVista combines curated, access-controlled visual reasoning data with five reasoning skills and nine multimodal capabilities. Its annotations support both multiple-choice scoring and open-ended analysis of model reasoning.
- Data sources: LogicVista sources and annotates samples from licensed, registered, and paid materials to reduce data-leakage risk.The dataset also includes creator permissions for IQ tests and related evaluation materials.
- Annotation process: Five annotators collected images, answers, and explanations over three months, then cross-validated explanations through multiple quality checks.The data is structured in JSON for retrieval and processing in the evaluation pipeline.
- Capabilities: The benchmark presents context-free multiple-choice images across nine capabilities, targeting underlying reasoning rather than real-life contextual cues.Capabilities describe the modalities through which logical reasoning questions are delivered, including diagrams, OCR, patterns, graphs, tables, 3D shapes, puzzles, sequences, and physics.
- Reasoning skills: The five reasoning skills are inductive, deductive, numerical, spatial, and mechanical reasoning.Examples include predicting patterns, applying general principles, solving arithmetic, reasoning about spatial transformations, and analyzing gears or physical systems.
- Evaluation annotations: Figure 4 supports evaluation of open-ended model outputs by extracting multiple-choice selections and comparing responses with annotated answers and reasoning.The evaluator uses the question, answer choices, and generated response as context.
4 Evaluation Setup
The evaluation tests representative multimodal language models with an LLM-based answer extractor and reports both overall and disaggregated performance. Scores are computed separately for reasoning skills and multimodal capabilities.
- Models: Eight representative open- and closed-source MLLMs were selected for evaluation, including LLaVA, MiniGPT4, Otter, GPT-4 Vision, BLIP-2, InstructBLIP, and pix2struct.pix2struct was fine-tuned to understand chart and diagram data.
- Answer extraction: The LLM-based extractor isolates answer choices from full-sentence outputs and compares them with ground-truth answers.This accommodates model responses that do not directly output a single option letter.
- Overall scoring: The overall logical reasoning score is the percentage of evaluated samples marked correct across all categories.Each sample contributes a binary correctness indicator to the total score.
- Category scoring: Reasoning-skill scores and multimodal-capability scores are computed as percentages of correct samples within their respective categories.The category denominators are the total samples in each reasoning-skill or capability category.
5 LogicVista Benchmarking and Performance Interpretation
LogicVista reveals substantial weaknesses in multimodal logical reasoning, with performance varying across reasoning types and visual input formats. Larger models tend to achieve better average performance, but diagrammatic reasoning remains especially challenging.
- Overall performance: Many multimodal LLMs perform below a random-guessing baseline on LogicVista’s five logical reasoning categories.The benchmark evaluates models against a baseline assuming five choices per question.
- Overall performance: Recognition-focused benchmarks and training data help explain why models often struggle with logical reasoning tasks.Datasets such as COCO emphasize recognition rather than complex reasoning.
- Logical reasoning skills: Models perform best on deductive, numerical, and mechanical reasoning, while induction and spatial reasoning yield lower performance.The paper relates the stronger categories to their greater prevalence in real-life scenarios and training data.
- Visual capabilities: OCR tasks generally outperform diagrammatic tasks because extracted text can be processed by the underlying language model without the same visual-relationship demands.Diagrams mainly contain lines and shapes, making intricate object relationships harder to extract.
- Model size and performance: Figure 5 suggests a positive correlation between model size and average LogicVista performance.The figure includes a 95% confidence interval for the regression estimate and uses dot size to show duplicated parameter counts.
6 Conclusion
LogicVista addresses the limited evaluation of logical reasoning in multimodal LLMs by benchmarking diverse reasoning capabilities. Its scope is deliberately focused on explicit reasoning tasks, leaving reasoning integrated with real-world scenarios for future study.
- Motivation: Reasoning abilities are important for complex AI tasks, but multimodal benchmarks and training datasets have largely emphasized traditional recognition.The paper identifies this imbalance as a limitation in current evaluation coverage.
- Benchmark contribution: LogicVista provides 448 samples covering five distinct reasoning skills for evaluating multimodal LLM logical reasoning.The benchmark is intended to clarify the current state of these capabilities.
- Evaluation approach: The benchmark uses an LLM-based multiple-choice answer extractor to reduce nondeterminism in multimodal model outputs.This supports more straightforward quantitative evaluation.
- Scope boundary: LogicVista primarily evaluates explicit logical reasoning tasks isolated from real-life contexts.The paper identifies blended abstract reasoning and real-world scenarios as a future research direction.
A Examples of LogicVista Logical Reasoning Data
The appendix illustrates LogicVista’s logical reasoning data through examples spanning inductive, deductive, numerical, spatial, and mechanical reasoning. Questions use varied answer formats and include reasoning annotations or concise answer rationales.
- Inductive reasoning: Inductive examples are organized into three sample cases requiring inductive logical reasoning skills.These examples are listed in Tables 5–7.
- Deductive reasoning: Deductive examples include syllogistic and textual reasoning questions with answers supported by logical implications.Examples include determining what follows from statements about sports players, swallows, and people.
- Numerical reasoning: Numerical examples are presented across three sample cases requiring numerical logical reasoning skills.These examples are listed in Tables 11–13.
- Spatial reasoning: Spatial examples ask models to identify rotations, constructible figures, and objects corresponding to top views.The examples are grouped into three spatial-reasoning cases.
- Mechanical reasoning: Mechanical examples are grouped into three cases involving physical relationships and object properties.The appendix includes questions such as identifying the heaviest ball.
B Examples of Different LogicVista Capabilities Data
The appendix demonstrates LogicVista’s multimodal capability data through diagram, OCR, and mixed examples. These samples cover mechanical, deductive, and numerical-style questions with multiple-choice answers and brief rationales.
- Capability formats: LogicVista capability examples are organized into three cases containing diagram, OCR, and mixed data.Tables 20–22 present the three capability-oriented sample sets.
- Diagram: A diagram example asks which ball is heaviest and permits an insufficient-information answer.The recorded answer is D.
- OCR: An OCR example asks which object is most likely to sink and selects scissors because they contain metal.The recorded answer is B.
- Mixed capability data: A numerical-style capability example asks which legal-sector spending statement is false between Year 4 and projected Year 5.The recorded answer is D.