PaperlayerView exact revision on arXiv

Source-linked AI summary

MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts

Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, Jianfeng Gao

arXiv:2310.02255v3cs.CVcs.AIcs.CLcs.LG

TL;DR

Mathematical reasoning in visual contexts has not been systematically evaluated, despite the importance of combining visual understanding with mathematical reasoning. MathVista addresses this gap with a diverse benchmark and evaluates 12 foundation models. GPT-4V achieves the best reported accuracy, but a substantial gap remains relative to human performance.

  • Problem

    Existing benchmarks provide limited evidence about mathematical reasoning in visual contexts because they are mainly textual or narrowly focused on particular tasks or visual settings.

  • Method

    MathVista combines 28 existing multimodal datasets and three newly created datasets into a benchmark spanning seven mathematical reasoning types, five tasks, and diverse visual contexts.

  • Results

    GPT-4V achieves 49.9% overall accuracy, outperforming Multimodal Bard by 15.1% while remaining 10.4% below human performance.

  • Takeaways & Limitations

    MathVista provides a systematic test of mathematical reasoning integrated with visual comprehension and identifies substantial room for improvement in foundation models.

  • Takeaways & Limitations

    Dataset coverage, scalability, and generalizability to other domains remain concerns, and only 85.6% of examples have annotations with non-unified formats.

Abstract

from arXiv · show

Large Language Models (LLMs) and Large Multimodal Models (LMMs) exhibit impressive problem-solving skills in many tasks and domains, but their ability in mathematical reasoning in visual contexts has not been systematically studied. To bridge this gap, we present MathVista, a benchmark designed to combine challenges from diverse mathematical and visual tasks. It consists of 6,141 examples, derived from 28 existing multimodal datasets involving mathematics and 3 newly created datasets (i.e., IQTest, FunctionQA, and PaperQA). Completing these tasks requires fine-grained, deep visual understanding and compositional reasoning, which all state-of-the-art foundation models find challenging. With MathVista, we have conducted a comprehensive, quantitative evaluation of 12 prominent foundation models. The best-performing GPT-4V model achieves an overall accuracy of 49.9%, substantially outperforming Bard, the second-best performer, by 15.1%. Our in-depth analysis reveals that the superiority of GPT-4V is mainly attributed to its enhanced visual perception and mathematical reasoning. However, GPT-4V still falls short of human performance by 10.4%, as it often struggles to understand complex figures and perform rigorous reasoning. This significant gap underscores the critical role that MathVista will play in the development of general-purpose AI agents capable of tackling mathematically intensive and visually rich real-world tasks. We further explore the new ability of self-verification, the application of self-consistency, and the interactive chatbot capabilities of GPT-4V, highlighting its promising potential for future research. The project is available at https://mathvista.github.io/.

1 INTRODUCTION

MathVista addresses the lack of systematic evaluation for mathematical reasoning in visual contexts by combining diverse mathematical tasks, reasoning types, and visual settings. Evaluation shows that foundation models remain challenged, although GPT-4V substantially outperforms other models while still trailing humans overall.

  • Existing benchmarks mostly assess mathematical reasoning in text or narrow visual settings, leaving comprehensive vision-language evaluation largely unexplored.
  • MathVista combines seven reasoning types, five primary tasks, diverse visual contexts, 28 existing multimodal datasets, and three newly created datasets.The new datasets are IQTest, FunctionQA, and PaperQA.
  • Human annotators achieve 60.3% accuracy, demonstrating that MathVista is a challenging benchmark involving advanced mathematical and scientific topics.
  • 34.8% accuracy makes Multimodal Bard the best-performing LMM in the reported comparison, while augmented PoT GPT-4 reaches 33.9%.CoT GPT-4, the best-performing unaugmented LLM, achieves 29.2%.
  • 49.9% accuracy makes GPT-4V state of the art, improving 15.1% over Multimodal Bard but remaining 10.4% below human performance.The analysis attributes GPT-4V’s superiority mainly to visual perception and mathematical reasoning.
  • The study also examines GPT-4V’s self-verification, self-consistency, and goal-directed multi-turn human-AI dialogue capabilities.

2 THE MATHVISTA DATASET

MathVista is constructed to cover varied mathematical skills, tasks, and visual contexts through curated existing datasets and three new datasets. Its metadata, quality procedures, and split design support detailed model analysis, while coverage and annotation consistency remain bounded by source heterogeneity.

  • The collection guidelines target multiple tasks and topics, diverse visual contexts and mathematical skills, and varying challenge levels.
  • The taxonomy covers seven reasoning types and five primary tasks, including figure question answering, geometry problem solving, math word problems, textbook question answering, and visual question answering.
  • The benchmark includes 2,666 examples from nine multimodal MathQA datasets, with each source capped at 400 examples for balanced representation.
  • Nineteen reviewed VQA datasets were selected because they contain publicly available instances involving mathematical reasoning.
  • The three new datasets address logical reasoning on puzzle diagrams, algebraic reasoning over functional plots, and scientific reasoning with academic figures.IQTest has 228 examples, FunctionQA 400, and PaperQA 107.
  • Manual annotation and review produced 99.2% inter-annotation consistency across the 736 newly collected questions.Only six questions exhibited disagreements in annotated answers.
  • Metadata records question type, answer type, language, source, category, task, grade level, and visual context for comprehensive analysis.
  • MathVista contains 6,141 examples divided into a 1,000-example testmini split and a 5,141-example test split.Test labels are withheld to prevent data contamination, with online evaluation maintained.

3 EXPERIMENTS

The experiments benchmark text-only, augmented, and multimodal foundation models on MATHVISTA, combining quantitative evaluation with fine-grained and qualitative analyses. GPT-4V performs best overall, while errors remain tied to visual perception, mathematical reasoning, and imperfect auxiliary visual information.

  • Experimental Setup: MATHVISTA evaluates foundation models through response generation, answer extraction, and score calculation, including text-only, augmented-LLM, and LMM setups.The benchmark includes prompting with CoT or PoT and augmentation with image captions and OCR text.
  • Experimental Results: 29.2% is achieved by 2-shot GPT-4 with CoT among text-only LLMs, while augmented GPT-4 with PoT reaches 33.9%.The augmented model generates Python programs for execution, promoting rigorous reasoning.
  • Experimental Results: 49.9% is achieved by GPT-4V, improving 15.1% over Multimodal Bard, but remaining 10.4% below human performance.Multimodal Bard scores 34.8%, or 58% of the 60.3% human baseline.
  • Fine-Grained Results: GPT-4V surpasses human performance in several categories, including GPS, TQA, algebraic reasoning, function plots, geometry diagrams, scatter plots, and tables.Its exceptions include logical reasoning and numeric commonsense reasoning.
  • Qualitative Analysis: Augmented-LLM performance is limited by poor captions, OCR failures to detect geometric shapes, and insufficient mathematical reasoning.External visual information can therefore provide inaccurate or incomplete descriptions of mathematical content.
  • Qualitative Analysis: 44.6% of Bard predictions contain both incorrect answers and incorrect explanations, while 49.6% of incorrect explanations contain hallucinations.Bard also gives correct answers with partially or completely incorrect explanations in 6.8% and 8.1% of cases, respectively.

4 RELATED WORK

Prior benchmarks largely assess mathematical reasoning in text or narrow visual settings, while VQA datasets contain only limited math-related content. MathVista addresses this gap by combining existing datasets with three new datasets for a holistic multimodal evaluation.

  • Mathematical Reasoning Benchmarks: Many mathematical reasoning benchmarks focus solely on text-based tasks, creating a need for robust multimodal benchmarks in scientific domains.GSM-8K is cited as an example of a benchmark exhibiting performance saturation.
  • Visual Question Answering: Existing visual reasoning datasets cover settings such as natural images, abstract scenes, geometry diagrams, figures, charts, documents, and synthetic images.The paper introduces IQTest, FunctionQA, and PaperQA to create a holistic benchmark for mathematical reasoning.
  • Foundation Models: Generative foundation models support diverse task solutions without fine-tuning, while specialized methods enhance chart reasoning and image-text understanding.Examples include PixStruct, MatCha, UniChart, LLaVA, miniGPT4, InstructBLIP, Bard, and LLaVAR.

5 CONCLUSION

MATHVISTA systematically evaluates mathematical reasoning in visually complex scenarios and shows substantial progress alongside a persistent gap between GPT-4V and humans.

  • MATHVISTA systematically analyzes the mathematical reasoning capabilities of state-of-the-art models in visually complex scenarios.
  • 10.4% separates GPT-4V, the best-performing model, from human performance.
  • GPT-4V’s self-verification, self-consistency, and chatbot interactions provide insights for future investigations.

A DETAILED RELATED WORK

Prior benchmarks often evaluate mathematical reasoning textually or within narrow visual settings, while general vision-language benchmarks assess broader image understanding. MATHVISTA addresses this underexplored intersection alongside rapidly developing foundation models and prompting methods.

  • Mathematical reasoning benchmarks: Most mathematical reasoning benchmarks are textual only, despite mathematical information and reasoning being substantially represented visually.
  • Vision-language reasoning benchmarks: General-purpose vision-language benchmarks evaluate open-ended questions about images, with or without task-specific training.
  • Generative foundation models and their evaluation: Generative foundation models are trained on web-scale data and can solve diverse downstream tasks without task-specific fine-tuning.
  • Generative foundation models and their evaluation: Vision-language foundation models are trained on paired and interleaved image-text data.
  • Recent work of LLM prompting and GPT-4V: Chain-of-thought, program-of-thought, and inductive reasoning have been used to enhance LLM reasoning capabilities.
  • Recent work of LLM prompting and GPT-4V: GPT-4V shows promising vision-language reasoning performance, but fine-grained studies of its strengths and limitations remain underexplored.

B LIMITATIONS OF THE BENCHMARK

MATHVISTA combines mathematical and visual tasks but acknowledges limits in coverage, scalability, generalizability, and model understanding of complex figures and rigorous reasoning.

  • MATHVISTA combines mathematical and visual tasks, while models still face challenges with complex figure understanding and rigorous reasoning.
  • The benchmark may underrepresent certain types of mathematical problems and visual contexts.
  • Its specialized, labor-intensive data collection raises concerns about scalability and generalizability to other domains.
  • Only 85.6% of examples have annotations, and the annotations lack a unified format and structure.
  • Future iterations aim to broaden problems and visual contexts, unify annotations, refine data noise, and update the leaderboard for new models.
  • Visual coverage: The benchmark includes visual contexts such as geometry diagrams, synthetic scenes, bar charts, natural images, scientific figures, tables, function plots, abstract scenes, and puzzle tests.
  • Visual coverage: Additional visual contexts include scatter plots, line plots, pie charts, document images, medical images, word clouds, maps, radar charts, violin plots, and heatmaps.

C.4 SOURCE DATASET SUMMARY

MATHVISTA summarizes heterogeneous source datasets and uses annotation tools and expert labeling to identify mathematical reasoning, while measuring agreement among annotators.

  • Source datasets: The source-dataset summary covers 31 datasets, including newly annotated FunctionQA, IQTest, and PaperQA.
  • Source datasets: The source-dataset table records each dataset’s category, task, visual context, and primary mathematical reasoning skill types.
  • Data selection: A dictionary of quantity words supports automatic selection of questions likely to involve mathematical reasoning.
  • Annotation: An annotation tool enables expert annotators to label whether problems involve mathematical reasoning.
  • Annotation: Annotators use Yes, No, or Unsure labels, with examples and detailed instructions guiding the decisions.
  • Agreement analysis: Fleiss Kappa measures agreement reliability among the three annotators.

D.3 ANNOTATING THREE NEW DATASETS

The appendix presents graphical interfaces for annotating the three new source datasets and labeling mathematical reasoning skills.

  • Figure 23 shows the GUI used to annotate the new source datasets.
  • Figure 24 shows the GUI used to label mathematical reasoning skills.

E MORE DATASET ANALYSIS

MATHVISTA combines multilingual, varied-format questions with diverse dataset categories, educational levels, visual contexts, and mathematical reasoning types. Its analysis describes how these dimensions support broad evaluation, while documenting answer formats and human-assessment procedures.

  • Question distribution: 6.57% of MATHVISTA questions are non-English, while English questions average 15.58 words and reach a maximum length of 213 words.
  • Question distribution: Questions include multiple-choice and free-form formats, with free-form answers standardized across integers, floating-point numbers, and lists.
  • Dataset category and task type: MATHVISTA balances math-targeted and general VQA datasets at 55.4% and 44.6%, respectively.
  • Visual context: MATHVISTA spans over 10 visual contexts, including geometry diagrams, synthetic scenes, bar charts, natural images, and scientific figures.
  • Mathematical reasoning ability: Seven mathematical reasoning types are represented, with arithmetic most frequent and logical reasoning least frequent.
  • Human assessment: Human-performance evaluation used five qualified annotators per testmini question, retaining participants with at least a high-school diploma or equivalent attainment.

F.7 MULTIMODAL BARD ASSESSMENT TASK

The Multimodal Bard assessment used a worker interface and compared model accuracy on the test subset. Testmini closely mirrors test-subset performance, supporting its use for model development under limited computing resources.

  • Assessment setup: Human assessment workers were compensated at $18 per hour through the AMT interface.
  • Model comparison: Table 13 compares two heuristic baselines, two augmented LLMs, and LLaVA-LLaMA-2-13B on MATHVISTA's test subset.
  • Subset validity: Minor score differences between test and testmini indicate that testmini mirrors the test subset and can support model development with limited computing resources.

G.2 SCORES FOR MATH REASONING TYPES

Model performance varies across mathematical reasoning types and visual contexts. GPT-4V generally leads, while open-source models and augmented LLMs show narrower strengths in particular domains.

  • Mathematical reasoning types: GPT-4V outperforms other baselines in most mathematical reasoning categories, except logical and numeric commonsense reasoning.
  • Proprietary LMM comparison: 51.0% versus 47.8% shows GPT-4V slightly ahead of Multimodal Bard in geometry reasoning, while 53.0% versus 46.5% shows a larger algebraic-reasoning lead.
  • Open-source LMMs: LLaVA achieves the best overall accuracy among open-source LMMs and leads their fine-grained geometry, logical, and statistical reasoning scores.
  • Augmented LLMs: CoT GPT-4 gained 26.2% over random chance in scientific reasoning after augmentation with OCR texts and Bard captions.
  • Visual contexts: GPT-4V outperformed human performance in function plots, geometry diagrams, scatter plots, tables, and other visual contexts.

G.4 SCORES ACROSS DIFFERENT GRADE LEVELS

MATHVISTA performance varies across grade levels and prompting settings, while GPT-4V substantially advances over prior baselines but remains below humans overall. Few-shot gains are inconsistent, and multimodal models exhibit calculation, hallucination, and reasoning failures.

  • Grade-level performance: 70.4% is human accuracy on elementary-school questions, compared with 52.6% on college-level questions.
  • Grade-level performance: 61.8% is GPT-4V accuracy on high-school problems, exceeding humans at 58.2% and Multimodal Bard at 50.3%.
  • Few-shot settings: Larger numbers of few-shot examples do not consistently improve LLM performance and can produce unstable drops, especially with augmented inputs.
  • Failure analysis: Multimodal Bard failures include incorrect calculations, visual-perception hallucinations, restricted responses to people images, and incorrect reasoning.

H.3.1 ALGEBRAIC REASONING

Algebraic reasoning in MATHVISTA requires models to interpret function plots and infer their properties. GPT-4V performs strongly on some function-plot tasks but still struggles with low-resolution figures and algebraic relationships.

  • Performance: GPT-4V surpasses competing models and humans on algebraic reasoning involving function plots.
  • Visual limitations: Both GPT-4V and other LLMs struggle to comprehend low-resolution figures.
  • Successful reasoning: Multimodal Bard and GPT-4V can identify a depicted function and compute the requested answer to specified decimal places.
  • Failure cases: GPT-4V struggles with algebraic relationships even when a plot intersects the y-axis at y = 1.
  • Failure cases: GPT-4V confuses f(x) with f ′(x), while Multimodal Bard gives the correct answer but inaccurate information about f(x)’s properties.

H.3.2 ARITHMETIC REASONING

MATHVISTA reveals uneven performance across visual mathematical reasoning contexts: GPT-4V excels on several structured formats but remains weak on abstract scenes, logical reasoning, and fine-grained visual interpretation. Self-verification can improve performance, yet it does not reliably resolve difficult perceptual or reasoning failures.

  • Logical reasoning: 21.6% accuracy in logical reasoning leaves GPT-4V only 8.1% above random chance.These tasks require abstract thinking to infer patterns in numbers or shapes.
  • Abstract and chart contexts: Foundation models significantly underperform humans on abstract scenes and bar-chart reasoning, including tasks that do not require complex understanding.Multimodal Bard and GPT-4V both fail on some elementary-style abstract-scene problems and bar-chart questions.
  • Structured visual contexts: GPT-4V achieves strong results on function plots, geometry diagrams, tables, and other visual contexts, sometimes surpassing human performance.It accurately identifies functions and properties in digital and hand-drawn plots, solves elementary and college-level geometry problems, and handles complex tables.
  • Perception-dependent reasoning: Fine-grained visual interpretation remains a bottleneck across line plots, natural images, scatter plots, and numeric commonsense tasks.Reported failure sources include inaccurate OCR, poor value grounding, limited object recognition, and difficulty with optical illusions or detailed scatter-plot questions.
  • Self-verification: Self-verification improves GPT-4V’s MATHVISTA performance but does not guarantee correctness in intricate visual-perception or mathematical-reasoning scenarios.The ability is also weaker on non-English tasks and is not observed as robustly in other evaluated foundation models.
Loading 2310.02255v3…