MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts

1 INTRODUCTION

• Visual mathematical reasoning has applications in education, statistical analysis, and theorem proving and scientific discovery, yet prior benchmarks were narrow or included few math-dependent natural-scene questions.The paper notes that foundation models’ mathematical reasoning in visual contexts had not been systematically examined.Source: §1 · pp. 1–2
• 49.9% accuracy: GPT-4V improves 15.1% over Multimodal Bard, retains a 10.4% overall-accuracy gap to humans, and exceeds them on algebraic reasoning and complex visual contexts.The paper attributes GPT-4V’s advantage mainly to visual perception and mathematical reasoning.Source: Figure 1 · p. 2, §1 · p. 3
• MathVista combines 28 existing multimodal datasets and three new datasets into 6,141 examples, including 736 newly curated examples, spanning seven reasoning types, five task categories, and diverse visual contexts.IQTest targets logical reasoning on puzzle figures, FunctionQA algebraic reasoning over functional plots, and PaperQA scientific reasoning with academic paper figures.Source: §1 · p. 2
• Across 12 foundation models, human annotators achieve 60.3% accuracy on this challenging benchmark, which includes advanced topics such as college curricula and scientific reasoning.The evaluation includes three LLMs, two proprietary LMMs, and seven open-source LMMs.Source: §1 · p. 2
• CoT GPT-4 reaches 29.2%, Multimodal Bard 34.8%, and Bard-caption/OCR-augmented PoT GPT-4 33.9%; Bard failures involve calculation errors and hallucinations from visual perception and textual reasoning.Multimodal Bard’s 34.8% is reported as 58% of human performance (60.3%).Source: §1 · p. 3
• Analyses highlight GPT-4V’s emergent self-verification, self-consistency, and goal-directed multi-turn dialogue capabilities.The paper also attributes its superiority mainly to stronger visual perception and mathematical reasoning.Source: §1 · p. 3

2.1 COLLECTION GUIDELINES

• MathVista’s guidelines target real-world breadth, diverse visual and mathematical challenges, varied difficulty, and deterministic evaluation across seven reasoning types and five primary tasks.The reasoning types are algebraic, arithmetic, geometry, logical, numeric common sense, scientific, and statistical; the tasks are figure question answering, geometry problem solving, math word problems, textbook question answering, and visual question answering. Visual contexts range from natural images and diagrams to charts, scientific figures, tables, function plots, and puzzle figures.Source: §2.1 · pp. 3–4

2.2 DATA COLLECTION

• The benchmark combines nine MathQA datasets totaling 2,666 examples with 19 math-related VQA datasets totaling 2,739 examples.MathQA sources were capped at 400 examples each. For VQA, heuristic filtering selected 4,949 candidates from a review of more than 70 datasets, then three expert annotators used majority voting to finalize the 2,739 examples, with each source capped at 400.Source: §2.2 · p. 4
• Three new datasets address gaps in puzzle-diagram, functional-plot, and academic-figure reasoning: IQTest, FunctionQA, and PaperQA.IQTest has 228 examples on inductive reasoning, abstract thinking, pattern prediction, and calculations; FunctionQA has 400 examples on functional-plot perception and algebraic reasoning; PaperQA has 107 examples derived from academic illustrations.Source: §2.2 · p. 4

2.3 METADATA ANNOTATION

• For 1,000 examples, automatic and human annotations matched exactly on reasoning-type sets 94.1% of the time and on individual labels 98.79% of the time.MathVista metadata includes question type, answer type, language, source, category, task, grade level, visual context, and mathematical reasoning type.Source: §2.3 · pp. 4–5

2.4 DATA PREPARATION AND RELEASE

• 6,141 examples form MathVista’s benchmark, with 1,000 in testmini and 5,141 in the standard test set.Testmini is intended for development validation or limited computing resources; test labels are withheld to prevent contamination, with evaluation provided online. Its KL divergence and total variation from the full-set distribution are 0.008 and 0.035.Source: §2.4 · p. 5

2.5 DATA ANALYSIS

• MathVista draws on 31 source datasets, with multiple-choice and free-form questions whose answers include integers, floating-point numbers, or lists.The paper describes a large unique number of images, questions, and answers as supporting pattern diversity.Source: §2.5 · p. 5

3.1 EVALUATION PROTOCOLS

• Evaluation extracts a short answer from each response, normalizes it to the required format, and computes accuracy for multiple-choice and numerical-answer tasks.The LLM-based GPT-4 answer extractor achieved more than 99.5% accuracy in a preliminary study of 200 examples.Source: §3.1 · p. 6

3.2 EXPERIMENTAL SETUP

• The evaluation covers text-only LLMs, LLMs augmented with image captions or OCR, and open-source and proprietary LMMs, including GPT-4V.Text-only models were tested with zero-shot and two-shot CoT and PoT; GPT-4V was evaluated manually using the playground chatbot because it had no API access.Source: §3.2 · p. 6

3.3 EXPERIMENTAL RESULTS

• 49.9% overall accuracy makes GPT-4V the top model, 15.1% above Bard, while remaining 10.4% below human performance.Bard scored 34.8% and humans 60.3%. Text-only GPT-4 with 2-shot CoT scored 29.2%; augmented GPT-4 with 2-shot PoT scored 33.9%.Source: §3.3 · pp. 6–7, Table 2 · p. 7

3.4 FINE-GRAINED RESULTS

• GPT-4V surpasses most baselines across task categories but trails humans on logical and numeric-commonsense reasoning, while exceeding them in selected math and visual tasks.It surpasses human performance on geometry problem solving, textbook question answering, algebraic reasoning, function plots, geometry diagrams, scatter plots, and tables. Augmented-LLM gaps are attributed to poor captions, OCR’s inability to detect shapes, and limited mathematical reasoning.Source: §3.4 · p. 8

3.5 QUALITATIVE ANALYSIS

• 44.6% of Bard predictions had incorrect answers and explanations, and 49.6% of its incorrect explanations contained hallucinations.Bard also gave partially incorrect explanations for 6.8% and completely incorrect explanations for 8.1% of predictions despite correct answers; some explanations with correct reasoning still hallucinated (18.6%) or had wrong calculations (19.5%). Human annotators evaluated 250 examples.Source: §3.5 · pp. 8–9, Figure 4 · p. 8
• Bard examples show that it can answer correctly with sound reasoning, but can also give a correct answer for the wrong reason, such as misapplying the law of cosines.The example contrasts identifying an injective function correctly with a geometry answer that should have relied on isosceles-triangle properties.Source: Figure 5 · p. 8, §3.5 · p. 9
• Augmented GPT-4 can use accurate OCR for structural reasoning, but incorrect or hallucinated captions can impair visual perception and mathematical reasoning.In an example, PoT GPT-4 used accurate OCR for statistical reasoning, while a caption included hallucinated details; CoT GPT-4 nevertheless reached the correct answer from partially correct information.Source: §3.5 · pp. 9–10, Figure 6 · p. 9

4 RELATED WORK

• Most mathematical-reasoning benchmarks focus on text, and GSM-8K shows performance saturation, motivating robust multimodal benchmarks as LMMs rise (Li et al., 2023a).The work introduces a dataset for mathematical reasoning that incorporates visual contexts.Source: §4 · p. 10
• VQA datasets assess LMM visual reasoning, and studies extend beyond natural images to abstract scenes, geometry diagrams, charts, documents, and synthetic images.MATHVISTA adds IQTest, FunctionQA, and PaperQA to support a holistic mathematical-reasoning benchmark.Source: §4 · p. 10
• Foundation models span general-purpose systems, chart-specialized models, and document-focused models; MATHVISTA evaluates their mathematical reasoning across varied visual contexts.Examples include GPT-3, ChatGPT, GPT-4, Claude, and LLaMA; PixStruct, MatCha, and UniChart for chart reasoning; and LLaVAR for document understanding and math comprehension.Source: §4 · p. 10

• MATHVISTA systematically analyzes the mathematical reasoning capabilities of state-of-the-art models in visually complex scenarios.The evaluation covers 12 prominent foundation models.Source: §5 · p. 10
• 10.4% separates GPT-4V, the best-performing model, from human performance.The conclusion identifies this gap as a direction for research on integrating mathematical reasoning with visual comprehension.Source: §5 · p. 10
• The study explores GPT-4V’s self-verification, self-consistency, and chatbot interactions to inform future investigations.Source: §5 · p. 10

Mathematical reasoning benchmarks

• Most math-reasoning benchmarks are text-only, some are saturated, and existing multimodal benchmarks offer limited coverage of rigorous and scientific domains.The paper notes GPT-4's 92.0% accuracy on GSM-8K as an example of saturation.Source: A DETAILED RELATED WORK · p. 20

Vision-language reasoning benchmarks

• Prior vision-language benchmarks assess general image question answering, specialized visual skills, and abilities such as external-knowledge, common-sense, scientific, and medical reasoning.Examples include VQA (Goyal et al., 2017), geometry-diagram benchmarks (Seo et al., 2015; Lu et al., 2021a; Chen et al., 2022a; Cao & Xiao, 2022), and scientific-knowledge datasets (Lu et al., 2022; Kembhavi et al., 2017; 2016).Source: A DETAILED RELATED WORK · p. 20

Generative foundation models and their evaluation

• Generative foundation models are evaluated on text-grounded tasks, while PixStruct (Lee et al., 2023), MatCha (Liu et al., 2022), and UniChart (Masry et al., 2023) target visual math or chart reasoning.The cited models are described as using specialized pretraining recipes.Source: A DETAILED RELATED WORK · p. 20

Vision-language reasoning benchmarks

• MathVista benchmarks logical, arithmetic, and statistical reasoning across image-based word problems, natural scenes, geometric shapes, and plots.The paper also introduces IQTest, FunctionQA, and PaperQA as new datasets.Source: A DETAILED RELATED WORK · p. 20

Recent work of LLM prompting and GPT-4V

• The paper evaluates zero-shot, few-shot, chain-of-thought, program-of-thought, and tool-augmented prompting on MathVista, and develops LLM baselines using program-of-thought.Program-aided methods are noted for precise logical reasoning and arithmetic calculations (Drori & Verma, 2021).Source: A DETAILED RELATED WORK · p. 21
• The paper provides a comprehensive quantitative and qualitative study of GPT-4V and other LLMs on mathematical reasoning in visual contexts, an area it says remained underexplored.It identifies Zhang et al. (2023b) as concurrent work on whether LMMs perform vision and language tasks consistently or independently.Source: A DETAILED RELATED WORK · p. 21

B LIMITATIONS OF THE BENCHMARK

• MathVista may omit some problem and visual types, is labor-intensive to extend beyond its math-focused domains, and evaluates only recent prominent models.These factors leave its scalability and generalizability to other domains as concerns.Source: B LIMITATIONS OF THE BENCHMARK · p. 21
• 85.6% of examples have annotations, but provider-sourced annotations vary in format, including logic forms, natural-language solutions, and theorems.The examples include Geometry3K logic forms (Lu et al., 2021a), TabMWP natural-language solutions (Lu et al., 2023b), and TheoremQA theorems (Chen et al., 2023).Source: B LIMITATIONS OF THE BENCHMARK · p. 21
• The authors plan to broaden problem and visual-context coverage, unify annotations, refine potential data noise using community feedback, and update the leaderboard as new models emerge.They describe MathVista as an ongoing research effort.Source: B LIMITATIONS OF THE BENCHMARK · p. 21

C.1 MATHEMATICAL REASONING DEFINITION

• Table 3 defines seven mathematical reasoning categories in MathVista.The supplied passage identifies the categories’ number but does not include their definitions or proportions.Source: §C.1 · p. 22, Table 3 · p. 22

C.2 MATHEMATICAL REASONING EXAMPLES

• Table 4 presents examples for each of the seven mathematical reasoning categories.The supplied caption does not provide the examples themselves.Source: Table 4 · p. 23

C.3 VISUAL CONTEXT TYPES

• Figures 7–12 show visual-context examples for geometry diagrams, synthetic scenes, bar charts, natural images, scientific figures, and tables.Source: Figure 7 · p. 24, Figure 8 · p. 24, Figure 9 · p. 24
• Figures 13–18 show visual-context examples for function plots, abstract scenes, puzzle tests, scatter plots, line plots, and pie charts.Source: Figure 13 · p. 25, Figure 14 · p. 25, Figure 15 · p. 25
• Figures 19–21 show examples for document and medical images, plus other types including word clouds, map charts, radar charts, violin plots, and heatmap charts.Source: Figure 19 · p. 26, Figure 20 · p. 26, Figure 21 · p. 26

C.4 SOURCE DATASET SUMMARY

• Table 5 summarizes 31 source datasets by category, task, visual context, and primary mathematical reasoning skills, including new FunctionQA, IQTest, and PaperQA datasets.Source: Table 5 · p. 27

D.1 AUTOMATIC SELECTION OF MATHEMATICAL PROBLEMS

• Table 6 lists quantity words used to automatically select questions likely to involve mathematical reasoning.Source: Table 6 · p. 28

D.2 HUMAN LABELING OF MATHEMATICAL PROBLEMS

• The authors developed a GUI tool to let expert annotators label whether problems involve mathematical reasoning.Source: §D.2 · p. 28, Figure 22 · p. 28
• Annotators received detailed instructions and positive and negative examples for identifying mathematical reasoning.Source: §D.2 · pp. 28–29, Table 7 · p. 28
• The labeling options were Yes, No, and Unsure, with annotators advised to use Unsure sparingly.Source: §D.2 · p. 29
• 0.775 Fleiss Kappa indicates substantial agreement among three annotators labeling examples for mathematical reasoning.The study used Fleiss Kappa to assess inter-annotator agreement.Source: §D.2 · p. 29

D.3 ANNOTATING THREE NEW DATASETS

• A GUI is provided for annotating the new source datasets.Figure 23 shows the annotation interface.Source: Figure 23 · p. 29

D.4 HUMAN LABELING OF MATHEMATICAL REASONING

• A separate GUI is provided for labeling mathematical reasoning skills.Figure 24 shows the labeling interface.Source: Figure 24 · p. 29

E MORE DATASET ANALYSIS

• MATHVISTA includes 6.57% non-English questions (English questions average 15.58 words, with a maximum of 213) and balances math-targeted and general VQA sources at 55.4% and 44.6% across five tasks.Source: E MORE DATASET ANALYSIS · p. 30, Figure 26 · p. 30, Figure 27 · p. 31
• Questions span elementary, high-school, college, and not-applicable grade levels, providing a framework for evaluation across educational contexts.Source: E MORE DATASET ANALYSIS · pp. 30–31, Figure 28 · p. 31
• MATHVISTA covers over 10 visual contexts and seven reasoning types; arithmetic is most frequent, logical reasoning least frequent, and problems average 1.45 reasoning types.Source: E MORE DATASET ANALYSIS · pp. 31–32, Figure 29 · p. 32, Figure 30 · p. 32

F.1 FREQUENT GUESS

• The frequent-guess baseline predicts common answers by question type, including option B for two-choice questions and fixed integer, decimal, and list responses.The listed numeric and list defaults are 2, 1.2, 0.21, and [0, 2, 0, 2, 1, 7, 1, 2, 0, 3, 0, 6], based on testmini answer distributions.Source: §F.1 · p. 33

F.2 PROMPT FOR ANSWER EXTRACTION / F.3 PROMPTS FOR RESPONSE GENERATION

• The appendix provides a GPT-4 answer-extraction prompt with examples and instructions for extracting answers across question and answer types.Source: §F.2 · p. 33, Table 8 · p. 33, Table 9 · p. 34

F.4 PROMPT FOR CAPTION GENERATION

• Multimodal Bard is prompted to generate detailed image captions to augment language models with visual understanding.Source: §F.4 · p. 34, Table 10 · p. 34

F.5 MODEL HYPERPARAMETERS

• Experiments use default hyperparameters unless otherwise specified, with generation parameters detailed for the evaluated language and multimodal models.Source: §F.5 · p. 34, Table 11 · p. 34, Table 12 · p. 35

F.6 HUMAN PERFORMANCE

• The human-performance study assigned each testmini question to five qualified annotators and retained results only from annotators with at least a high-school diploma.Annotators had completed over 5,000 HIT tasks with acceptance scores above 0.99; high-school and college-level problems constitute 30.9% and 10.8% of MATHVISTA, respectively.Source: §F.6 · pp. 34–35

F.7 MULTIMODAL BARD ASSESSMENT TASK

• The Multimodal Bard assessment used an AMT worker interface, and workers were compensated at $18 per hour.Figure 32 shows a screenshot of the assessment interface.Source: §F.7 · p. 35, Figure 32 · p. 35

G.1 RESULTS ON THE TEST SET

• Small differences between test and testmini scores suggest that testmini mirrors the test subset and is useful for model development with limited computing resources.Table 13 reports test-subset accuracy scores for two heuristic baselines, CoT GPT-4, PoT GPT-4, and LLaVA-LLaMA-2-13B.Source: §G.1 · p. 36, Table 13 · p. 36

G.2 SCORES FOR MATH REASONING TYPES

• GPT-4V outperforms other baselines in most reasoning categories; Multimodal Bard is comparable in geometry (47.8% vs. 51.0%) and algebra (46.5% vs. 53.0%).The passage attributes Bard’s strengths to comprehending geometry diagrams and performing algebraic calculations.Source: §G.2 · p. 36, Figure 33 · p. 36
• Among open-source LMMs, LLaVA has the best overall MATHVISTA accuracy and the highest geometry, logical, and statistical reasoning scores, but lags GPT-4V and Multimodal Bard overall.The passage also says LLaMA-Adapter-V2 ties with LLaVA and exceeds GPT-4V by 2.7% in logical reasoning; InstructBLIP beats GPT-4V by 0.3% in numeric commonsense. LLaVAR is on par with Multimodal Bard, which is designed to enhance OCR-text and symbol detection.Source: §G.2 · pp. 36–37
• CoT GPT-4 with OCR texts and Bard captions gains 26.2% over random chance in scientific reasoning, a result linked to specialized OCR and captioning functionalities (Shen et al., 2023; Lu et al., 2023a).The passage presents this performance as evidence of enhanced applicability and accuracy in specific domains.Source: §G.2 · p. 37
• PoT GPT-4 outperforms Multimodal Bard in arithmetic, logical, numeric commonsense, and statistical reasoning, attributed to high-quality code generation for precise mathematical reasoning.The passage presents coding capabilities as effective for complex problem-solving.Source: §G.2 · p. 37

G.3 SCORES FOR VARIOUS VISUAL CONTEXTS

• Figure 34 reports accuracy scores of leading baselines across various visual contexts in MATHVISTA; the supplied passage gives no specific scores or comparisons.The caption identifies the figure’s scope but does not establish a winner.Source: Figure 34 · p. 37

G.4 SCORES ACROSS DIFFERENT GRADE LEVELS

• At high school, GPT-4V scores 61.8% versus humans’ 58.2%; humans score 70.4% on elementary and 52.6% on college questions.The elementary-level human–GPT-4V gap is largest across grade levels, while GPT-4V improves 20.9% over Multimodal Bard in that category.Source: §G.4 · pp. 37–38, Figure 35 · p. 38

G.5 ABLATION STUDY FOR LLMS

• Table 36 examines LLM performance under varying visual-information inputs, but the supplied passage reports no outcome values or winning setting.Source: §G.5 · p. 38

G.6 LLMS WITH DIFFERENT SHOTS

• More few-shot examples do not consistently help LLMs: gains and drops vary by model and prompt setting, and augmented CoT Claude-2 drops 3.4% from 2 to 4 shots.With PoT prompting, ChatGPT gains 3.4% and GPT-4 gains 1.4% from 2 to 4 shots; the authors conclude that augmented-information quality matters more.Source: §G.6 · p. 39, Figure 37 · p. 39

G.7 LMMS WITH DIFFERENT SHOTS

• IDEFICS shows modest improvement as its few-shot count increases on MATHVISTA, but consistent few-shot benefits for LMMs remain an open question.The passage notes prior 4-shot accuracy drops for BLIP-2 (Li et al., 2023b) and InstructBLIP on common-sense reasoning tasks (Li et al., 2023c).Source: §G.7 · p. 39, Figure 38 · p. 39

G.8 HALLUCINATIONS IN MODEL EXPLANATIONS; G.9 MORE EXAMPLES FOR MULTIMODAL BARD

• Examples document hallucinated information in model explanations and Multimodal Bard failures involving calculation, visual perception, image interpretation, and implicit angle measurement.The examples include hallucinations about content not depicted in an image or question, alongside Bard’s incorrect calculation, restricted response to images of people, perception-related hallucination, and angle error.Source: Figure 39 · p. 40, Figure 40 · p. 40, Figure 41 · p. 41

G.10 COMPARISONS OF DIFFERENT MODELS

• Model examples show failures on abstract statistical reasoning and ambiguity, while Multimodal Bard and GPT-4V correctly read a table and calculate its answer.In another example, only LLaVA answers correctly when other models identify jersey numbers but fail to resolve the question’s ambiguity; augmented CoT and PoT GPT-4 succeed on the candy-price problem with correct captions and OCR.Source: Figure 52 · p. 49, Figure 54 · p. 51, Figure 55 · p. 52

H A COMPARATIVE STUDY OF GPT-4V, BARD, AND OTHER MODELS; H.1 GPT-4V PLAYGROUND FOR MANUAL EVALUATION; H.2 LEADERBOARD SCORES

• 15.1% is GPT-4V’s improvement over second-best Multimodal Bard on the MATHVISTA testmini leaderboard; a 10.4% gap to human performance remains.GPT-4V was manually evaluated in a single chat round using the same queries as other LMMs, without additional feedback.Source: §H.1 · p. 54, Figure 58 · p. 55, §H.2 · p. 55

H.3 ABILITIES IN MATHEMATICAL REASONING; H.3.1 ALGEBRAIC REASONING

• GPT-4V surpasses competing models and humans on algebraic reasoning, but low-resolution plots and plots with multiple functions challenge it and other LLMs.Examples show GPT-4V correctly identifying a function and inferring its properties, yet confusing f(x) with f ′(x) when both functions appear.Source: §H.3.1 · p. 56, Figure 59 · p. 56, Figure 60 · p. 57

H.3.2 ARITHMETIC REASONING

• GPT-4V significantly improves on existing models in arithmetic reasoning, which requires fundamental operations combined with understanding visual contexts.Examples include finding a bar-chart difference and calculating probability from a statistical table; other models misread the chart or make probability errors.Source: §H.3.2 · p. 59, Figure 62 · p. 59, Figure 63 · p. 60

H.3.3 GEOMETRY REASONING

• GPT-4V’s geometry performance is comparable to humans, and it answers elementary- and college-level problems correctly with detailed explanations.The college-level example requires interpreting geometric symbols and relations, choosing a theorem, and calculating accurately.Source: §H.3.3 · p. 61, Figure 65 · p. 62

H.3.4 LOGICAL REASONING

• 21.6% accuracy in logical reasoning is a modest 8.1% improvement over random chance, and current foundation models struggle to deduce patterns from figures.The examples show GPT-4V reasoning through one abstract puzzle, while baselines fail to discern shapes or infer patterns in other problems.Source: §H.3.4 · p. 63, Figure 66 · p. 63, Figure 67 · p. 64

H.3.5 NUMERIC COMMONSENSE REASONING

• Numeric commonsense problems challenge existing models, including GPT-4V, which fails to infer a beaker’s maximum measurement despite succeeding on other examples.GPT-4V discerns an optical illusion and recognizes celebrities to calculate an age gap, while Multimodal Bard struggles with those examples.Source: §H.3.5 · p. 66, Figure 69 · p. 66, Figure 70 · p. 67

H.3.6 SCIENTIFIC REASONING

• GPT-4V substantially outperforms other foundation models on scientific reasoning, but a college-level example reveals room for improvement.The task demands interpreting domain-specific figures and applying scientific knowledge; GPT-4V fails on one problem where Multimodal Bard answers correctly with a logical explanation.Source: §H.3.6 · p. 69, Figure 74 · p. 71

H.3.7 STATISTICAL REASONING

• GPT-4V shows strong statistical reasoning, accurately answering questions about a format-rich table and a data-analysis table that other models struggle to interpret.The examples require extracting information from richly formatted or structured tables.Source: §H.3.7 · p. 72, Figure 75 · p. 72, Figure 76 · p. 73

H.4.1 ABSTRACT SCENE; H.4.2 BAR CHART

• Foundation models lag humans on abstract scenes and bar charts; GPT-4V and Multimodal Bard fail on elementary abstract-scene problems, and neither solves the shown bar-chart problems.The bar-chart examples are described as requiring no complex understanding or reasoning.Source: §H.4.1 · p. 74, §H.4.2 · p. 76, Figure 79 · p. 76

H.4.3 FUNCTION PLOT

• GPT-4V outperforms other baselines on function plots and exceeds human performance, correctly identifying functions and inferring properties from digital and hand-drawn plots.The examples demonstrate accurate reasoning across both plot types.Source: §H.4.3 · p. 77
• On a hand-drawn function plot, GPT-4V identifies the function expression and correctly determines its property.Source: Figure 82 · p. 78

H.4.4 GEOMETRY DIAGRAM

• GPT-4V correctly applies geometry theorems and performs calculations, surpassing other models and humans; Multimodal Bard makes calculation errors in the examples.Answering geometry questions requires recognizing fine-grained symbols, variables, and relations in diagrams before applying the relevant theorems.Source: §H.4.4 · p. 79, Figure 83 · p. 79, Figure 84 · p. 80

H.4.5 LINE PLOT

• Current models, including GPT-4V, underperform humans on line-plot reasoning; the authors speculate that OCR detection and value grounding difficulties contribute.The examples use low-resolution plots with many points.Source: §H.4.5 · p. 81, Figure 85 · p. 81, Figure 86 · p. 82

H.4.6 NATURAL IMAGE

• In natural images, limited OCR or object recognition can prevent correct answers to numeric and spatial reasoning questions.The examples show LLaVA and Multimodal Bard failing to identify a clock-panel number, while only GPT-4V correctly predicts an object's location.Source: §H.4.6 · p. 83, Figure 87 · p. 83, Figure 88 · p. 84

H.4.7 PUZZLE TEST

• Puzzle-text reasoning remains difficult: models must infer patterns in shapes and numbers, and GPT-4V recognizes a shape transformation but confuses options D and E.The examples also indicate limited logical reasoning on a number sequence.Source: §H.4.7 · p. 85, Figure 89 · p. 85, Figure 90 · p. 86

H.4.8 SCATTER PLOT

• GPT-4V outperforms other LMMs and humans overall on scatter-plot accuracy, yet it can fail when fine-grained relationships must be understood.Other models struggle with quantitative relationships in a template-generated plot, and current models, including GPT-4V, struggle with an academic-paper plot.Source: §H.4.8 · p. 87, Figure 91 · p. 87, Figure 92 · p. 88

H.4.9 SCIENTIFIC SCENE

• Scientific-scene questions challenge models to align concepts in text and figures; GPT-4V identifies food-web organisms and their relationships, while GPT-4V and Multimodal Bard ground physics symbols using domain knowledge.In one physics example, Bard reasons correctly but formats its answer incorrectly; GPT-4V supplies the correct answer and reasoning.Source: §H.4.9 · p. 89, Figure 93 · p. 89, Figure 94 · p. 90

H.4.10 SYNTHETIC SCENE

• GPT-4V improves on other models in synthetic scenes but remains significantly below human performance, despite examples requiring attribute recognition, arithmetic, and spatial comparisons.The examples involve subtracting objects by attributes and comparing counts based on object properties and positions.Source: §H.4.10 · p. 92, Figure 96 · p. 92, Figure 97 · p. 93

H.4.11 TABLE

• GPT-4V surpasses human performance on table-related reasoning tasks and is the only model reported to answer the push-up table question correctly.The section also describes GPT-4V pinpointing target cells in a complex academic table.Source: §H.4.11 · p. 94, Figure 98 · p. 94, Figure 99 · p. 95

H.4.12 OTHER VISUAL CONTEXTS

• GPT-4V achieves higher overall accuracy than the other models on reasoning tasks in other visual contexts and is the only model reported to answer the violin-plot question correctly.Source: §H.4.12 · p. 96, Figure 100 · p. 96

H.5 SELF-VERIFICATION IN GPT-4V

• GPT-4V can inspect its reasoning, reject invalid candidate answers or intermediate results, and try alternative approaches, but self-verification does not guarantee correctness and is weaker on non-English tasks.The authors report that this ability is not as robust in the other studied models and suggest future work on activating verification consistently and choosing alternative approaches to maximize task success.Source: §H.5 · p. 97

H.6 SELF-CONSISTENCY FOR GPT-4V

• Self-consistency—selecting the most frequent answer across multiple reasoning paths—can correct visual perception errors, calculation mistakes, and hallucinations, but is less effective when GPT-4V misreads complex visuals or misses salient image information.Source: §H.6 · p. 103

H.7 GPT-4V FOR MULTI-TURN HUMAN-AI INTERACTION

• In multi-turn conversations, GPT-4V uses user hints to correct visual errors, reassess reasoning and calculations, correct misinformation with domain knowledge, and aggregate context, but struggles with ambiguous hints and abstract visual concepts.Source: §H.7 · p. 109
• After a user supplies relevant domain knowledge, GPT-4V applies the correct theorem with visual cues to answer a scientific reasoning query.It initially applies an incorrect theorem before the user’s hint.Source: Figure 116 · p. 112
• GPT-4V fails to answer a complex AI-model table question without human help; iterative guidance on the “Model Size” column and further information eventually correct its errors.The task requires identifying relevant table rows and columns, up-to-date AI research knowledge, and advanced arithmetic reasoning.Source: Figure 117 · p. 113, Figure 118 · p. 114
• In an interactive chatbot example, GPT-4V fails to resolve an image’s ambiguity between the numbers 2 and 1 and the single number 21; the correct sum is 3.The visual cues indicate that 2 and 1 should be summed.Source: Figure 119 · p. 115
• Even after helpful hints, GPT-4V struggles to decipher the number of triangles in an abstract geometric shape.Its response counts 1 larger, 3 medium-sized, and 9 smaller triangles, for 13 total.Source: Figure 120 · p. 116, §H.7 · p. 116