GSM8K-V: Can Vision Language Models Solve Grade School Math Word Problems in Visual Contexts

1 Introduction

• GSM8K-V tests reasoning over implicit visual information distributed across multiple images, addressing a gap in benchmarks centered on explicit symbols.Existing visual math benchmarks mainly use structured inputs such as diagrams, plots, and labeled visual statements.Source: §1 · p. 1
• Each text problem becomes a semantically equivalent multi-image sequence requiring visual extraction and cross-scene integration rather than symbol reading.Relevant cues include counting objects, reading signboards, interpreting gestures, and inferring actions or state changes from scenes.Source: §1 · p. 2, Figure 1 · p. 2
• The benchmark specifically tests implicit inference of quantities, actions, and relations, as well as tracking entities and evidence across scenes.For example, a money exchange can imply a purchase even when that action is not stated explicitly.Source: §1 · p. 2
• The construction pipeline maps text problems into visual form and uses human verification to produce 1,319 samples with 5,343 images.Its stages decompose problems, generate scene descriptions, and render images before verification.Source: §1 · p. 2
• 27 of 34 VLMs exceed 90% on GSM8K, but Gemini-3-Pro reaches only 59.06% on GSM8K-V, compared with 91.15% human accuracy.VL-Rethinker-7B gains on MathVista, MathVerse, and MathVision but shows no gain on GSM8K-V (-0.6%).Source: §1 · p. 2

2.1 Mathematical Reasoning of VLMs

• Prior VLM research studies multimodal reasoning, contextual and structure-aware reasoning, and methods such as chain-of-thought supervision and reinforcement learning.Visual mathematical reasoning remains challenging because it requires grounding symbolic relations in visual contexts, integrating multimodal information, and performing multi-step inference.Source: §2.1 · p. 3

2.2 Mathematical Benchmarks for VLMs

• Visual math benchmarks span geometry, competition problems, redistributed text and diagram information, process evaluation, robustness, Olympiad-level tasks, video, and systematic visual reasoning.Examples include MathVista, MATH-Vision, MathVerse, MM-MATH, DynaMath, Omni-MATH, VideoMathQA, and VisuLogic.Source: §2.2 · p. 3
• Many existing benchmarks use explicit symbolic carriers, whereas GSM8K-V uses semantically equivalent everyday multi-scene versions of GSM8K problems.This enables direct comparison of text reasoning with visual reasoning dependent on perception, implicit inference, and cross-scene integration.Source: §2.2 · p. 3

3 Benchmark Construction

• GSM8K-V converts GSM8K problems into multiple visually expressed scenes, using generated descriptions, GPT-Image-1 rendering, and human validation and refinement.The benchmark construction pipeline is presented as a systematic process from text problems to scene descriptions and images.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Problem Decomposition and Allocation

• Problems are parsed into object, math value, and semantic triples, then grouped across scenes without inferring or merging atomic facts.The allocation principles include contextual grouping, isolating the final question, and retaining atomic mathematical facts.Source: §3.1 · pp. 3–4

3.2 Scene Description Generation

• Category-specific meta-description strategies guide GPT-4.1 to produce scene descriptions structured around objects, actions, and composition.The schema specifies entities carrying mathematical information, their states or activities, and spatial arrangements.Source: §3.2 · pp. 4–5

3.3 Image Generation

• Generated scenes undergo dual human cross-validation for consistency, completeness, and compliance, with refinement repeated until requirements are met.GSM8K-V contains 1,319 samples and 5,343 images, averaging four images per case across six broad and 13 fine-grained categories.Source: §3.3 · p. 5, Figure 3 · p. 5

4.1 Evaluation Setup

• The evaluation covers closed- and open-source VLMs, uses official APIs or vLLM for inference, and prompts models to reason over images and return an integer answer.Human annotators also answer directly from the generated images.Source: §4.1 · p. 5

4.2 Overall Performance

• Gemini-3-Pro drops from 95.68% on text GSM8K to 59.06% on GSM8K-V, while Kimi-K2.6 reaches 47.34% and 20 of 24 models exceed 90% on text.Table 1 compares models across six GSM8K-V categories and overall accuracy, with text-only GSM8K in the rightmost column.Source: §4.2 · pp. 5–6, Table 1 · p. 6
• Errors are particularly frequent in Ratio & Percentage and Other, where many models score below 20%, while Measurement performs relatively better.Measurement tasks typically involve fewer scene transitions and more explicit visual cues, imposing lower cross-scene integration demands.Source: §4.2 · p. 6
• Humans achieve 91.15% average accuracy and remain balanced across categories, while VLMs vary more and drop sharply on Other and Temporal.Gemini-3-Pro performs relatively well across categories requiring implicit reasoning but remains well below human accuracy.Source: §4.2 · pp. 6–7

5.3 Image Style Ablation

• Performance differences across Pixar, Ghibli, and real-world styles are small, with no systematic degradation under photorealistic rendering.GPT-4o performs comparably on real-world and Pixar styles.Source: Table 4 · p. 7, §5.3 · p. 7

5.1 Comparison between GSM8K-V and Other Mathematical Benchmark

• +3.10 MathVista, +3.90 MathVerse, and +3.42 MathVision points contrast with -0.42 points on GSM8K-V after reasoning post-training.The transfer gap supports GSM8K-V as a complementary diagnostic for implicit visual inference and cross-scene integration.Source: §5.1 · p. 7, Table 2 · p. 8

5.2 Cross-Scene Integration Analysis

• 25.11%→11.42% for GPT-4o and 28.31%→11.87% for GLM-4.5V under randomized order on the Cross-Scene Subset, showing strong dependence on scene order.The order effect is stronger on the 219-sample subset, where solving depends on correct scene order.Source: §5.2 · p. 7, Table 3 · p. 8

5.5 Error Analysis

• 53.2% of Gemini-3-Pro’s analyzed failures are IVIE, versus 42.8% EPE and 4.0% MSE, locating most errors before symbolic calculation.IVIE includes failures involving actions, groupings, state changes, scope constraints, and cross-scene relations.Source: Figure 4 · p. 8, §5.5 · p. 8

5.4 Modality Ablation

• OCR-based inputs yield extremely low accuracy; caption-based inputs improve performance but remain far below text-only despite semantic equivalence.The persistent gap points to visual understanding challenges, and captions do not fully preserve implicit relations needed for reasoning.Source: §5.4 · pp. 7–8, Table 5 · p. 9

• GSM8K-V evaluates mathematical reasoning when information is implicitly encoded across multiple images, using a semantically equivalent visual counterpart of GSM8K.The benchmark isolates visual perception and cross-scene integration from mathematical computation.Source: §6 · p. 9
• Across 34 VLMs, text-based reasoning is nearly saturated while frontier models still struggle with implicit visual reasoning.The conclusion also reports that models improved on structured math benchmarks show no gains on GSM8K-V.Source: §6 · p. 9
• GSM8K-V provides a complementary diagnostic dimension for implicit visual inference and cross-scene integration while sharing general visual-language capabilities with existing visual-mathematical benchmarks.Source: §6 · p. 9

• GSM8K-V covers everyday arithmetic word problems with integer answers, not geometry, algebra, proof, or scientific reasoning.Source: Limitations · p. 9
• Its synthetic scenes cannot fully reproduce the ambiguity, diversity, long-tail phenomena, or natural data distributions of real-world imagery.The authors frame it as a controlled diagnostic benchmark rather than a substitute for naturally collected visual data.Source: Limitations · p. 9
• Static multi-image sequences do not represent continuous video or active perception, such as changing viewpoints or gathering information through interaction.The authors describe the benchmark as not a complete simulation of real-world embodied reasoning.Source: Limitations · p. 9

• All GSM8K-V evaluation samples come from one-to-one mappings of GSM8K, which had already undergone ethical review.Source: Ethical considerations · p. 9
• Annotators received thorough training and made objective judgments without engaging in subjective human preferences or ethical concerns.Source: Ethical considerations · p. 9
• Annotators were compensated fairly for their workload and unanimously reported satisfactory task experiences and appropriate rewards.Source: Ethical considerations · p. 9

A Comparison between GSM8K-V and Other Mathematical Benchmark

• GSM8K-V tests implicit mathematical information across multiple images, unlike benchmarks centered on structured diagrams or explicit visual symbols.Its reasoning requires cross-scene integration and implicit visual inference; Table 6 compares benchmarks across multiple dimensions.Source: A Comparison between GSM8K-V and Other… · p. 13

B Detailed Evaluation Setup

• The evaluation setup section introduces detailed evaluation settings and modes.The supplied passage does not specify the settings themselves.Source: B Detailed Evaluation Setup · p. 13

B.1 Model Hyperparameters

• Open-source models use temperature 0.1 to reduce generation randomness.The passage refers to Table 7 for the model generation parameters.Source: §B.1 · p. 13

B.2 Complete Evaluation Results

• Table 8 reports performance across six visual reasoning categories, overall GSM8K-V accuracy, and text-only GSM8K accuracy.The supplied passage describes the table's coverage but does not provide its cell values.Source: §B.2 · p. 13

B.3 Evaluation Mode

• The evaluation-mode section provides definitions, examples, input formats, questions, and experimental prompts.These materials cover the evaluation modes described in the section.Source: §B.3 · p. 13

B.3.1 Text Only

• The text-only prompt specifies answer formatting and asks models to solve the supplied word problem step by step.The examples include a crayons change question and a pen-cost question.Source: §B.3.1 · p. 13

B.3.2 Implicit and Explicit

• Explicit textual queries generally yield slightly higher accuracy across most models, but state-of-the-art performance remains limited.The passage reports the explicit-setting result for Gemini-2.5-Pro but omits its numerical value.Source: §B.3.2 · p. 13
• The comparison distinguishes explicit visual symbols from implicit information that requires perception, cross-image integration, and inference from context.The supplied passage defines these dimensions and describes benchmark domains.Source: §B.3.2 · p. 14
• The performance gap extends beyond understanding the question to the complex multi-image reasoning required to solve problems.The authors state that strong textual reasoning on GSM8K does not translate to visual success.Source: §B.3.2 · pp. 13, 15

B.4 Modality Valid

• OCR-extracted text alone yields very low accuracy, and adding images produces only limited improvement, indicating that text recognition alone is insufficient.The results also indicate that errors involve capabilities beyond text recognition.Source: §B.4 · p. 16
• A scene sequence can imply an action without explicitly encoding it, requiring inference from continuity between scenes.In the example, a later scene retains the agent and table but omits the apples, leaving the act of eating to be inferred.Source: §B.4 · p. 16

B.5 Single-Image(r*) vs. Multi-Image(r*)

• Ordered multi-image scenes better preserve temporal and semantic structure than concatenating all information into one image, while randomized order can obscure cues.Examples also show failures involving scene-specific numbers and temporal alignment.Source: §B.5 · p. 19

B.6 Image Style Effects

• Pixar, Ghibli, and real-world renderings preserve the same mathematical semantics, scene structure, and ground-truth answer while varying only visual style.The examples attribute failures to extracting visual evidence and reconstructing relations, rather than primarily to a specific style.Source: §B.6 · p. 19

C.1 Math Information Classification

• GSM8K mathematical information is classified into 13 categories, including weight, distance, price, and count, with category-specific rendering strategies.The classification is intended to represent mathematical content properly in visual form.Source: §C.1 · p. 26

C.2 Assign Category

• GPT-4.1 assists with allocating categorized information to positions for visual representation.Figure 6 illustrates the assignment process.Source: §C.2 · p. 26

C.3 API Token Usage and Cost of GSM8K-V

• The construction-cost accounting covers GPT-4.1 APIs in Stages 1 and 2 and GPT-Image-1 for rendering in Stage 3.Pricing details are based on official documentation at the time of the experiment; amounts are not given in the supplied passage.Source: §C.3 · p. 26

C.4 Human cross check and refinement

• Two independent trained annotators review each problem and its scenes for consistency, completeness, and benchmark compliance.The verification process is intended to catch mismatches between original text, generated images, and ground-truth solutions.Source: §C.4 · p. 26

C.5 Robustness to Image Generation Models

• GPT-Image-1 and Nano Banana 2 yield highly consistent VLM results, whereas Midjourney v8 alpha and especially v7 reduce performance amid weaker visual consistency, clarity, and detail fidelity.The authors conclude that with sufficiently capable image generation, underperformance primarily reflects VLM limitations in implicit visual reasoning rather than generation artifacts.Source: §C.5 · p. 26, Table 11 · p. 29

D.1 Step1 Prompt

• The decomposition prompt extracts the mathematics needed to answer each GSM8K problem into structured (object, math value, semantic) triples and allocates it across scenes.It specifies classification rules, rendering strategies, scene allocation, and optional interference settings, producing JSON for visual construction.Source: §D.1 · p. 31

D.2 Meta Description Strategy

• Meta-description templates express mathematical information—such as measurements, ratios, clock times, prices, actions, and labels—as visually clear, renderable scenes while preserving semantics.Strategies specify how objects, actions, and composition convey each information type.Source: §D.2 · p. 31

D.3 Image Generation Prompts

• The image-generation prompt requires one image per scene in the specified order, with consistent style and character design and all numerical and descriptive details preserved.These constraints aim to improve consistency and fidelity between each scene and its assigned mathematical information.Source: §D.3 · p. 31

D.4 Controlled Interference

• Controlled interference preserves problem-relevant mathematics while adding visual clutter or semantically related distractors; interference-related Entity Identity Confusion accounts for only a small fraction of failures.The authors identify implicit visual inference, rather than distractor sensitivity alone, as the main bottleneck.Source: §D.4 · p. 31

D.5 Image Generation Prompts and Verification Principles

• Image construction uses iterative human verification for consistency, completeness, and compliance, refining or rerendering scenes that violate these principles.Verification checks preserved entities, quantities, and constraints; visually available solution information; and safety, formatting, and legible numerals, while visual-text links can use pointing, gaze, torso orientation, or holding or touching.Source: §D.5 · pp. 31, 36
• Prompts require objects to be fully visible, non-overlapping, easily countable, and visually distinct, with slight positional, size, or orientation variation.Objects are to be placed evenly within the specified semantic setting.Source: §D.5 · p. 37
• Speech-bubble scenes require a character matching the specified object, a bubble stating the semantic content, and a visible tail without overlap with other objects.Source: §D.5 · p. 37
• For labeled-object scenes, the object must match the specified object field, and its attached or inherent surface label must match the math value.Source: §D.5 · p. 37
• Figure 12 presents fine-grained error distributions for GPT-4o and Qwen2.5-VL-72B, with EPE, IVIE, and MSE decomposed into subtypes.Source: Figure 12 · p. 40

Error Analysis

• 53.4% of GPT-4o failures and 56.1% of Qwen2.5-VL-72B failures are EPE, while IVIE accounts for 44.0% and 41.6%, respectively.Together, EPE and IVIE account for the vast majority of analyzed failures, before final symbolic calculation.Source: E Error Analysis · p. 40
• CSIE is the largest IVIE component, followed by IAUE, CSSTE, SUE, and SGRE; models often miss intended relations despite relevant information being present.The appendix analysis finds IVIE remains a major source of failure for GPT-4o and Qwen2.5-VL-72B, beyond OCR, counting, or instrument reading.Source: E Error Analysis · p. 40
• 2.6% of GPT-4o errors and 2.3% of Qwen2.5-VL-72B errors are MSE, making symbolic calculation errors rare relative to perception and visual-inference errors.MSE covers equation construction or numerical computation after the relevant visual information has been recovered.Source: E Error Analysis · p. 40
• In the flower example, models misread or ignore package sizes, so their subsequent arithmetic follows incorrect visual grounding rather than a primary symbolic-calculation error.Gemini-2.5-Pro treats both options as three-flower packs; GPT-4o and Qwen2.5-VL-72B also fail to recover the package-size information.Source: E Error Analysis · p. 42
• In the apple example, models omit Beck’s eating event and compute 5 + 3 instead of 5 − 2 + 3, illustrating contextual semantic interpretation failure.The shared failure is missing the implied state change; Qwen2.5-VL-72B additionally miscounts Boris’s apples.Source: E Error Analysis · p. 44
• In the couch example, models treat a 2-foot relation as a local gap or direct length instead of applying it to twice the rug width.This is identified as a Scope Understanding Error within IVIE.Source: E Error Analysis · p. 46

F Category Case Study

• 18 feet is the ground-truth couch length, but another solution concludes the couch is 2 feet long from a rug dimension.The ground truth derives a rug length of 8 feet and then calculates 2 × 8 + 2.Source: F Category Case Study · pp. 47–48
• 5 hours is the supported answer for Tyler’s books, whereas an alternate solution reports 6.25 hours using a $4 wage rather than $5.The ground truth calculates a $25 shortfall and divides by the stated $5 hourly wage.Source: F Category Case Study · pp. 49–50
• 34 ounces is the ground-truth meat total, while one image-based solution reports 32 after counting 10 beef tips instead of 12.The ground truth uses 6 ounces, 12 one-ounce tips, and a 16-ounce steak; the alternate solution sums 6, 10, and 16.Source: F Category Case Study · p. 51
• Over four weeks, daily 30-minute morning and night meditation totals 840 minutes, and weekday 30-minute yoga totals 600 minutes, for 1,440 minutes (24 hours) combined.Source: F Category Case Study · p. 54
• 30 teachers is one image-based answer, while another estimates 10 by treating teachers as one-tenth of a 100-student school.The 30-teacher solution reads the student-to-teacher chart as a 5:1 ratio after deriving 150 students.Source: F Category Case Study · pp. 56–57
• 18 dollars is the ground-truth daily egg-sale income, while one solution reports 16 after counting 15 daily eggs rather than 16.The ground truth subtracts breakfast and muffin eggs from 16; the alternate solution begins with 7 eggs plus 8 more.Source: F Category Case Study · p. 58