PaperlayerView exact revision on arXiv

Source-linked AI summary

DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models

Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, Huan Zhang

arXiv:2411.00836v2cs.CVcs.AIcs.CL

TL;DR

Static visual math benchmarks do not readily test whether VLM reasoning remains robust across variants of a question, even though simple changes can expose consistent failures. DynaMath uses program-based seed questions to generate varied instances, and its evaluation finds worst-case accuracy substantially below average-case accuracy across models.

  • Problem

    Static visual math benchmarks do not readily assess whether VLMs maintain mathematical reasoning across question variations, despite consistent failures on some simple variants.

    Source: §1 · p. 2

  • Method

    DynaMath represents 501 multi-topic seed questions as Python programs that generate concrete variants, and evaluates average-case, worst-case, and repetition consistency.

    Source: §1 · p. 2, §3.3 · p. 6

  • Results

    Worst-case accuracy was close to or less than 50% of average-case accuracy across all considered VLMs, indicating unreliability on question variations.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The findings identify reasoning robustness as a weakness in leading VLMs when question conditions change, including on relatively simple mathematical questions.

    Source: §1 · pp. 2–3

    LLM-assisted dynamic benchmark generation remains future work; DynaMath is a human-crafted baseline for comparison on diversity and quality.

    Source: A LIMITATIONS AND FUTURE WORK · p. 16

Abstract

from arXiv · show

The rapid advancements in Vision-Language Models (VLMs) have shown great potential in tackling mathematical reasoning tasks that involve visual context. Unlike humans who can reliably apply solution steps to similar problems with minor modifications, we found that SOTA VLMs like GPT-4o can consistently fail in these scenarios, revealing limitations in their mathematical reasoning capabilities. In this paper, we investigate the mathematical reasoning robustness in VLMs and evaluate how well these models perform under different variants of the same question, such as changes in visual numerical values or function graphs. While several vision-based math benchmarks have been developed to assess VLMs' problem-solving capabilities, these benchmarks contain only static sets of problems and cannot easily evaluate mathematical reasoning robustness. To fill this gap, we introduce DynaMath, a dynamic visual math benchmark designed for in-depth assessment of VLMs. DynaMath includes 501 high-quality, multi-topic seed questions, each represented as a Python program. Those programs are carefully designed and annotated to enable the automatic generation of a much larger set of concrete questions, including many different types of visual and textual variations. DynaMath allows us to evaluate the generalization ability of VLMs, by assessing their performance under varying input conditions of a seed question. We evaluated 14 SOTA VLMs with 5,010 generated concrete questions. Our results show that the worst-case model accuracy, defined as the percentage of correctly answered seed questions in all 10 variants, is significantly lower than the average-case accuracy. Our analysis emphasizes the need to study the robustness of VLMs' reasoning abilities, and DynaMath provides valuable insights to guide the development of more reliable models for mathematical reasoning.

1 INTRODUCTION

DynaMath is introduced to assess whether VLMs can maintain mathematical reasoning across variations of a question, an ability that static benchmarks do not readily test. Across evaluated models, worst-case accuracy was close to or less than half of average-case accuracy.

1 INTRODUCTION

  • Existing visual math benchmarks use static testing questions, whereas DynaMath evaluates multiple concrete variants generated from each seed question.This allows evaluation sets to exceed the number of seeds, for example by 10×.Source: §1 · p. 2
  • VLM reasoning robustness is questioned because GPT-4o can fail consistently on variants of simple problems that humans can solve despite changes in conditions.For shifted absolute-value functions, GPT-4o answers correctly for some values but incorrectly for many non-zero shifts.Source: §1 · p. 2
  • DynaMath uses 501 Python-program seed questions across multiple topics to generate concrete instances with varied conditions, including numerical values, functions, graphs, and geometry.The questions span elementary through undergraduate difficulty, with high-school and undergraduate questions predominating.Source: §1 · p. 2
  • Worst-case accuracy was close to or less than 50% of average-case accuracy across all considered VLMs, indicating unreliability on question variations.The evaluation used 5,010 concrete problems generated from seed questions.Source: §1 · p. 3

2 RELATED WORK

Prior mathematical reasoning benchmarks span text-only and visual tasks, while recent work has also examined robustness in language models. Existing visual benchmarks nevertheless use static visual and textual content.

2 RELATED WORK

  • Text-based mathematical reasoning benchmarks and robustness studies have examined changing item names or value conditions in language questions.The cited work includes datasets for assessing mathematical reasoning and studies of dynamic benchmarks and robustness in language models.Source: §2 · p. 3
  • Visual math benchmarks cover geometry, multiple-choice questions, and broader topics, but their visual and textual components remain static.The passage names MATHVISTA, MATHVERSE, and MATH-V among the broader benchmarks.Source: §2 · p. 4
  • Prior VLM work includes closed-source models and open-source systems such as InternVL, LLaVA, Qwen-VL, and DeepSeek-VL.The passage surveys approaches ranging from frozen language models with limited trainable parameters to models trained on larger multimodal datasets.Source: §2 · p. 4

3 BENCHMARK DESIGN

DYNAMATH assesses VLMs’ mathematical reasoning robustness using seed questions that generate varied visual and textual problem instances. Its design combines curated questions, program-based generation, dataset statistics, and evaluation across variants.

3.1 DATASET COLLECTION

  • 501 seed questions span nine topics, from plane and solid geometry to statistics and arithmetic.Questions come from existing visual-math datasets, public resources, and 38 newly developed problems.Source: §3.1 · p. 4
  • Python programs encode each seed’s conditions, solution calculations, and image generation, enabling diverse concrete questions.470 programs generate images from sampled conditions; the remaining 31 use fixed images and vary textual elements. Variations include numerical values, geometry, function types, colors, symbols, graph structures, and real-life contexts.Source: §3.1 · p. 5

3.2 DATASET STATISTICS

  • Questions range from elementary to undergraduate levels, with 55.3% at high-school and 32.1% at undergraduate level.The dataset covers nine topics and includes 59.1% numerical, 34.7% multiple-choice, and 6.2% free-form questions; its free-form share contrasts with cited benchmarks that predominantly have more than 50% multiple-choice questions.Source: §3.2 · p. 5
  • Nearly 93% of seed questions have more than 10 possible variants, and 17.4% have more than 106.Approximately 30.5% have 10–102 possible variants.Source: §3.2 · p. 6

3.3 EVALUATION PROTOCOLS

  • Answer extraction uses prompt engineering and template matching, while repetition consistency represents confidence in an answer across repeated generations.Repetition consistency is calculated from K repetitions of a response to a given variant.Source: §3.3 · p. 6
  • The protocol generates 10 variants for each of 501 seeds and evaluates average-case and worst-case accuracy, unlike single-instance benchmarks.The two measures are defined over generated answers and ground truth; reasoning robustness is the ratio between average-case and worst-case performance.Source: §3.3 · p. 6

4 EXPERIMENT

The experiments evaluate VLMs on 5,010 generated variants of 501 seed questions, with comparisons across models, topics, difficulty, and repeated outputs. Worst-case accuracy falls well below average-case performance, while results and qualitative examples reveal sensitivity to changed conditions and several recurring error types.

4.1 EXPERIMENTAL SETUPS

  • The setup tested closed- and open-source VLMs on 10 variants per seed, with zero-shot and three-shot CoT configurations for closed-source models and temperature 0.0 throughout.Human performance was measured using 1,002 questions comprising two variants per seed, answered by 20 STEM undergraduates or graduates; few-shot demonstrations were topic-specific.Source: §4.1 · p. 7

4.2 EXPERIMENTAL RESULTS

  • 35.3% was Claude-3.5’s highest zero-shot worst-case accuracy; Qwen2-VL-72B scored 28.3%, and Llava-v1.6-vicuna-13B scored 2.8%.Worst-case accuracy counts a seed as correct only when all 10 variants are answered correctly.Source: Table 3 · p. 8, §4.2 · p. 8
  • 64.8% was Claude-3.5’s highest zero-shot average accuracy, compared with 77.3% human performance; three-shot CoT did not improve every closed-source model.GPT-4o rose from 63.7% to 64.9%, while Claude-3.5 fell from 64.8% to 62.5% and Gemini Pro 1.5 from 60.5% to 58.7%.Source: §4.2 · p. 8
  • On Puzzle Test, GPT-4o and InternVL2-76B reached average-case accuracies of 51.8% and 35.3%, respectively, while humans scored 73.5%.All open-source models had 0.0% worst-case accuracy on this topic except InternVL2-76B at 5.9%; VLM average accuracy also decreased with difficulty, unlike human performance.Source: §4.2 · p. 8
  • Reasoning robustness was highest for Claude-3.5 and GPT-4o; closed-source models generally exceeded open-source models, and robustness scaled with open-source model size.Across topics, GPT-4o and Qwen2-VL-72B were particularly robust in Arithmetic and Algebra, while both were weak on Puzzle Test.Source: §4.2 · p. 9, Figure 5 · p. 9
  • Repetition consistency ranged from 92% to 99%, yet some seeds produced consistent failures in selected variants; these cases affected 21.8% of GPT-4o’s 501 seeds.The reported rates were 18.4% for Gemini Pro 1.5, 29.9% for Qwen2-VL-72B, and 28.3% for InternVL2-76B.Source: §4.2 · p. 9, Table 4 · p. 9

4.3 QUALITY STUDY

  • GPT-4o answered one function-graph variant correctly but misread the function’s position with high confidence after a slight image shift.Claude 3.5 Sonnet’s answer remained 2π when sinusoidal periods changed; this memorization phenomenon accounted for 4.1% of its 171 incorrect questions.Source: §4.3 · pp. 9–10, Figure 7 · p. 10
  • Figure-reading errors made up 33.1% of Claude 3.5 Sonnet’s analyzed errors, reasoning errors 26.6%, calculation errors 18.9%, and hallucinations 17.8%.The analysis examined 169 failed questions and also categorized knowledge errors.Source: §4.3 · p. 10

5 CONCLUSION

DYNAMATH uses program-based problem generation to examine VLM reasoning robustness under changed visual conditions. The evaluation finds that leading open- and closed-source models are sensitive to these changes despite unchanged underlying capabilities.

5 CONCLUSION

  • DYNAMATH uses program-based problem generation to create diverse variants by altering visual conditions in seed problems.The benchmark is designed to systematically analyze mathematical reasoning robustness in leading VLMs.Source: §5 · p. 10
  • Leading open- and closed-source VLMs are sensitive to variant condition changes, although the required underlying capabilities remain the same.The paper presents this as an evaluation finding.Source: §5 · p. 10
  • DYNAMATH, a dynamic visual math benchmark, evaluates VLM robustness; leading closed- and open-source models are sensitive to question-variant condition changes despite unchanged underlying capabilities, revealing weaknesses and error causes.Source: §5 · p. 10

A LIMITATIONS AND FUTURE WORK

DYNAMATH’s current scope and scalability are constrained by question difficulty, seed selection, single-type variants, and the human effort required to curate visual problems. The authors discuss LLM-assisted generation as a possible way to reduce that effort, while retaining human quality checks.

A LIMITATIONS AND FUTURE WORK

  • DYNAMATH’s difficulty is relatively limited compared with MATH-V because adapting very challenging questions to program structures requires substantial human effort.The benchmark is described as matching MATHVERSE and MATHVISTA in difficulty.Source: A LIMITATIONS AND FUTURE WORK · p. 16
  • Seed selection may introduce bias, and the current generation process varies only one type per seed rather than combining types such as color and numerical-value variants.Puzzle Tests, the most challenging topic for VLMs, account for 3.4% of the dataset.Source: A LIMITATIONS AND FUTURE WORK · p. 16
  • Scaling requires substantial human effort to curate seeds and design dynamic visual elements, including drawing-program quality control and verification.These requirements complicate full automation.Source: A LIMITATIONS AND FUTURE WORK · p. 16
  • LLMs could generate diverse problems and parameterizable Python programs for visual variants, using DYNAMATH seeds as demonstrations to reduce manual effort.Human intervention would still be needed to select correct, high-quality samples.Source: A LIMITATIONS AND FUTURE WORK · p. 16
  • LLM-assisted dynamic benchmark generation remains future work; DYNAMATH is a human-crafted baseline for comparing future systems on diversity and quality.The authors also propose exploring adversarial training and feedback-based approaches, while noting cost-effective adaptation to VLMs remains an open challenge.Source: A LIMITATIONS AND FUTURE WORK · p. 16

B VARIATION TYPES OF DYNAMATH

DYNAMATH varies seed questions in six distinct ways, using that variation diversity to evaluate VLM visual robustness.

B VARIATION TYPES OF DYNAMATH

C DETAILED EXPERIMENT SETUP

The appendix details response prompts, answer extraction, few-shot chain-of-thought examples, model settings, and sample problems across DYNAMATH topics.

C.1 PROMPTS FOR RESPONSE GENERATION

  • Response-generation prompts cover multiple-choice, float, and text question types.Table 5 is identified as showing prompts for different question and answer types.Source: §C.1 · p. 17, Table 5 · p. 18

C.2 PROMPTS FOR ANSWER EXTRACTION

  • Answer extraction asks for solution and short-answer fields in JSON, but some smaller open-source models do not reliably follow that format.Supplementary scripts check and extract informal answers from inconsistent responses.Source: §C.2 · pp. 17–18

C.3 PROMPTS FOR FEW-SHOT CHAIN OF THOUGHT

  • The few-shot chain-of-thought appendix includes a plane-geometry example that reasons from a tangent-radius right angle and applies Pythagoras’ theorem.The example asks for the sine of an angle given a semicircle diameter and tangent length.Source: §C.3 · pp. 18–19

C.4 MODEL HYPERPARAMETERS

  • Generation parameters are set to defaults except temperature, which is set to 0 for closed- and open-source models to reduce randomness.Table 6 is captioned as listing hyperparameters for various VLMs.Source: §C.4 · p. 20, Table 6 · p. 20

D VARIANT EXAMPLES FOR DIFFERENT TOPICS IN DYNAMATH

E MORE DEMONSTRATION EXAMPLES FOR FEW-SHOT EXPERIMENT

These examples illustrate worked solutions to visual and text-based math questions, including weight puzzles, arithmetic, and interpreting a scientific figure.

E MORE DEMONSTRATION EXAMPLES FOR FEW-SHOT EXPERIMENT

F ERROR TYPE EXAMPLES

The appendix illustrates VLM errors on example problems and examines robustness across image variants, including variation types that remain challenging for leading models.

F ERROR TYPE EXAMPLES

  • Claude 3.5 examples span figure-reading, calculation, reasoning, hallucination, and knowledge errors, including misread coordinates and height and unsupported reasoning.Examples also include an incorrect sector angle and a missing-number answer whose final reasoning is described as incorrect.Source: F ERROR TYPE EXAMPLES · pp. 32, 33, 35, 36

G QUESTION VARIANT EXAMPLES

H.1 REASONING ROBUSTNESS ON DIFFERENT VARIATION TYPES

  • GPT-4o and Qwen2-VL-72B are sensitive to graph-structure, geometric-transformation, and function-type changes; Qwen2-VL-72B is also vulnerable to symbolic substitution.Source: §H.1 · p. 39, Figure 12 · p. 39

H.2 ADDITIONAL FAILURE CASE ANALYSIS

The additional analyses relate failure patterns to difficulty and topic, report detailed reasoning-robustness results, and examine prompt changes and memorization examples.

H.2 ADDITIONAL FAILURE CASE ANALYSIS

  • High-school and undergraduate problems account for most failure cases; reasoning, hallucination, figure-reading, and calculation errors are more prevalent than knowledge errors.Source: §H.2 · p. 39, Figure 13 · p. 39
  • Failure patterns differ by topic: puzzle tests show reasoning errors, several visual topics show figure-reading errors, and solid geometry and algebra show calculation errors.Plane geometry has high incidences of hallucination and reasoning errors.Source: §H.2 · p. 40, Figure 14 · p. 40

H.3 DETAILED REASONING ROBUSTNESS RESULTS OF ZERO TEMPERATURE

  • Table 7 presents reasoning robustness for 14 models across DynaMath, with results organized by overall accuracy, question topics, and difficulty levels.Source: §H.3 · p. 40, Table 7 · p. 40

H.4 RESULTS OF DIFFERENT PROMPT TEMPLATE

  • The designed prompt improved average and worst-case accuracy for GPT-4o and Qwen2-VL-72B, but did not significantly improve reasoning robustness.Source: §H.4 · p. 41, Table 8 · p. 41

H.5 MORE ON MEMORIZATION PHENOMENON

  • With the new prompt, GPT-4o still judged a shifted absolute function non-differentiable at x = 0, while Claude-3.5-Sonnet kept reading a sinusoid's period as 2π despite image changes.Source: §H.5 · p. 41

H.6 EVALUATING THE ROBUSTNESS OF DYNAMATH

The appendix probes possible DynaMath data leakage with in-context examples and reports repeated-evaluation variance. Few-shot gains were marginal, while measured variance was small.

H.6 EVALUATING THE ROBUSTNESS OF DYNAMATH

  • Using variants 1–3 as few-shot context produced only marginal gains; full pretraining or fine-tuning leakage tests were infeasible, and the same variant-4 question served as a control.The in-context experiments used GPT-4o and Qwen2-72b.Source: §H.6 · pp. 41–42

H.7 VARIANCE OF AVERAGE ACCURACY

  • Across five repetitions, average-accuracy variance was approximately 1–2 percentage points for GPT-4o, Gemini, Qwen2-VL, and InternVL2.Source: §H.7 · p. 42, Table 10 · p. 42

H.8 MORE RESULTS ON CIRCULAR CONSISTENCY

The authors assess circular consistency on DynaMath multiple-choice questions for GPT-4o and Qwen2-VL 76B, finding high consistency under changes to answer-choice order.

H.8 MORE RESULTS ON CIRCULAR CONSISTENCY

  • 90.2% for GPT-4o and 92.2% for Qwen2-VL 76B were their repetition-consistency scores under circular shifts.The authors report that outputs were consistent in most cases regardless of choice order.Source: §H.8 · p. 42
  • Circular consistency applies to multiple-choice questions with static question content, where only answer-choice order changes.The authors initially did not test it because DynaMath primarily focuses on image-based variants.Source: §H.8 · p. 42
  • The authors evaluated circular consistency for two representative models on DynaMath MCQs, following Liu et al. (2023).Source: §H.8 · p. 42
Loading 2411.00836v2…