PaperlayerView exact revision on arXiv

Source-linked AI summary

MathVis: Can AI Reason through Drawing? A Benchmark for Mathematical Figure Generation

Jingjing Wei, Hao Zhao, Xinke Li, Linxi Yu, Xiaosong Gao, Shanshan Xiao

arXiv:2610.12954v1cs.AI

TL;DR

Existing evaluations provide limited evidence about advanced mathematical figure construction and do not readily separate coding errors from mathematical interpretation errors. MathVis benchmarks executable figure generation with a Gold IR and atomic rubrics; on its test set, strict D-PSR varied across evaluated configurations, while the benchmark identifies continuing difficulty with spatial and geometric construction.

  • Problem

    Existing mathematical evaluations focus largely on scalar calculations or elementary geometry, leaving limited evidence about advanced spatial construction and separating code errors from geometric interpretation errors.

    Source: §1 · p. 3

  • Method

    MathVis benchmarks executable TikZ/SVG figure generation using Gold IR, mathematical predicates, presentation scores, and strict-delivery criteria.

    Source: §1 · p. 3, §7 · p. 16

  • Results

    Strict D-PSR across the 16 evaluated configurations ranged from 9.18% to 30.73%, and evaluated LLMs still had difficulty with spatial and geometric construction.

    Source: §1 · p. 3, §7 · p. 16

  • Takeaways & Limitations

    MathVis provides a common test bed for studying how models connect symbolic and visual representations in advanced mathematics.

    Source: §1 · p. 3

    The 512 tasks are not a random sample of the full difficulty distribution, so conclusions are restricted to the documented task set and Harnesses.

    Source: §6 · p. 15

Abstract

from arXiv · show

This paper introduces MathVis, a benchmark designed to evaluate LLMs' ability to visualize frontier mathematical problems and perform diagram-assisted reasoning. The benchmark consists of 512 tasks, each comprising a mathematical statement, a reference diagram, executable TikZ or SVG code, a Gold Intermediate Representation (IR), atomic rubrics, and error codes. Through rigorous evaluation, GPT-6-Astra, Claude Fable 5, and Gemini 3.1 Pro Preview achieve pass@1 scores of 31.25%, 26.56%, and 22.85%, respectively. We found that although the tasks require visualization, LLMs produce outputs dominated by formal languages, symbols, and graphics programs, rather than explicit diagram-mediated reasoning. Furthermore, while the solutions generated by LLMs may contain all the necessary elements, they are often visually confusing to humans. Ultimately, this benchmark aims to visualize mathematical problems and promote research on enabling LLMs to simplify their reasoning via visual diagrams.

1 Introduction

Mathematical diagrams can make complex relations more intuitive, but producing dependable figures requires models to align symbols with space, satisfy geometric constraints, generate precise code, and preserve theorem-level consistency. MathVis targets gaps in prior evaluation and tests whether models can connect symbolic and visual representations in advanced mathematics.

1 Introduction

  • Advanced mathematical figures can compress abstract structural relations and reduce symbolic ambiguity, but even small topological or metric deviations can derail deduction.The paper cites high-dimensional topological relations, morphism compositions, and manifold coordinate transformations as examples.Source: §1 · p. 2
  • Precise mathematical figure generation stress-tests four capabilities: cross-modal alignment, implicit geometric reasoning, structured code generation, and semantic diagram verification.The challenges include resolving spatial constraints without explicit coordinates and maintaining consistency between local elements and global theorems.Source: §1 · p. 2
  • Existing mathematical evaluations provide limited evidence about advanced spatial construction and do not readily distinguish code errors from errors in geometric interpretation.Many focus on scalar calculations or elementary geometry.Source: §1 · p. 3
  • MathVis covers 27 mathematical sub-disciplines, including differential geometry and complex analysis, and pairs executable code with Gold IR to separate programming errors from mathematical interpretation errors.Here, “advanced” refers to upper-level undergraduate, graduate, or research-facing objects or relations whose figures require more than elementary planar construction.Source: §1 · p. 3
  • Evaluated LLMs still struggle with spatial and geometric construction, and generated figures may include requested elements without organizing them into representations that support reliable reasoning.The evaluation does not establish that access to diagrams causally improves reasoning.Source: §1 · p. 3

2 Related Work

Related benchmarks address educational figure generation, formal geometry, theorem explanation, and general vector graphics, but their scopes differ from MathVis’s focus on executable, verifiable figures for advanced mathematics. The paper identifies gaps in both educational level and disciplinary depth.

2 Related Work

  • Educational benchmarks cover K–12 graphics or hint illustrations, whereas MathVis applies verification to mathematical structures from research-level material.MathemaTikZ includes 3,793 curriculum figures, while Math2Visual provides 1,903 annotated examples; both focus on basic mathematics [Malik et al., 2025; Wang et al., 2025a].Source: §2 · p. 4
  • Formal geometry systems use constraints or coordinate solvers for construction, but their scope remains elementary planar geometry rather than advanced structures such as bundles or homotopy relations.MathVis extends intermediate representations and constraint verification to a cross-disciplinary Gold IR with hard and soft atomic predicates.Source: §2 · p. 4
  • TheoremExplainAgent generates theorem-explanation videos, while AutomaTikZ, DeTikZify, and TikZero address scientific figure synthesis or program recovery rather than MathVis’s advanced static-mathematics focus.TheoremExplainBench evaluates 240 theorems [Ku et al., 2025], and AutomaTikZ releases approximately 120,000 captioned TikZ figure pairs [Belouadi et al., 2024a].Source: §2 · p. 4
  • Visual Sketchpad, SGP-Bench, VGBench, MathVerse, and MATH-Vision study visual reasoning or graphics competence, while MathVis targets executable, verifiable figures generated from advanced mathematical statements.The paper distinguishes MATH-Vision [Wang et al., 2024] from Math-Vision Diagrams [Kashyap et al., 2026].Source: §2 · pp. 4–5
  • The paper frames two practical gaps: mathematical figure generation largely targets school-level material, while broader theorem or scientific-figure resources may lack organized coverage of advanced mathematical disciplines.The comparison concerns task coverage, not the overall value of those resources.Source: §2 · p. 5

3 Method

MathVis combines a curated and audited task set with expert-reviewed task packages and a ground-truth evaluation framework. Its protocol retains partial diagnostic scores while requiring technical validity and complete satisfaction of mathematical and presentation constraints for strict delivery.

3.2 Benchmark Construction

  • Figure 4 summarizes a three-stage curation pipeline and verification architecture that produce frozen task packages.The evaluation set is frozen with hidden evaluator assets.Source: §3.2 · p. 6, Figure 4 · p. 6

3.3 Evaluation Set Coverage

  • The evaluation set contains 512 questions from 14 source collections and 27 sub-disciplines; its 201 task-family labels are descriptive metadata, not train/dev/test splits.The set is not claimed to represent the overall difficulty distribution, and Figure 5 summarizes sub-discipline composition.Source: §3.3 · p. 6, Figure 5 · p. 7

3.4 Task Packages and Annotation Quality

  • Task authors create problems and figures, domain reviewers assess mathematical and visual properties, and verification experts convert accepted solutions into Gold IR, atomic predicates, and automated checks.Independent dual-expert review, adjudication, validator testing, and model trial runs contribute to auditable quality control before freezing.Source: §3.4 · pp. 6–7

3.5 Evaluation Protocol

  • pass@3 measures whether at least one of three outputs strictly passes, while pass^3 measures whether all three pass; strict delivery requires technical validity, every mathematical hard predicate, and complete soft-constraint satisfaction.The normalized rubric score preserves partial diagnostic information; aggregate rubric coverage is not tabulated in this version.Source: §3.5 · p. 7

4 Experiments

The experiments compare 16 model configurations under shared benchmark settings, then examine agent challenge examples and detailed mathematical constructions. Across the cases, tool use does not consistently resolve mathematical or structural errors.

4.1 Models and Runs

  • Models receive the same prompts and output contract without reference figures, and must return executable TikZ or SVG code.A Harness fixes the execution protocol, including prompts, sampling parameters, code extraction, compilation, rendering, validators, and run logs.Source: §4.1 · pp. 7–8

4.2 Main Results

  • Observed strict D-PSR spans 9.18%–30.73% across 16 configurations; GPT-6-Astra ranks highest, followed by Claude Fable 5 at 25.13%.Figure 6 orders the evaluated configurations by observed strict D-PSR; the paper cautions that these are descriptive results for this evaluation set, not a universal ranking.Source: Figure 6 · p. 8, §4.2 · p. 9

4.6 Harness and Agent Challenge Examples

  • Agent and direct baselines are independent configurations evaluated under the same tasks, prompts, and budget definitions; challenge examples do not transfer baseline outcomes to agents.An agent is recorded as still failing only when its final artifact has D=0 after the preregistered three-round budget.Source: §4.6 · p. 9, Table 1 · p. 9

4.7 Detailed Case Analysis

  • Across Rouché and Mandelbrot cases, tool-augmented runs still miss hard mathematical constraints or structural relations, and extra steps do not reliably repair off-track constructions.The cases include missed Rouché constraints and incomplete Mandelbrot external-ray and co-landing structure.Source: §4.7 · p. 9

4.8 Agent Test Instance: Application of Rouché’s Theorem

  • The Rouché rubric gates strict delivery on hard mathematical constraints; GPT-5.6-sol mis-specifies the contour and image curves, while Claude Fable 5 misses image labels and the equal-zero conclusion.The GPT-5.6-sol figure also lacks a valid three-zero or three-turn winding representation.Source: §4.8 · pp. 11, 13

SUMMARY

  • Step-by-step guidance appeared to improve some outputs, but the agent-plus-Harness configuration did not yield consistent improvement, and the observation is not a measured effect size.The guidance asks models to draw the two images on the contour, identify loops, and relate them to zeros.Source: SUMMARY · p. 13

4.9 Agent Test Instance: Mandelbrot Set

  • Mandelbrot outputs miss required Böttcher equipotentials and complete rays: Gemini misplaces the period-4 branch, GPT-5.6-sol has an inaccurate boundary, and Claude Opus 4.8 returns no code after truncation.The GPT-5.6-sol output also has incomplete characteristic rays and insufficiently strict co-landing points.Source: §4.9 · pp. 13, 15

5 Discussion

The discussion connects observed failures to possible training and deployment choices. It recommends treating strict correctness, validation, readability, and deployment costs as distinct concerns.

5 Discussion

  • Gold IR and atomic predicates make failure modes observable, supporting targeted curriculum design, hard negatives, verifier-guided fine-tuning, and repair rather than image-level optimization alone.The paper suggests compilation-only objectives may be insufficient because failures also concern mathematical relations, label binding, global constraints, and readable spatial organization.Source: §5 · p. 15
  • Deployment metrics should match operating mode: candidate selection may suit systems with trusted validators, while direct single-artifact delivery needs a strict output-level gate.Soft scores can prioritize repairs but should not override mathematical or technical hard gates.Source: §5 · p. 15
  • The proposed layered deployment architecture is not yet validated for latency, cost, or user benefit, which should be explicit deployment parameters.The proposed order is compile and render, verify mathematical predicates, then assess readability after hard correctness.Source: §5 · p. 15

6 Limitations and Responsible Use

The paper limits its conclusions to documented tasks and Harnesses and cautions against broad population-level claims. It also highlights risks from benchmark sampling and imperfect automatic validation.

6 Limitations and Responsible Use

  • The evaluated configurations support transparent comparisons, but not unbiased population-level claims about all mathematical problems, model families, or agent settings.Broader generalization requires additional sources, independent expert calibration, and replication under new configurations.Source: §6 · p. 15
  • The 512 tasks are not a random sample of the full difficulty distribution, and source coverage and expert-authored predicates may reflect available conventions.Results can also change with prompts, providers, sampling parameters, renderers, tool permissions, and repair budgets.Source: §6 · p. 15
  • Independent agent and direct conditions do not justify transferring a baseline outcome to an agent; validators can miss equivalent forms or accept misleading figures, so users should verify propositions.Verification is especially important for unfamiliar notation, new theorems, or settings where an incorrect diagram could affect a proof or explanation.Source: §6 · p. 15

7 Conclusion

MathVis separates technical validity, mathematical correctness, visual presentation, and strict delivery, with observed strict D-PSR scores of 9.18%–30.73% across 16 configurations. The conclusion also records artifact, evaluation, and review boundaries.

7 Conclusion

  • Strict D-PSR values for the 16 evaluated configurations range from 9.18% to 30.73%.The benchmark separates technical validity, mathematical predicates, graded presentation scores, and full-specification strict delivery.Source: §7 · p. 16

AI Usage Declaration

  • Generative AI assisted literature discovery, drafting, editing, typesetting, and some dataset and evaluation organization, with authors reviewing the outputs and taking responsibility.The authors independently verified the reported statistics, mathematical definitions, citations, and final artifacts.Source: AI Usage Declaration · p. 16

Ethics Statement

  • Automatic evaluation alone does not establish classroom or research suitability; teachers and mathematical experts should verify figures against the propositions.The paper documents a review workflow but reports no finalized inter-rater summary table.Source: Ethics Statement · p. 16

Reproducibility Statement

  • The released artifact supports output-level recomputation for three frozen configurations, while the remaining configurations cannot be independently reconstructed from it.Their aggregate rows can be checked for consistency with stated denominators.Source: Reproducibility Statement · p. 16

Artifact Inventory

  • The analysis release includes strict-run and rubric files, Gold IR records, TSV summaries, question lists, and XLSX scorecards, with identity metadata removed.Source records preserve source-figure rights status.Source: Artifact Inventory · p. 16

Model Family Scorecard Notes

  • Only three of the 16 configurations have question-level logs; the other 13 have aggregate scorecard rows and do not support question-level or paired reanalysis.Provider routes, temperature settings, and output budgets are not fully tabulated for every configuration; missing operational fields remain NA.Source: Model Family Scorecard Notes · pp. 16–17

A Contributions

The contributions establish MathVis as an evaluation-only benchmark for advanced mathematical figure generation, pairing executable programs with structured semantic and presentation evaluation. Its curation screens candidate questions, specifies auditable instances, and checks reference artifacts before expert review.

A Contributions

  • MathVis extends mathematical figure evaluation beyond predominantly K–12 settings to 27 sub-disciplines and releases prompts, references, Gold IR, rubrics, source information, error codes, and accepted equivalent transformations.The benchmark is evaluation-only and targets advanced mathematical figure generation.Source: A Contributions · p. 19
  • The framework treats figure generation as constrained executable programs and separately evaluates execution, mathematical semantics, visual expression, and strict deliverability.Its unified metrics also retain formal definitions of repeated-generation and rubric diagnostics.Source: A Contributions · p. 19
  • The paper reports scorecard metrics for 16 configurations on 512 official questions under a three-rollout protocol, noting which configurations retain question-level logs.Related work includes executable or structured verification approaches, while MathVis uses code executability as a technical gate and Gold IR with atomic predicates as semantic gates.Source: A Contributions · p. 19, B Method-Related Work · p. 19

C.1 Question Screening Process Statistics

  • Screening began with 8,114 candidate propositions and froze 512 official questions after sequential checks for source completeness, visualizability, unique instances, and auditable verification.Candidates were removed at the first triggered screening stage to avoid double counting.Source: §C.1 · pp. 19–20

C.2 Stage 1: Theorem Screening and Instance Specification

  • Theorem screening favors propositions with stable, auditable visual structure; controlled variants may test whether models track the theorem rather than memorize one standard figure.Constructors freeze a source-supported instance and define public prompts separately from hidden atomic constraints, equivalences, and evidence rules.Source: §C.2 · pp. 20–21

C.3 Stage 2: Candidate Reference Figure Generation and Automatic Pre-check

  • Reference artifacts use paths suited to the mathematical content, and automatic pre-checks test schema, rendering, resource bounds, and low-cost mathematical quantities before expert review.Model-assisted drafts may accelerate writing but do not define ground truth; expert verification determines admission.Source: §C.3 · p. 21

C.4 Stage 3: Expert Verification and Arbitration

Stage 3 reviews frozen task packages at the predicate level and returns or removes those with unresolved ambiguity, invalid references, or inadequate constraints. The documented calibration protocol has not yet yielded reported aggregate agreement statistics.

C.4 Stage 3: Expert Verification and Arbitration

  • Stage 3 experts verify source tracking, instance validity, prompts, reference programs, Gold IR, constraints, and equivalent forms, returning or removing packages with unresolved defects.Decisions are recorded down to the predicate level.Source: §C.4 · p. 21
  • The protocol designates a stratified 5% calibration subset for independent dual-person review, with third-expert arbitration for disagreements on admission or hard constraints.The paper documents intended agreement measures but reports no final aggregated calibration coefficients.Source: §C.4 · p. 22
  • The four elimination stages reduced 8,114 collected questions to 512; the third-stage count of 627 concerns propositions lacking a source-supported canonical instance after revision.This does not mean every proposition with multiple legal drawings is automatically excluded.Source: §C.4 · p. 22
  • Predicates cover an understanding-to-action chain from parsing and relation extraction through program synthesis, rendering, and verification, while error codes distinguish omissions, relation errors, misbindings, and unreadable delivery.Alternative layouts are allowed, but visually similar approximations cannot replace the mathematical propositions.Source: §C.4 · pp. 22–23
  • The current version escalates weak automatic evidence or high-impact cases for expert review, but does not report aggregate calibration statistics.Eliminated questions and their revision reasons are retained as part of the auditable curation process.Source: §C.4 · p. 23

D Complex Analysis Predicate Family

The complex-analysis predicate family applies only the checks that are mathematically relevant to each question. The expert rubric template records predicate-level decisions and separates hard constraints from weighted soft judgments.

D Complex Analysis Predicate Family

E Expert Verification Rubric Template

  • Stage 3 separates task admission from model-output scoring so errors in reference programs or Gold IR are not misjudged as model failures.The task-admission rubric is identified in Table 4.Source: E Expert Verification Rubric Template · p. 24, Table 4 · p. 24
  • Hard constraints are binary and non-compensatory, while soft constraints use 0, 0.5, or 1 scores with positive weights normalized within each question.Experts may mark evidence insufficient, triggering escalated review rather than a pass.Source: E Expert Verification Rubric Template · p. 24

F.1 Task Definition and Output Contract

MathVis evaluates whether models can turn mathematical prompts into executable, verifiable figures under separate technical, mathematical, and rubric-based checks. Its output contract withholds reference artifacts from models and treats strict success as requiring the relevant gates to pass.

F.1 Task Definition and Output Contract

  • MathVis follows a problem-driven pipeline from source problem and Gold IR through executable code, rendering, and technical, structural, visual, and expert verification.Models receive only the self-contained description and output contract, not the reference figure or program.Source: §F.1 · p. 25

F.2 Technical Gate 𝑇

  • Strict delivery keeps technical and mathematical failures non-compensatory, while retaining partial soft scores for diagnostic reporting.The observed technical pass rate counts outputs passing all four technical checks.Source: §F.1 · p. 25, §F.2 · p. 25

F.5 Rubric Coverage

  • Rubric coverage uses three independent outputs per question: pass@3 succeeds if at least one strictly passes, whereas pass^3 requires all three.The paper identifies education as the primary setting and recommends the full rubric hard gate for educational use.Source: §F.6 · pp. 26–27

G.1 Model Scorecard

  • Table 6 reports pass@1, pass@3, and pass^3 over 512 question-level outcomes, and D-PSR, T, and S-PSR over 1,536 rollout-level outcomes.pass@3 counts at least one strict pass among three outputs; pass^3 requires all three.Source: Table 6 · p. 28

G.2.1 Metric Separation

Across the 16 evaluated configurations, strict deliverability varies substantially. The paper separates technical validity from mathematical correctness and uses Gold IR to classify errors at the proposition level.

G.2.1 Metric Separation

  • Technical validity ranges from 71.55% to 94.40%, but these T values are not mathematical hard-predicate accuracy.Aggregate rates for the five predicate groups are not reported because output-level predicate records are unavailable.Source: §G.2.1 · p. 28

G.2.2 Questions and Observed Scores

  • 9.18%–30.73% is the observed strict D-PSR range across the 16 evaluated configurations on the benchmark questions.Source: §G.2.2 · p. 28

G.2.3 Qualitative Error Analysis

  • Gold IR supports proposition-level error checks, exposing relation, mapping and formula, completeness, and technical failures.Examples include reversed arrows, incorrect signs or branches, incomplete required facts, and compilation or rendering failures.Source: §G.2.3 · p. 28
Loading 2610.12954v1…