PaperlayerView exact revision on arXiv

Source-linked AI summary

Solving Is Not Drawing: A Benchmark for Diagrammatic Reasoning in Olympiad Geometry

Hsien Xin Peng, Anthony Kim, Alvin Li, Calvin Supasanya, Shivank Garg, Kevin Zhu

arXiv:2608.18111v1cs.AI

TL;DR

Geometry benchmarks assess problem solving, but whether models can also construct faithful diagrams remains unclear. This paper introduces a benchmark for diagrammatic reasoning and finds that models’ diagrams are markedly less faithful than their problem-solving performance, with average compile success of 36.14%.

  • Problem

    It is unclear whether models that solve geometry problems can also produce faithful diagrams, a distinct capability not systematically evaluated.

    Source: §1 · p. 1, §2.1 · p. 2

  • Method

    The authors introduce an open-source benchmark of 954 olympiad geometry problems, including a 297-problem hard subset, paired with solutions, human-authored diagrams, and diagrammatic-reasoning metrics.

    Source: §1 · pp. 1–2

  • Results

    Models’ diagrams are markedly less faithful than their problem-solving performance, with average compile success of 36.14%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Strong mathematical reasoning does not imply the ability to construct accurate geometric diagrams.

    Source: §1 · p. 2

    The authors did not rerun models on problem solving and instead used published baseline accuracies.

    Source: §7 · p. 6

Abstract

from arXiv · show

Foundation models such as GPT and Claude now solve olympiad-level mathematics with remarkable proficiency, so much so that geometry problem solving has become a standard proxy for their mathematical reasoning. Yet solving a geometry problem and drawing the figure it depends on are not the same skill: progress often hinges on a faithful diagram with the right auxiliary constructions and incidences, and it is unclear that a model which reasons its way to the answer can also produce one. A growing collection of benchmarks, including MathVista, and MathVerse, measures whether models reach the correct answer, but to our knowledge, none isolate the distinct ability to construct the diagram itself, leaving this capability unmeasured. We introduce an open-source benchmark that targets this gap: 954 self-contained olympiad geometry problems, with a 297-problem hard subset, each paired with its solution and a human-authored, high-fidelity diagram in renderable Asymptote code, together with a suite of text-, code-, image-, VLM-, and constraint-based metrics for what we term diagrammatic reasoning. Evaluating current foundation models reveals a pronounced gap between solving and drawing: their diagrams are markedly less faithful, with an average compile success rate of only 36.14\%. Strong mathematical reasoning, we find, does not imply the ability to construct accurate geometric diagrams. Our benchmark and dataset can be accessed at https://huggingface.co/datasets/max98765/hard_geometry_problems_with_diagrams.

1. Introduction

This paper treats faithful geometric diagram construction as a capability distinct from solving geometry problems and introduces a benchmark to measure it. Experiments show that models’ diagrams lag their problem-solving performance.

1. Introduction

  • Existing benchmarks measure correct answers or valid reasoning traces but do not isolate diagram construction, allowing models to score well despite inconsistent or impossible diagrams.Examples include OlympiadBench, MathVista, MathVerse, and We-Math.Source: §1 · p. 1
  • The open-source benchmark contains 954 olympiad geometry problems, including a 297-problem hard subset, each paired with a full solution and a human-authored ground-truth diagram.Diagrams are available as images and renderable code.Source: §1 · pp. 1–2
  • The benchmark uses Asymptote code for diagrams and pairs the dataset with text-, code-, image-, VLM-, and constraint-based evaluation metrics.Asymptote’s geometric primitives support precise constructions, and its text-code format produces figures that are modifiable and comparable to ground truth.Source: §1 · p. 2
  • 36.14% average compile success rate, with diagram fidelity well below problem-solving performance, reveals a pronounced gap between solving and drawing.The gap persists even when models receive the full reference solution and are asked only to render a consistent figure.Source: §1 · pp. 1–2
  • Diagrammatic reasoning is defined as generating a geometrically consistent, high-fidelity diagram, a capability distinct from problem solving.The paper argues that a model that reasons correctly but draws inconsistently cannot be trusted to support geometric work, verify constructions, or provide reliable diagrams to downstream agents.Source: §1 · pp. 1–2

2. Related Works

Prior work offers manual, code-based, solver-based, and learned approaches to geometry diagram generation, but diagrammatic fidelity remains distinct from mathematical problem-solving. Existing geometry evaluations and datasets largely emphasize answers or general visual reasoning rather than fidelity to theorem-driven constructions.

2.1. Geometric Diagram Generation

  • Foundation models can generate diagram code without dedicated training, but whether its renderings faithfully represent the geometry has not been systematically evaluated.Prior approaches include manual tools such as GeoGebra (Hohenwarter, 2007), code libraries and languages such as matplotlib and Asymptote, formal-geometry solvers, and learned generation. Diffusion models can produce high-fidelity images but may sacrifice mathematical precision; GeoUni uses GRPO and geometry-specific rewards but requires a dedicated MLLM (Cheng et al., 2025; Shao et al., 2024).Source: §2.1 · p. 2

2.2. Geometry Metrics and Evaluations

  • Answer and reasoning-trace benchmarks may miss inconsistent diagrammatic representations, motivating fidelity metrics that compare generated diagrams with ground-truth constructions.Image similarity metrics include LPIPS (Zhang et al., 2018); code metrics include BLEU, chrF++ (Popovi´c, 2017), and CodeBLEU (Ren et al., 2020), which compare token distributions rather than geometric behavior. The paper also adopts LLM or VLM judging to assess geometric reasoning more directly (Zheng et al., 2023).Source: §2.2 · p. 2

2.3. Geometry Datasets

3. Dataset Curation

The dataset pairs 954 self-contained olympiad geometry problems with human-authored Asymptote diagrams, and identifies 297 problems rated as a harder subset. Curation excludes image-dependent problems and retains only problems rated at least 7/10 in difficulty.

Problem Sources

  • 954 olympiad geometry problems are paired with human-authored Asymptote diagrams treated as ground truth, with 297 retained for the harder subset.Problems came primarily from AMC, AIME, USAMO, USAJMO, and IMO exams; GPT-5.4-mini screened problems for geometry and self-containment and summarized solution approaches.Source: §3 · p. 3

Self-Containment

  • A problem is self-contained when its solution is reproducible from the statement alone; image-dependent problems are excluded to support text-only evaluation and avoid input-diagram misreading.The text-only input also prevents a misread diagram from corrupting a model’s reasoning at the outset.Source: §3 · p. 3

Difficulty Filtering

  • 297 problems rated 7 or higher on GPT-5.4-mini’s 1–10 difficulty scale form the hard subset, balancing challenge with comparability in benchmark size.The paper gives a fuller dataset breakdown in Appendix C.2.Source: §3 · p. 3

4. Methodology

The methodology evaluates five foundation models in two prompting settings, using text-, code-, image-, VLM-, and compilation-based metrics to assess generated diagrams.

4.1. Models

  • The evaluation tests Claude Sonnet 4.6, Gemini 3.1 Flash-lite, GPT-5.4-mini, Kimi K2.5, and GLM 5.1 on code-based diagram generation.Source: §4.1 · p. 3

4.2. Experiments

  • With-solution prompts supply the problem and full reference solution; without-solution prompts supply only the problem, jointly testing solution reasoning and drawing.The solution-anchored setting assesses whether models can create a diagram faithful to a given solution, avoiding penalties when models choose a different valid path.Source: §4.2 · p. 3

4.3. Metrics

  • Keyword Match F1, BLEU, chrF++, CodeBLEU, and ROUGE-L provide coarse code-overlap indicators, but cannot establish geometric correctness.The metrics compare key Asymptote functions or word, character, syntactic, semantic, and sequence overlap; formulaic boilerplate can inflate scores, while valid code using different functions may be penalized.Source: §4.3 · p. 3
  • LPIPS compares rendered-diagram features, GPT-5.4-mini judges geometric similarity, and Compile Success Rate measures how often Asymptote programs render without error.Lower LPIPS values indicate greater similarity; compilation is a basic check of whether an output is usable.Source: §4.3 · pp. 3–4

5. Results

Results show a persistent separation between strong textual problem-solving and faithful diagram construction; renderability and added solution context do not reliably close the fidelity gap.

5.1. Solving Performance Does Not Predict Diagram Fidelity

  • BLEU scores of 13.9–19.1, CodeBLEU scores of 12.4–17.7, and VLM-critic scores no higher than 53.2/100 show that solving correctly does not ensure geometric similarity.The paper characterizes diagrammatic reasoning as distinct from textual problem solving rather than a by-product of it.Source: §5.1 · p. 4

5.3. Effect of Providing the Reference Solution

  • Providing the full reference solution modestly and consistently improves text- and code-similarity metrics, but rendered-image metrics change little or often decline, leaving geometric correctness largely unimproved.The solution context shifts generated code toward recognizable reference text without resolving the diagram-construction bottleneck.Source: §5.3 · p. 4

5.4. Comparison with Baseline Performance

  • 36.14% average compile success and under 54/100 VLM-critic fidelity contrast with near-perfect published solving accuracy, showing models solve textual geometry more reliably than they construct faithful diagrams.On AIME 2025, the GPT-5 line reports 94.6% accuracy without tools and 100% with code execution; the Claude Sonnet line reports roughly 87% without tools and 100% with tools.Source: §5.4 · pp. 4–5, Table 1 · p. 5

5.2. Rendering More Is Not Rendering Better

  • Around 59% compile success for Gemini 3.1 Flash-lite does not yield above-mean VLM-critic scores, while Kimi K2.5 and Claude Sonnet 4.6 score highest despite compiling less often.Compile rate alone is a poor proxy for diagram quality because renderable code can still depict the wrong figure.Source: §5.2 · p. 4, Figure 2 · p. 5

6. Conclusion

Foundation models show a gap between mathematical problem solving and producing diagrams consistent with textual solutions. The benchmark evaluates diagrammatic reasoning using olympiad geometry problems paired with human-authored solutions, ground-truth diagrams, and multiple metrics.

6. Conclusion

  • Models fail to generate diagrams consistent with textual solutions, even when those solutions are provided.The conclusion argues that diagrammatic reasoning and geometric awareness remain to be brought up to par with textual reasoning.Source: §6 · p. 5
  • 36.14% average compile success and 46.58% average VLM-critic score reveal a pronounced gap between models’ diagram generation and problem-solving ability.The conclusion reports both averages for generated diagrams on the benchmark.Source: §6 · p. 5
  • The benchmark pairs olympiad geometry problems with human-authored solutions and renderable Asymptote ground-truth diagrams, evaluated using text-, code-, image-, and VLM-based metrics.The authors describe this as the first concrete evaluation of foundation models’ diagrammatic reasoning, to their knowledge.Source: §6 · p. 5

7. Limitations · A. Prompts

The study’s limitations stem from constrained evaluation and imperfect diagram-fidelity metrics. Its VLM critic compares generated and reference diagrams for geometric similarity using a structured scoring prompt.

7. Limitations

  • Published system-card accuracies replaced direct problem-solving evaluations, while each model generated only one stochastic diagram per problem.Compile-success and fidelity scores may vary across runs.Source: §7 · p. 6
  • Text- and code-overlap metrics can reward shared Asymptote boilerplate, while SymPy-based symbolic matching could improve evaluation reliability.The direct VLM fidelity measure may misjudge borderline diagrams because of the judge model’s visual and geometric limitations.Source: §7 · p. 6

A.2. VLM Critic

  • GPT-5.4-mini judges geometric similarity by comparing rendered ground-truth and generated diagrams.The prompt supplies both images to the judge.Source: §A.2 · p. 9
  • The critic checks geometric objects, labels, incidences, relative layout, markings, and important constructions, while ignoring minor rendering artifacts.Ignored differences include antialiasing, line thickness, small font differences, and harmless scaling or cropping.Source: §A.2 · p. 9
  • The VLM critic returns JSON with a 0–10 score, summary, missing and extra elements, wrong geometry, and label issues.The specified fields are score_0_10, summary, missing_elements, wrong_geometry, extra_elements, and label_issues.Source: §A.2 · p. 9
  • Scores of 9–10 indicate essentially matching geometry and labels, 7–8 mostly correct diagrams with minor issues, and 5–6 recognizable constructions with several meaningful issues.The scoring rubric grades similarity on a ten-point scale.Source: §A.2 · p. 9
  • Scores of 3–4 indicate major geometry mismatch, 1–2 mostly wrong or unreadable diagrams, and 0 blank, non-diagram, or unrelated output.Source: §A.2 · p. 10

A.1. Experiments

Evaluations used a detailed system prompt for generating Asymptote diagrams, with the user prompt varying between settings with and without a solution and approach.

A.1. Experiments

  • Models were instructed to produce complete, self-contained Asymptote scripts, follow diagram and formatting guidelines, and output only code.The system prompt also specified package use, canvas sizing, labels, colors, and angle-unit conventions.Source: §A.1 · p. 9
  • With-solution evaluations followed with a user prompt beginning “Create a diagram for:”, while evaluations without a solution and approach used a different user prompt with the same opening.Source: §A.1 · p. 9

B. Metrics

The section defines complementary text/code, image-, vision-language-, and compilation-based metrics for assessing generated geometry diagrams, while noting that code overlap is only a coarse indicator of geometric correctness.

Text- and Code-Based Matching Metrics

  • Keyword Match F1 and BLEU, chrF++, CodeBLEU, and ROUGE-L measure code matches or overlap, but cannot capture geometric relations and may reward boilerplate or penalize equivalent constructions.The authors retain them as cheap, coarse indicators of consistency because Asymptote code is formulaic and reuses functions.Source: B. Metrics · p. 10

Image-Based Metrics

  • LPIPS compares deep feature representations of rendered diagrams, with lower values indicating greater visual similarity.Its feature activations are weighted and normalized across spatial positions and layers.Source: B. Metrics · p. 10

VLM-Based Metrics

  • GPT-5.4-mini scores each generated diagram against the ground truth from 0 to 10, then the scores are rescaled to 0-100.The full judging prompt is provided in Appendix A.Source: B. Metrics · p. 10

Compilation Metrics

  • Compile Success Rate is the fraction of generated Asymptote programs that render without error; it measures usability, not geometric correctness, and is a lower bound on diagram quality.Source: B. Metrics · p. 10

C. Dataset · C.1. Curation · C.2. Breakdown

The dataset was curated from 1,441 web-scraped datapoints by filtering out problems that were not geometric or self-contained, then characterized by source and difficulty.

C.1. Curation

  • 1,441 raw datapoints were web-scraped before non-geometric and non-self-contained problems were filtered out.Figure 4 provides an overview of the curation pipeline.Source: §C.1 · p. 10, Figure 4 · p. 11

C.2. Breakdown

  • The dataset breakdown is organized by original problem source and difficulty score.The passage introduces a full breakdown along these dimensions.Source: §C.2 · p. 11
  • Table 2 presents the final dataset breakdown based on difficulty.The caption does not provide the table’s values.Source: Table 2 · p. 11
  • Figure 5 presents a breakdown of problems by difficulty score.The caption does not specify category counts or scores.Source: Figure 5 · p. 12
  • Figure 6 presents a breakdown of all problems by difficulty score.The caption does not specify category counts or scores.Source: Figure 6 · p. 12

Related papers

Loading 2608.18111v1…