PaperlayerView exact revision on arXiv

Source-linked AI summary

Solving Is Not Drawing: A Benchmark for Diagrammatic Reasoning in Olympiad Geometry

Hsien Xin Peng, Anthony Kim, Alvin Li, Calvin Supasanya, Shivank Garg, Kevin Zhu

arXiv:2608.18111v1cs.AI

TL;DR

Models can solve olympiad geometry problems without reliably drawing the diagrams those solutions depend on, but this ability has not been systematically measured. This paper introduces a benchmark and evaluation suite for diagrammatic reasoning, finding a pronounced gap between solving and drawing, with diagrams compiling successfully only 36.14% of the time on average.

  • Problem

    Existing benchmarks measure whether models solve geometry problems, but do not systematically evaluate whether they can construct geometrically faithful diagrams.

  • Method

    The authors release 954 olympiad geometry problems, including a 297-problem hard subset, with solutions, human-authored Asymptote diagrams, and multimodal diagrammatic-reasoning metrics.

  • Results

    36.14% average compile success and markedly lower diagram fidelity than problem-solving performance reveal a pronounced gap between solving and drawing.

  • Takeaways & Limitations

    Strong mathematical reasoning does not imply the ability to construct accurate geometric diagrams.

  • Takeaways & Limitations

    The evaluation generates only one stochastic diagram per problem, and its automatic metrics are imperfect proxies for geometric correctness.

Abstract

from arXiv · show

Foundation models such as GPT and Claude now solve olympiad-level mathematics with remarkable proficiency, so much so that geometry problem solving has become a standard proxy for their mathematical reasoning. Yet solving a geometry problem and drawing the figure it depends on are not the same skill: progress often hinges on a faithful diagram with the right auxiliary constructions and incidences, and it is unclear that a model which reasons its way to the answer can also produce one. A growing collection of benchmarks, including MathVista, and MathVerse, measures whether models reach the correct answer, but to our knowledge, none isolate the distinct ability to construct the diagram itself, leaving this capability unmeasured. We introduce an open-source benchmark that targets this gap: 954 self-contained olympiad geometry problems, with a 297-problem hard subset, each paired with its solution and a human-authored, high-fidelity diagram in renderable Asymptote code, together with a suite of text-, code-, image-, VLM-, and constraint-based metrics for what we term diagrammatic reasoning. Evaluating current foundation models reveals a pronounced gap between solving and drawing: their diagrams are markedly less faithful, with an average compile success rate of only 36.14\%. Strong mathematical reasoning, we find, does not imply the ability to construct accurate geometric diagrams. Our benchmark and dataset can be accessed at https://huggingface.co/datasets/max98765/hard_geometry_problems_with_diagrams.

1. Introduction

The paper introduces diagrammatic reasoning as a capability distinct from solving olympiad geometry problems, because accurate solutions often depend on faithful diagrams. Its benchmark shows that foundation models’ strong mathematical reasoning does not translate into accurate diagram construction.

  • Motivation: Geometry problems often require accurate diagrams with appropriate auxiliary constructions and correct incidences, beyond textual reasoning alone.This motivates distinguishing diagram construction from problem solving.
  • Benchmark gap: Existing benchmarks measure answers or reasoning traces but do not isolate whether models can construct the diagram itself.They typically consume diagrams as input or treat them as incidental.
  • Benchmark: The benchmark contains 954 self-contained olympiad geometry problems, including a 297-problem hard subset, with solutions and human-authored high-fidelity diagrams.Each diagram is provided as an image and renderable Asymptote code.
  • Method: Asymptote code keeps diagram generation in a text modality while supporting precise, modifiable geometric constructions and direct comparison with ground truth.Its native geometric primitives are suited to compass-and-straightedge constructions.
  • Results: 36.14% average compile success marks a pronounced solve–draw gap, with diagram fidelity well below problem-solving performance.The gap persists even when models receive the full reference solution and are asked only to render a consistent figure.
  • Contributions: The authors define diagrammatic reasoning as generating geometrically consistent, high-fidelity diagrams and release metrics spanning text, code, image, VLM, and constraint evaluations.Their model evaluations show that strong mathematical reasoning does not imply accurate geometric diagram construction.

2. Related Works

Prior work generates geometry diagrams manually, through code, formal constraint languages, or multimodal models, while existing benchmarks largely evaluate problem solving rather than diagram fidelity. This work targets the unmeasured gap between textual mathematical reasoning and faithful geometric construction.

  • Diagram Generation: Geometry diagrams can be generated with GeoGebra, matplotlib, Asymptote, or formal geometry languages paired with constraint solvers.These approaches span direct manual construction, code-based rendering, and solver-based satisfaction of geometric constraints.
  • Diagram Generation: GeoUni trains a unified multimodal model with GRPO and geometry-specific rewards to generate accurate diagrams, but requires a dedicated MLLM.The passage contrasts this training-heavy approach with text- and code-based generation that may suffice.
  • Benchmark Gap: Existing geometry benchmarks measure answer correctness or reasoning traces, not whether models maintain geometrically consistent diagrammatic representations.A model may solve correctly through textual reasoning while its implied diagram is inconsistent with the geometry.
  • Evaluation Metrics: LPIPS evaluates image similarity, while BLEU, chrF++, and CodeBLEU compare code tokens, so these metrics do not directly capture geometric behavior.LLM and Vision-Language Model judges are also used to assess geometric reasoning more directly.
  • Benchmark Gap: GeoQA, Geometry3K, UniGeo, MathVista, MathVerse, and We-Math emphasize answer prediction or general visual mathematical reasoning rather than diagram fidelity.Scientific vector-graphics benchmarks such as Garg et al. and vTikZ likewise focus on general diagrams rather than theorem-driven geometry.

3. Dataset Curation

The dataset draws on web-scraped olympiad problems and solutions, retaining self-contained problems whose solutions can be reproduced from text alone. GPT-5.4-mini difficulty ratings then identify a 297-problem hard subset using a cutoff of 7 or above.

  • Problem Sources: Problems were web-scraped from olympiad exams including AMC, AIME, USAMO, USAJMO, and IMO, along with online solutions.The collection covers problems dating back to the 1980s.
  • Self-Containment: Self-contained problems have solutions reproducible from the statement alone and exclude problems dependent on input images.This keeps the benchmark accessible to text-only models and avoids errors caused by misreading an input diagram.
  • Difficulty Filtering: 297 problems comprise the hard subset after GPT-5.4-mini retained problems rated 7 or above on a 1-to-10 difficulty scale.The cutoff was chosen to preserve challenge and maintain comparability in size with other evaluation benchmarks.

4. Methodology

The methodology evaluates five models under two levels of solution guidance and measures diagram quality through complementary code-, text-, image-, VLM-, and compilation-based metrics. The guided setting anchors models to the intended construction, while the unguided setting tests diagram generation from the problem alone.

  • Models: Five models—Claude Sonnet 4.6, Gemini 3.1 Flash-lite, GPT-5.4-mini, Kimi K2.5, and GLM 5.1—generate code-based diagrams for evaluation.
  • Evaluation settings: Each model is evaluated with and without the full reference solution, allowing comparison between unguided generation and generation anchored to the intended construction.The second setting addresses the risk of unfairly penalizing valid solution paths that diverge from a single ground-truth reference.
  • Metrics: Text- and code-based metrics combine Keyword Match F1 with BLEU, chrF++, CodeBLEU, and ROUGE-L to assess code features and textual or structural overlap.Keyword Match F1 cross-references key Asymptote functions, while CodeBLEU incorporates syntactic and semantic program signals.
  • Metrics: LPIPS compares deep feature representations of rendered diagrams, with lower values indicating greater visual similarity to the ground truth.
  • Metrics: GPT-5.4-mini serves as a vision-language judge that scores each generated diagram against the ground truth for geometric similarity.
  • Metrics: Compile Success Rate measures the fraction of generated Asymptote programs that render without error, checking whether outputs are usable at all.

5. Results

Results show a pronounced gap between solving olympiad geometry problems and constructing faithful diagrams: diagram-fidelity metrics remain low despite near-saturated published problem-solving accuracy. Compile success is only weakly related to geometric quality, and providing solutions improves code similarity more than rendered fidelity.

  • Diagrammatic fidelity: Diagram-fidelity scores remain low across models: BLEU ranges from 13.9 to 19.1, CodeBLEU from 12.4 to 17.7, and VLM Critic never exceeds 53.2/100.A correctly solved problem can still yield a diagram sharing little geometric structure with the reference.
  • Compile success versus fidelity: Around 59% compile success in both settings makes Gemini 3.1 Flash-lite the most reliable compiler, but Kimi K2.5 and Claude Sonnet 4.6 achieve higher judge scores while compiling less often.Thus, the most renderable models are not the most geometrically faithful.
  • Compile success versus fidelity: Compile rate alone is a poor proxy for diagrammatic quality because renderable code can still depict the wrong figure.Figure 2 likewise shows that higher evaluable rates do not correspond to higher judge scores.
  • Effect of solution context: Providing the full reference solution modestly and consistently improves text- and code-similarity metrics, while rendered-image metrics change little and often worsen.The solution context appears to encourage reuse of recognizable textual structure without substantially improving the rendered diagram.
  • Solving versus drawing: 36.14% average compile success contrasts with near-perfect underlying problem-solving accuracy, exposing a large gap between textual geometry solving and accurate diagram construction.The comparison uses published AIME 2025 accuracies rather than newly measured problem-solving performance.

6. Conclusion

The paper concludes that foundation models’ strong mathematical reasoning does not extend to generating diagrams consistent with textual solutions. It introduces a high-quality olympiad geometry benchmark and multimodal metrics to concretely evaluate this gap.

  • Conclusion: Foundation models fail to generate diagrams consistent with textual geometry solutions, even when those solutions are provided.The paper identifies diagrammatic reasoning as distinct from mathematical reasoning and relevant to geometry problems requiring geometric awareness.
  • Conclusion: Providing the full reference solution improves text and code similarity but changes rendered-image metrics little and often negatively.Solution context raises code overlap without improving diagram fidelity.
  • Conclusion: The benchmark pairs olympiad geometry problems with human-authored solutions and ground-truth diagrams in renderable Asymptote code.It is designed to measure diagrammatic reasoning in a high-quality, self-contained evaluation setting.
  • Conclusion: The evaluation combines text-, code-, image-, and VLM-based metrics to provide the first concrete assessment of models’ diagrammatic reasoning.This metric suite measures diagram quality from multiple complementary perspectives.

7. Limitations · A. Prompts

Compute constraints and API costs limited evaluation to published problem-solving baselines and one stochastic diagram sample per problem. The VLM critic used GPT-5.4-mini to judge geometric similarity through structured criteria and a 0–10 rubric.

  • 7. Limitations: Compute constraints and high API costs prevented the authors from rerunning evaluated models on problem-solving tasks, so baselines came from published system cards.AIME 2025 performance is given as an example of a published baseline.
  • 7. Limitations: 36.14% compile success and fidelity scores reflect one diagram sample per problem, so stochastic generation may make reported results vary.The authors generated only a single diagram per problem rather than sampling multiple times.
  • A.2. VLM Critic: The VLM-as-a-judge metric uses GPT-5.4-mini to compare a ground-truth rendered image with a generated rendered image.Both images are appended to the full evaluation prompt.
  • A.2. VLM Critic: The critic evaluates geometric similarity rather than artistic style, checking objects, labels, incidences, layout, markings, and important construction elements.Minor antialiasing, line thickness, font differences, scaling, and cropping artifacts are ignored.
  • A.2. VLM Critic: The critic must return JSON containing a score, summary, missing elements, wrong geometry, extra elements, and label issues.The specified fields are score_0_10, summary, missing_elements, wrong_geometry, extra_elements, and label_issues.
  • A.2. VLM Critic: Scores of 9–10 indicate essentially identical geometry and labels, while 7–8 indicates mostly correct diagrams with minor missing or misplaced details.The rubric also defines 5–6 as recognizable core constructions with several meaningful issues.
  • A.2. VLM Critic: Scores of 3–4 indicate major geometry mismatch with some relevant structure, 1–2 mostly wrong or unreadable output, and 0 blank, non-diagram, or unrelated output.These categories complete the lower end of the VLM critic’s scoring rubric.

A.1. Experiments

The evaluations instruct foundation models to generate complete, self-contained Asymptote diagrams, using distinct prompts for settings with and without solutions or approaches.

  • Evaluation prompts: The system prompt frames models as expert data-visualization programmers specializing in research-quality Asymptote diagrams.It requires complete, self-contained Asymptote scripts and permits importing packages as needed.
  • Evaluation prompts: In the with-solution setting, evaluations follow the system prompt with a user request to create a diagram for the supplied problem.The user prompt is introduced specifically for evaluations conducted with a solution.
  • Evaluation prompts: Without a solution and approach, evaluations instead use a separate user prompt that asks models to create a diagram for the problem.This prompt is explicitly reserved for evaluations without a solution and approach.

B. Metrics

The benchmark evaluates diagrammatic reasoning through text/code matching, image similarity, VLM judgment, and compilation metrics. These measures approximate reference fidelity and usability, but code-overlap metrics cannot capture geometric relations, while compilation alone does not establish geometric correctness.

  • Text- and Code-Based Matching Metrics: Keyword Match F1 cross-references key Asymptote functions, alongside BLEU, chrF++, CodeBLEU, and ROUGE-L overlap metrics.These metrics identify direct matches between generated and ground-truth code and provide a rough indication of reference closeness.
  • Text- and Code-Based Matching Metrics: Code-overlap metrics are inexpensive and approximate reference similarity, but they cannot capture geometric relations in the diagrams.CodeBLEU adds syntactic and semantic program signals, whereas BLEU, chrF++, and ROUGE-L measure textual overlap variants.
  • Image-Based Metrics: LPIPS compares deep feature representations of rendered diagrams, with lower values indicating greater visual similarity.The metric uses normalized activation tensors across feature-map layers with learned weights.
  • VLM- and Compilation Metrics: GPT-5.4-mini judges each generated diagram from 0 to 10, rescaled to 0-100, while Compile Success Rate measures error-free rendering.Compilation captures usability rather than geometric correctness and therefore serves as a lower bound on diagram quality.

C. Dataset · C.1. Curation · C.2. Breakdown

The dataset was curated through a filtering pipeline applied to web-scraped geometry problems, and its final composition is broken down by source and difficulty. The curation process began with 1,441 raw datapoints and excluded problems that were non-geometric or not self-contained.

  • C. Dataset: The dataset section presents the benchmark’s overall organization and its curation and difficulty-breakdown components.The listed subsections cover dataset curation and the final dataset breakdown.
  • C.1. Curation: The curation process used a filtering pipeline to construct the dataset.Figure 4 provides an overview of the problems filtered during dataset curation.
  • C.1. Curation: 1,441 raw datapoints were collected through web scraping before dataset filtering.The figure caption states that the process began by web-scraping a total of 1,441 raw datapoints.
  • C.1. Curation: The filtering removed problems that were not geometric or not self-contained.These criteria are explicitly identified as exclusions in the curation process.
  • C.2. Breakdown: The final dataset is broken down according to the original problem source and difficulty score.The paper introduces a full breakdown using these two dimensions.
  • C.2. Breakdown: Table 2 reports the final dataset breakdown based on difficulty.The table is specifically titled “Final Dataset Breakdown based on Difficulty.”
  • C.2. Breakdown: Figure 5 shows the distribution of problems by difficulty score.Its caption identifies the figure as a breakdown of problems by difficulty score.
  • C.2. Breakdown: Figure 6 presents the difficulty-score breakdown across all problems.Its caption specifies that the figure covers all problems.
Loading 2608.18111v1…