Source-linked AI summary
GGBench: A Geometric Generative Reasoning Benchmark for Unified Multimodal Models
Jingxuan Wei, Caijun Jia, Xi Bai, Xinglong Xu, Siyuan Li, Linzhuang Sun, Bihui Yu, Conghui He, Lijun Wu, Cheng Tan
TL;DR
Existing benchmarks largely separate discriminative understanding from generation, leaving integrated generative reasoning under-evaluated. GGBench uses aligned language, executable code, and diagrams to assess geometric construction; reasoning-grounded code construction achieves higher geometric correctness, interpretability, and human-rated quality than purely generative approaches.
Problem
Existing benchmarks mainly assess discriminative understanding or generation separately, while geometric problem-solving requires planning and generating diagrams through spatial reasoning.
Source: §1 · p. 2Method
GGBench evaluates geometric construction using aligned natural-language instructions, executable GeoGebra code, and rendered diagrams.
Source: §1 · p. 3, §3.1 · p. 5Results
Reasoning-grounded models that produce and execute code achieve substantially higher geometric correctness, interpretability, and human-rated quality, while a consistent gap remains versus end-to-end image generation.
Source: §4.3 · p. 11, §6 · p. 15Takeaways & Limitations
Category-wise diagnostics show that high-fidelity diagram generation alone does not guarantee correct geometric reasoning, underscoring construction-level verification.
Source: §4.4 · p. 12
Abstract
from arXiv · showhide
The advent of Unified Multimodal Models (UMMs) signals a paradigm shift in artificial intelligence, moving from passive perception to active, cross-modal generation. Despite their unprecedented ability to synthesize information, a critical gap persists in evaluation: existing benchmarks primarily assess discriminative understanding or unconstrained image generation separately, failing to measure the integrated cognitive process of generative reasoning. To bridge this gap, we propose that geometric construction provides an ideal testbed as it inherently demands a fusion of language comprehension and precise visual generation. We introduce GGBench, a benchmark designed specifically to evaluate geometric generative reasoning. It provides a comprehensive framework for systematically diagnosing a model's ability to not only understand and reason but to actively construct a solution, thereby setting a more rigorous standard for the next generation of intelligent systems. Project website: https://opendatalab-raiser.github.io/GGBench/.
1 Introduction
Existing benchmarks largely separate discriminative understanding from generation, leaving integrated generative reasoning under-evaluated. GGBench uses geometric construction to assess that integration through aligned text, code, and images.
1 Introduction
- Multimodal benchmarks add visual context but largely remain discriminative, asking models to answer or classify rather than create new content.Source: §1 · p. 2
- Existing benchmarks primarily test discriminative reasoning, while understanding and generation are often evaluated separately rather than as an integrated process.This leaves models’ ability to comprehend, reason, and generate a complex outcome unmeasured.Source: §1 · p. 2
- Geometric construction is a suitable testbed because it requires constrained-language comprehension, formal multi-step planning, and figures that satisfy the constraints.Source: §1 · p. 3
- GGBench aligns natural-language instructions, executable code, and rendered images, enabling systematic analysis and direct verification of construction steps.Figure 2 illustrates the step-by-step evaluation, with executable code supporting precise automated verification.Source: Figure 2 · p. 2, §1 · p. 3
2 Related Work
Prior work has expanded mathematical and multimodal evaluation, but holistic assessment of complete, verifiable geometric construction remains missing. GGBench connects code-based verification to aligned reasoning and visual output in geometry.
2.1 The Evolution of Mathematical Reasoning Benchmarks
- Early geometry and visual-math benchmarks broadened toward process evaluation and diagram generation, but still lack holistic, formally verifiable construction from scratch.Examples include MM-MATH [44] and Math2Visual [47].Source: §2.1 · p. 3
2.2 The Rise of Unified Multimodal Models
- UMMs combine multimodal interpretation with structured generation, yet benchmarks generally assess these capabilities separately rather than requiring a verifiable generated artifact.Source: §2.2 · p. 4
2.3 Code-based Evaluation for Reasoning Tasks
- Code-based evaluation supports structured assessment across reasoning tasks; GGBench extends it to geometry by aligning textual reasoning, code execution, and visual output.Source: §2.3 · p. 4
3 The GGBench Benchmark
GGBench evaluates unified multimodal models through geometric construction tasks, aligning text, executable code, and rendered diagrams. Its dataset and benchmark comparisons emphasize multi-step, verifiable construction across a range of geometric skills.
3.1 Overview
- GGBench assesses end-to-end geometric construction with aligned text, code, and images, while its components can separately probe reasoning steps and code generation.GeoGebra code provides machine-verifiable ground truth for geometric correctness and precision.Source: §3.1 · p. 5
3.2 Dataset construction
- The construction pipeline adapts problems into ordered GeoGebra operations, generates synchronized reasoning, code, and diagrams, then filters drafts through quality control and expert review.It retains 1,411 high-quality aligned text–code–image items.Source: §3.2 · pp. 5–6
3.3 Dataset Analysis
- Among 1,411 problems, 57.83% are medium difficulty and 56.6% are straightedge-and-compass constructions; questions average 189.83 tokens and 5.08 images.Question lengths range from 68 to 483 tokens; each problem has 3–7 diagrams, and the most common setting is five images (60.9%; 860/1,411).Source: §3.3 · p. 7
- The 3,097 category annotations span eight reasoning categories, with procedural tasks foundational and higher-order categories demanding symbolic reasoning and multi-step dependencies.Applications of Geometric Theorems and Measurement & Ratios become more frequent in the hard subset, as Figure 4 illustrates.Source: §3.3 · p. 7, Table 2 · p. 7, Figure 4 · p. 8
3.4 Comparison with Existing Benchmarks
- GGBench requires constructing diagrams that satisfy formal constraints, shifting evaluation from answer selection to constructive evidence.The benchmark unifies understanding, reasoning, and generation in a verifiable framework.Source: §3.4 · pp. 8–9
- Its 100%-aligned text, code, and image triplets tie multi-step reasoning to executable constructions for process-level and geometry-aware verification.The benchmark enforces multi-step reasoning in every sample; executable code operationalizes auxiliary objects and dependencies rather than leaving them narrated.Source: §3.4 · p. 8
- GGBench provides 100% text, image, and code support, uniquely combining full tri-modal coverage with executable code among the compared benchmarks.The comparison covers understanding, generation, multi-step reasoning, and modality support.Source: Table 3 · p. 9, §3.4 · p. 9
4 Experiment
GGBench evaluates multimodal geometric reasoning through direct image generation and code-driven construction, scoring planning and visual construction across models, task types, categories, and difficulty levels.
4 Experiment
- All models receive problem text and a reference image under shared prompts and processing, with inference temperature fixed at 0.0 and code required to follow GeoGebra syntax.The setup fixes temperature to make outputs deterministic and parallelizes inference for efficiency.Source: §4 · p. 9
4.1 Baselines
- The benchmark compares end-to-end UMMs that generate images with models that plan, produce executable GeoGebra code, and render diagrams.Qwen-Image and Seedream 4.0 provide final images without stepwise explanations; other Track A models produce multi-stage visual outputs.Source: §4.1 · pp. 9–10
4.2 Metrics
- The four-stage protocol scores planning, intermediate process, final result, and overall performance; VLM-I averages intermediate and final scores.The planning rubric scores coherence, step completeness, and geometric correctness; intermediate scoring assesses step accuracy and process consistency.Source: §4.2 · p. 10
4.3 Main Results
- Code-driven models outperform end-to-end generators on intermediate and final constructions; GPT-5 leads overall at VLM-I=57.08 and Human=83.06.The reported comparison covers both evaluation tracks and the overall scores.Source: §4.3 · pp. 10–11, Table 4 · p. 11
- GPT-5 achieves the highest code execution accuracy at pass@1=79.02, ahead of Claude Sonnet 4.5 (75.34) and DeepSeek-V3.1 (70.09).Gemini-2.5-Pro has high code-text similarity but lower Pass@1, so textual resemblance does not necessarily predict executable success.Source: Table 5 · p. 11, §4.3 · p. 11
4.4 Performance by Construction Category
- GPT-5 leads across eight construction categories, while Measurement & Ratios and Applications of Theorems score 10–15 points below simpler procedural tasks.Basic Constructions and Circle Properties yield the highest scores overall.Source: Figure 5 · p. 12, §4.4 · p. 12
4.5 Performance by Question Type
- Models generally perform best on straightedge-and-compass tasks; GPT-5 scores 72.54 there, while analytic-construction results are lower, including 51.90 for GPT-5.The reported task types are analytic construction (AC), geometric transformation construction (GTC), and straightedge-and-compass construction (SCC).Source: Figure 6 · p. 13, §4.5 · p. 13
4.6 Performance by Difficulty Level
- The evaluation spans Easy, Medium, and Hard tasks; GPT-5 scores 72.70 on Easy and 66.77 on Hard, while Qwen3-14B and GPT-4 drop more than 20 points from Easy to Hard.DeepSeek V3.1 reaches 73.57 on Easy problems.Source: §4.6 · pp. 13–14, Figure 7 · p. 14
5 Error Analysis
The error analysis identifies recurring failures in geometric reasoning, spatial relations, goal interpretation, and code execution.
5 Error Analysis
- Four common errors are theorem misuse, structural or contextual confusion, conflating construction with numeric goals, and code-level syntax or reserved-keyword failures.One illustrated case reverses the requested containment: the model draws a circle inside a square instead of a rectangle inscribed in a circle.Source: §5 · p. 14
- In the illustrated containment reversal, surface-level textual correctness still produces a construction that violates the geometric constraints.The example is presented as a failure to ground symbolic reasoning in those constraints.Source: §5 · p. 14
- The analysis examines incorrect benchmark outputs to characterize these failure patterns.Source: §5 · p. 14
6 Conclusion
GGBench evaluates geometric generative reasoning with aligned language, executable code, and rendered diagrams. Experiments report a gap between end-to-end image generation and reasoning-grounded construction, while the benchmark proposes construction-level evaluation as a foundation for future multimodal research.
6 Conclusion
- GGBench evaluates whether models can understand, reason, and construct verifiable solutions.Each problem aligns natural-language instructions, executable GeoGebra code, and rendered diagrams.Source: §6 · p. 15
- The unified protocol measures textual planning, code executability and geometric equivalence, and visual diagram fidelity.The paper reports that UMMs often fail precise geometric constraints, whereas code-generating models show higher logical coherence and geometric correctness but remain limited in visual expressivity.Source: §6 · p. 15
- The authors present GGBench as a foundation for research on grounded, verifiable multimodal intelligence and a broader approach linking language, logic, and construction.Source: §6 · p. 15
A Data Curation Prompts
The curation pipeline screens candidate problems, reformulates suitable geometry tasks, generates stepwise construction answers, and applies automated and human validation before dataset finalization.
A.1 LLM-i - Reformulation Suitability Screening
- GPT-5 makes a binary true/false decision on whether a raw problem can be reformulated as a geometric construction task, enabling triage before manual verification.The suitability prompt requests a single token without explanation.Source: §A.1 · p. 21, Figure 9 · p. 21
A.2 LLM-ii - Geometric Problem Generation
- LLM-ii rewrites pre-screened questions as construction problems, specifying given elements, target relations, and goals without revealing solutions or GeoGebra code.It also assigns difficulty, construction-type, and geometric-skill metadata.Source: §A.2 · p. 21, Figure 10 · p. 22
A.3 LLM-iii - Answer Generation
- LLM-iii generates stepwise construction reasoning aligned with executable GeoGebra code and requires snapshots to show the construction process.The answer-generation prompt also requests reasoning and object labels, with constraints on valid syntax and construction steps.Source: §A.3 · p. 22, §A.4 · p. 23, Figure 11 · p. 23
A.4 LLM-iv Prompt for Final Automatic Screening
- LLM-iv checks generated tasks and solutions for structural, syntactic, and geometric requirements, returning a binary pass/fail score with diagnostic reasoning in JSON.The prompt is used for final automatic screening before dataset inclusion.Source: §A.4 · p. 22, Figure 12 · p. 24
A.5 Human Checking & Filtering Standards
- Human annotators screen for geometric relevance, diagram–text correspondence, and code executability and consistency; unsuitable or inconsistent items are excluded.The manual process complements automated filtering and checks code execution where applicable.Source: §A.5 · pp. 23–24
A.6 Expert Validation Standards
- Expert validation checks reasoning rigor, drawing consistency, result reproducibility, presentation clarity, and coherence across content, structure, visualization, and code execution.Only items within the intended scope of multimodal geometric reasoning are retained in the finalized dataset.Source: §A.6 · pp. 24–25
- The final expert checkpoint is described as establishing mathematical soundness, visual consistency, and executable reliability for the retained dataset.Source: §A.6 · p. 25
B.1 Experimental Setups
The appendix specifies scoring procedures for textual reasoning, final diagrams, and intermediate construction steps, then standardizes inference prompts for code-generating and image-generating models.
B.1 Experimental Setups
- The setup fixes temperature at 0.0, requires GeoGebra command syntax, and evaluates models under the same prompts and pipelines.Inference is parallelized for efficiency.Source: §B.1 · p. 25
B.2 VLM-T Scoring Prompt
- VLM-T compares model construction steps with expert references for logical completeness, geometric correctness, and stepwise coherence, scoring them from 1 to 5.Equivalent strategies are accepted, and stylistic variation is ignored.Source: §B.2 · p. 25, Figure 13 · p. 26
B.3 VLM-I-Res Scoring Prompt
- VLM-I-Res judges a model’s final rendered diagram against the reference for geometric consistency and constraint satisfaction, emphasizing structure over superficial appearance.The rating rubric runs from 1 to 5.Source: §B.3 · p. 26, Figure 14 · p. 27
B.4 VLM-I-Mid Scoring Prompt
- VLM-I-Mid averages Step Accuracy and Process Consistency ratings, then rescales the result to [0, 100].Step Accuracy checks alignment between textual instructions and corresponding subfigures; Process Consistency checks whether each step coherently extends the previous figure.Source: §B.4 · p. 26
B.5 Prompt for Model Inference
- Inference uses separate standardized prompts for LLM/LRM models that generate GeoGebra code and UMMs that generate diagrams directly from text.The code track enforces valid, inspectable construction steps; the image track requests procedural descriptions without code.Source: §B.5 · pp. 27–28
B.6 Human Evaluation
The authors compare automated VLM-I scores with expert judgments using a blinded human-evaluation protocol. The scores have a high Pearson correlation with average human ratings.
B.6 Human Evaluation
- Three domain experts evaluated 100 randomly selected samples per model after standardized training, using a double-blind review process.Two experts scored each sample independently; a third arbitrated differences exceeding 1 point, and the final score averaged the expert ratings.Source: §B.6 · pp. 28–29
- Experts jointly assessed reasoning soundness, drawing accuracy, and text-image consistency on a 1–5 scale ranging from invalid results to perfect matches.Intermediate ratings distinguish major errors, partial correctness, and mostly correct results.Source: §B.6 · p. 29
- r = 0.9295 between VLM-I scores and average human ratings, indicating close alignment with human-perceived quality.The authors report this Pearson correlation in Figure 19.Source: §B.6 · p. 30
C.1 Examples across Difficulty Levels
The examples span elementary circle geometry, a rotation-and-intersection problem, and a multi-stage regular-polygon construction. They illustrate increasing demands on spatial reasoning and procedural depth.
Easy-level task
- The easy circle task constructs two chords parallel to specified diameters, testing directional consistency, auxiliary constructions, and endpoint placement.The task uses localized constraints and basic geometric primitives.Source: §C.1 · pp. 30, 33, Figure 20 · p. 33
Medium-level task
- The medium task rotates a triangle within a regular pentagon and finds an intersection, requiring chained dependencies and preservation of geometric invariants.It tests rigid-motion understanding, label consistency, and intersection logic.Source: §C.1 · pp. 31, 34, Figure 21 · p. 34
Hard-level task
- The hard task builds an inscribed square, octagon, and 16-gon from a tangent circle, demanding hierarchical decomposition and long-horizon consistency.The construction uses recursive angle bisection and repeated symmetric placement under geometric constraints.Source: §C.1 · p. 31, Figure 22 · p. 35