Source-linked AI summary
GeomVerse: A Systematic Evaluation of Large Models for Geometric Reasoning
Mehran Kazemi, Hamidreza Alvari, Ankit Anand, Jialin Wu, Xi Chen, Radu Soricut
TL;DR
Existing mathematical reasoning benchmarks mostly use text, leaving limited evidence about VLMs handling problems that require both text and images. GeomVerse addresses this gap with procedurally generated geometry problems spanning controllable reasoning difficulties, finding that current VLMs struggle with geometry and longer reasoning chains more than prior benchmarks suggest.
Problem
Existing mathematical reasoning datasets mostly focus on textual problems, while geometry often requires both textual and visual understanding for evaluating VLMs.
Method
GeomVerse is a synthetically generated geometry dataset designed to evaluate VLM reasoning across general and geometry-specific difficulty axes.
Results
Current VLMs are less capable in geometry than previous benchmarks suggest, struggle more with increasing reasoning depth than width, and show limited out-of-distribution gains from finetuning.
Takeaways & Limitations
GeomVerse exposes reasoning gaps that may extend beyond geometry and provides a systematic benchmark for studying VLM reasoning over text and images.
Takeaways & Limitations
The generator can produce chains rather than trees even when pbranch = 1.0.
Abstract
from arXiv · showhide
Large language models have shown impressive results for multi-hop mathematical reasoning when the input question is only textual. Many mathematical reasoning problems, however, contain both text and image. With the ever-increasing adoption of vision language models (VLMs), understanding their reasoning abilities for such problems is crucial. In this paper, we evaluate the reasoning capabilities of VLMs along various axes through the lens of geometry problems. We procedurally create a synthetic dataset of geometry questions with controllable difficulty levels along multiple axes, thus enabling a systematic evaluation. The empirical results obtained using our benchmark for state-of-the-art VLMs indicate that these models are not as capable in subjects like geometry (and, by generalization, other topics requiring similar reasoning) as suggested by previous benchmarks. This is made especially clear by the construction of our benchmark at various depth levels, since solving higher-depth problems requires long chains of reasoning rather than additional memorized knowledge. We release the dataset for further research in this area.
1 Introduction
GeomVerse addresses the limited diagnostic value of broad VLM benchmarks by procedurally generating geometry problems that require controllable multi-hop reasoning over text and images. Its evaluation finds substantial reasoning weaknesses, especially as depth increases, while identifying several robustness and fine-tuning patterns.
- Motivation: Existing geometry evaluations may show overall capability but provide little information about specific VLM strengths and weaknesses or where research should focus.
- Benchmark motivation and contribution: GeomVerse procedurally generates geometry questions requiring multi-hop mathematical reasoning over both text and image, with controllable difficulty across general and geometry-specific axes.The benchmark uses geometry to measure reasoning factors that may extend beyond geometry and support systematic VLM evaluation.
- Main findings: Current VLMs are less capable in geometry than previous benchmarks suggest, indicating immature performance for applications such as AI tutoring.The depth-controlled construction also reveals a reasoning-capacity gap that may extend beyond geometry.
- Main findings: Fine-tuning VLMs to generate complete solutions substantially improves in-distribution performance but does not generalize to out-of-distribution problems.
- Main findings: VLMs struggle more as reasoning depth increases than as reasoning width increases, while remaining relatively robust to question and image representation.
- Main findings: Fine-tuning on synthetic geometry data improves performance on real geometry questions.
2 Related Work
Related work spans VLMs for visual reasoning, datasets for multi-hop logical and mathematical reasoning, and approaches for improving multi-hop performance. GeomVerse is positioned within this literature as a geometry benchmark for more systematic analysis.
- Vision-Language Models: Recent VLMs perform promisingly on captioning, question answering, and visual reasoning, but multimodal multi-hop mathematical reasoning remains less investigated.Because VLMs are generative black boxes, measuring their comprehension and answers on multi-hop questions is important.
- Multi-Hop Reasoning Datasets: Existing datasets cover multi-hop logical and mathematical reasoning over text, while other work provides geometry datasets requiring geometric reasoning.
- Dataset comparison: GeomVerse is compared with recent and widely used multi-hop logical or mathematical reasoning datasets.The comparison uses ∼ when a dataset contains a property only to a limited extent.
- Multi-Hop Reasoning Approaches: Prior multi-hop reasoning approaches include pre-training, fine-tuning with or without explicit solution generation, in-context learning with solutions, and problem decomposition.
3 The GeomVerse Dataset
GeomVerse constructs geometry problems by mapping geometric facts and formulas onto logical-theory reasoning, with controllable depth, width, and distractors. Its backward-generation process produces solvable, automatically annotated multimodal questions across standard and non-standard shapes.
- Multi-Hop Logical Reasoning: Depth counts reasoning hops, while width counts the maximum branching in a proof; width 1 forms a chain, whereas larger widths form reasoning trees.Distractors are facts or rules outside the proof of the queried value.
- From Logical to Geometric Reasoning: GeomVerse represents given geometric values as facts and geometry formulas as rules, deriving hidden values through deductive computation.Unlike ordinary logical theories, models must supply the geometry knowledge and perform the computations required by each rule.
- Creating the GeomVerse: The generator adds distractors by supplying an intermediate value, thereby reducing the required reasoning depth while retaining irrelevant shapes and values.Questions and solutions are produced automatically from the generated facts, target output, and tracked rules.
- Creating the GeomVerse: BackwardGenerate starts from a requested output type and depth, recursively appending connected shapes and formulas so missing inputs become derivable facts.Branching is controlled by selecting one or multiple connectable input elements, while generated solutions record the required rules and computations.
- Creating the GeomVerse: To maintain quality without human review, the dataset retains only generated examples whose shapes do not overlap.Random assignments violating geometric constraints are discarded and regenerated.
4 Experiments
Experiments show that VLM performance drops sharply as geometric reasoning depth increases, with substantial gaps in generalization and robustness. Chain-of-thought finetuning improves results, but models still exhibit planning, formula, visual-understanding, and hallucination failures.
- Performance as a Function of Depth: Increasing reasoning depth causes substantial performance degradation, revealing a gap between current VLMs and human performance on multi-hop geometry.GPT4V solves mainly Depth 1 examples, while finetuned models also decline monotonically with depth; humans outperform the best models, though the problems challenge humans too.
- Performance as a Function of Depth: FT-CoT substantially outperforms direct-label finetuning across all depths, demonstrating the value of training models to generate complete solutions.The comparison also supports the quality of GeomVerse's automatically generated solutions.
- Shape Generalization: Models generalize significantly worse to test problems containing unseen shapes than to in-distribution shape combinations.The FT-CoT model was finetuned on a restricted set of shapes and evaluated separately on seen-only versus unseen-shape problems.
- Performance as a Function of Width: Increasing reasoning width affects performance less than increasing depth, although some models still decline as more independent subproblems are added.The smaller width effect may reflect that wider problems require solving multiple independent Depth 1 problems, where models perform relatively well.
- Distracting Information: Distracting shapes and values significantly reduce accuracy, even when the underlying problem is effectively reduced to a lower-depth task.Depth 1 performance drops with distractors, and the comparison isolates sensitivity to irrelevant information.
- Failure Modes: The main observed failures include computation errors, incorrect proof planning, wrong formulas, shape misunderstandings, and hallucinated values.FT-CoT makes fewer planning errors than the text-only model in the manually verified examples.
- Finetuning on GeomVerse: Finetuning on GeomVerse improves the base model, while CoT finetuning provides greater gains and benefits out-of-distribution examples.Absolute performance remains low, partly because of the small model size.
- Sensitivity to Low-Level Visual Features: Models are robust to changes in opacity and line width but not to changes in image font size.The evaluation varies one low-level visual feature at a time.
5 Conclusion
The paper introduces a procedurally generated geometry dataset for evaluating multimodal multi-hop reasoning across general and geometry-specific difficulty axes. Its analysis identifies both strengths and gaps in VLM reasoning and suggests extending the dataset with more shapes, formulas, and broader question coverage.
- 5 Conclusion: GeomVerse procedurally generates geometry questions requiring multi-hop reasoning over both text and images.The dataset supports systematic analysis of general and geometry-specific reasoning abilities.
- 5 Conclusion: The dataset can be extended with additional standard and non-standard shapes, more formulas, and problems covering a larger class of geometry questions.
A More Results: Other Axes of Difficulty
The paper evaluates additional difficulty axes beyond the main experiments and provides corresponding examples in an appendix.
- A More Results: Other Axes of Difficulty: Additional experiments examine other difficulty axes, with representative samples provided in Section D.
A.1 Standard vs Non-Standard Shapes
GeomVerse compares standard and non-standard shapes while varying image annotation and information placement. Overall performance changes little across shape sets, while finetuning improves handling of non-standard shapes.
- A.1 Standard vs Non-Standard Shapes: Overall, models show no big performance drop when moving from standard shapes to all shapes.The comparison uses depth 2 examples with medium difficulty to control another difficulty axis.
- A.1 Standard vs Non-Standard Shapes: Finetuning reverses FS-CoT’s advantage on standard shapes, indicating improved handling of non-standard in-distribution shapes.The paper explicitly attributes this change to finetuning teaching models to deal with non-standard shapes.
- A.2 More Info in Text or on Image: For FT, placing information mostly in text yields lower accuracy than placing it mostly on the image.The paper suggests this setting requires mapping textual information to image elements before reasoning.
- A.3 Image Annotation: FT performs slightly better with individual annotation, whereas FT-CoT performs slightly better with coordinate annotation.Individual annotation labels sides, angles, and shapes directly; coordinate annotation represents them through labeled coordinates.
A.4 Variablized Inputs
The paper tests whether variablized inputs and different question types affect geometric reasoning. Adding variables degrades performance, while model preferences differ across side, angle, and area/perimeter questions.
- A.4 Variablized Inputs: Variablized geometry questions require inferring a variable before deriving the requested numerical element.The study selects 17 of 68 formulas whose variablized versions require solving one additional one-dimensional linear equation.
- A.4 Variablized Inputs: Performance degrades as variables are included in the questions.The experiment variablizes selected formulas with probability ρ, adding an extra one-dimensional linear equation for the chosen cases.
- Question Types: FS-CoT Text-Only performs nearly equally across side, angle, and area/perimeter questions, with slight preference for angle and area/perimeter.The FT model finds angle questions substantially easier, followed by side questions.
- Question Types: FT-CoT gains more on side and area/perimeter questions, while side questions are easiest for this model.The paper relates this pattern partly to easier arithmetic for side questions and possible difficulty computing arcsin in some angle questions.
B Implementation Details
The implementation uses diverse synthetic training and validation data, selects checkpoints by validation performance, and standardizes few-shot demonstrations. Relaxed accuracy accounts for differences caused by intermediate rounding.
- B Implementation Details: Finetuning uses 10k training examples and 2k validation examples with randomly varied difficulty-axis parameters.Examples overlapping test-set solutions are removed from the training and validation data.
- B Implementation Details: The model is trained for 10k steps with learning rate 0.0005 and batch size 128, selecting the best validation checkpoint.Validation performance is measured every 2000 steps.
- B Implementation Details: Few-shot experiments reuse four manually selected demonstrations covering diverse depths, widths, variables, information locations, and question types.The same four examples are used across all few-shot experiments.
- B Implementation Details: Relaxed accuracy accommodates small final-answer differences caused by alternative intermediate rounding orders.The paper gives examples where rounding before versus after division produces different results.
C Sample Process for Algorithm 1
Algorithm 1 generates depth-controlled geometry problems by selecting shapes and formulas, backward-chaining derived quantities, and converting the resulting facts into a question.
- C Sample Process for Algorithm 1: The generator begins with a target shape and formula, then selects explicit inputs and assigns a random value to one provided element.In the example, a rectangle-area formula is selected and AC is provided while another side is derived.
- C Sample Process for Algorithm 1: Each subsequent step selects a connected shape and formula to derive another needed quantity, increasing reasoning depth or width.The example derives CD through a right triangle, then derives CE from a semicircle perimeter and an angle through supplementary-angle reasoning.
- C Sample Process for Algorithm 1: For the depth-3 example, the facts are LAC, PSemiCircle, and DDEF, while the query is the area AABCD.The deduction chain applies successive rules until the rectangle area is obtained.
- C Sample Process for Algorithm 1: The final facts and required extra information are rendered into a templated natural-language geometry question.The example asks for the rectangle’s area after specifying a side length, triangle and angle relations, and semicircle perimeter.
D Samples from GeomVerse
This section presents samples from GeomVerse’s test-set variations, with corresponding questions, solutions, and images.
- D Samples from GeomVerse: The section illustrates several GeomVerse test-set variations through sample questions and solutions.The corresponding images appear in Figure 15, while the questions and solutions are provided in Tables 3 and 4.
- D Samples from GeomVerse: Figure 15 contains the images corresponding to the sampled questions.
- D Samples from GeomVerse: Tables 3 and 4 provide the questions and solutions associated with those images.
E Limitations
GeomVerse is constrained by the geometry structures and deduction procedures supported by its generation algorithm, and its benchmark does not measure creative problem solving.
- E Limitations: Algorithm 1 cannot generate geometry problems outside class P, whose shapes form a tree connected through single sides or vertical angles and solved bottom-up.This excludes problems whose structure does not match that tree-based representation.
- E Limitations: Problems requiring a property of a larger triangle from two subtriangles are not generated by Algorithm 1, although such cases could be added as non-standard shapes.
- E Limitations: GeomVerse problems follow a logical deduction procedure and therefore do not assess models’ creativity in solving problems.The evaluation instead targets the ability to follow a deduction procedure.