Source-linked AI summary
G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Model
Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, Lingpeng Kong
TL;DR
The paper addresses the limited investigation of geometric reasoning by MLLMs, which struggle to understand basic geometric elements and their relationships. It constructs Geo170K by enriching existing data with geometry-specific visual-text information, then develops G-LLaVA. G-LLaVA surpasses GPT-4-V on MathVista geometry problems with only 7B parameters.
Problem
MLLMs have limited ability to solve geometric problems because they struggle to comprehend basic geometric elements and their relationships.
Method
The paper uses geometry characteristics and text-only LLMs to enrich existing datasets with geometric image captions and question-answer data, producing Geo170K and G-LLaVA.
Results
G-LLaVA surpasses GPT-4-V on MathVista geometry problems with only 7B parameters.
Takeaways & Limitations
The work provides a geometry-focused multimodal dataset and model for improving MLLMs’ geometric problem-solving ability.
Abstract
from arXiv · showhide
Large language models (LLMs) have shown remarkable proficiency in human-level reasoning and generation capabilities, which encourages extensive research on their application in mathematical problem solving. However, current work has been largely focused on text-based mathematical problems, with limited investigation in problems involving geometric information. Addressing this gap, we aim to enable LLMs to solve geometric problems by understanding image input. We first analyze the limitations of current Multimodal Large Language Models (MLLMs) in this area: they struggle to accurately comprehending basic geometric elements and their relationships. To overcome these challenges, we take advantage of the unique characteristics of geometric problems (such as unique geometric logical form, and geometric scalability) and the capacity of the textual LLMs to build an enriched multimodal geometry dataset based on existing data. The augmented dataset, Geo170K, contains more than 170K geometric image-caption and question-answer pairs. Utilizing our constructed Geo170K dataset, we develop G-LLaVA, which demonstrates exceptional performance in solving geometric problems, significantly outperforming GPT-4-V on the MathVista benchmark with only 7B parameters.
1 Introduction
Geometric reasoning remains difficult for MLLMs because they often misinterpret geometric figures, while existing datasets provide limited scale, descriptions, and solution diversity. The paper addresses these gaps with Geo170K and G-LLaVA, achieving strong benchmark results.
- MLLMs struggle to accurately comprehend geometric figures, limiting their ability to solve geometric problems.
- Existing geometric datasets are small, lack image descriptions, and cover limited problem-solving methods.
- Geo170K combines around 60,000 geometric image-caption pairs with more than 110,000 question-answer pairs.
- Geo170K is 28 times larger than GeoQA+, expanding coverage of geometric problems.
- G-LLaVA-13B outperforms LLaVA-13B by 27.4 on the GPS minitest split of MathVista.
- G-LLaVA-7B surpasses GPT4-V on geometry problem-solving questions.
2 Related Work
Related work spans multimodal language models, geometric problem-solving datasets, and LLM-based data generation. Despite progress, state-of-the-art MLLMs and existing geometry datasets retain important limitations.
- Multi-Modal Large Language Model: MLLMs can synthesize visual descriptions and dialogue but still struggle with geometric problems involving diagrams and figures.
- Geometry Problem Solving: Geometry problem solving is a challenging visual mathematical reasoning task studied through progressively enhanced datasets and methods.
- Geometry Problem Solving: Current geometry datasets remain limited in scale, and traditional models trail methods using LLMs for mathematical word problems.
- Data Generation via LLM: Prior work has used pretrained language models and ChatGPT to generate or improve training datasets.
3 Observation
The paper observes that even strong MLLMs have difficulty identifying simple geometric shapes and elements. It motivates geometry-specific data generation by contrasting these failures with the clear relationships in geometric diagrams.
- Most state-of-the-art MLLMs have difficulty comprehending geometric figures, even when the figures are simple.
- The example data pairs textual geometry questions and answers with generated descriptions of the corresponding diagram.
- Smaller MLLMs such as LLaVA1.5 and MiniGPT4 have greater difficulty identifying geometric shape types accurately.
- Inaccurate diagram interpretation may be a major cause of failures in solving geometric problems.
- Geometric diagrams typically contain clear, well-defined relationships among their elements that can support dataset development.
4 Geometric Data Generation
The paper identifies three limitations in existing geometry datasets: insufficient volume, missing diagram descriptions, and limited diversity of solution methods. It responds by constructing multimodal geometry data from existing resources.
- Existing geometry datasets contain only a few thousand examples at the largest scale.
- They lack detailed descriptions of geometric images.
- They provide limited diversity in problem-solving methodologies and answer pathways.
- The proposed approach uses geometry characteristics to construct a multimodal geometry dataset from existing datasets.
4.1 Geometric Cross-Modal Alignment Data
The paper generates geometric image captions from human-labeled question-answer pairs using text-only ChatGPT, supporting cross-modal understanding of diagrams.
- Text-only ChatGPT creates geometric image descriptions from human-labeled QA pairs through inverse information recovery.The captions are intended to support alignment between image and text modalities.
- The captions describe geometric shapes, lines, points, and their relationships by interpreting human-labeled logical forms.
- The generated diagram descriptions provide the basis for producing additional contrastive question-answer pairs.
Contrastive QA Pairs for Basic Elements
The contrastive QA generation process converts labeled geometric logic forms into detailed diagram information and tests recognition of basic elements and relationships.
- The logic representation encodes triangle similarity, side lengths, point-line incidence, and perpendicular relationships.
- The generation process first translates human-labeled logic forms into detailed information items and a diagram summary.
- It then generates contrastive QA pairs from the resulting information and diagram summary.
- Questions test whether specific geometric elements exist and whether stated relationships accurately describe the diagram.
4.2 Geometric Instruction Data
Geometric instruction data expands existing QA pairs with alignment, equation solving, value scaling, reformulated conditions, and paraphrased language to improve geometric problem solving and generalization.
- 4.2 Geometric Instruction Data: Alignment improves diagram interpretation, but the aligned model remains limited at solving geometric problems.
- 4.2 Geometric Instruction Data: The resulting instruction dataset contains more than 110k QA pairs and is described as the largest public geometric QA dataset available.
- 4.2.1 Equation Solving (ES): Equation Solving replaces specific values with unknown variables and prompts the LLM to construct solutions by solving equations.This abstraction focuses attention on underlying mathematical concepts and relationships rather than fixed numerical values.
- 4.2.2 Value Scaling (VS): Value Scaling changes length values while preserving correctness when all lengths in the same diagram are scaled simultaneously.The passage distinguishes lengths from quantities such as angles, which cannot be scaled in the same way.
- 4.2.3 Re-Formulating Condition as Unknown (RCU): Re-Formulating Condition as Unknown reverses questions to ask for values originally present in the conditions and retains only correctly answered generated data.The strategy repeatedly exposes the model to dependencies among variables, equations, and solutions.
- 4.2.4 Sentence Paraphrase (SP): Sentence Paraphrase varies both questions and answers to expose the LLM to broader phrasing and language variation.The stated goal is greater robustness to diverse sentence structures and differently phrased questions.
5 Model Architecture and Training
G-LLaVA uses a LLaVA-based vision-language architecture and trains it in two phases with language-modeling loss, while the examples illustrate geometric reasoning transformations and outputs.
- The model combines an LLM, a pretrained vision transformer image encoder, and a projection layer that maps visual features into the LLM dimension.
- During inference, image features become visual tokens, are projected into the LLM embedding space, and are concatenated with text embeddings.The resulting multimodal sequence is provided to the LLM for next-token generation.
- A railway-barrier example applies similar triangles to relate the arms’ lengths and vertical displacement, yielding y = b×d/a.
- With arm lengths 1.0 and 16.0 units and a 0.5-unit descent, the example computes an 8.0-unit rise for the longer arm.
- The training procedure has geometric visual-language alignment followed by geometric instruction tuning.
- The language-modeling objective uses the figure, input sentence, target sentence tokens, and target-output length.The notation identifies F as the model, I as the geometric figure, and S_in and S_tar as input and target sentences.
6 Experiments
Experiments evaluate G-LLaVA on MathVista geometry problems and GeoQA, comparing it with multimodal and conventional baselines across settings. Results show gains from Geo170K-based training, geometric alignment, and broad problem coverage.
- 6.2 Main Experiment: G-LLaVA-7B surpasses GPT4-V on geometric problem-solving questions in the MathVista minitest benchmark.
- 6.3 Comparison with Conventional Methods: G-LLaVA improves Top-1 accuracy over conventional state-of-the-art methods on GeoQA and exceeds their Top-10 accuracy.The evaluation reports Top-1 accuracy, whereas several prior studies report Top-10 accuracy.
- 6.4 Performance Across Problem Difficulties: G-LLaVA consistently outperforms baseline models across problem difficulty levels measured by the number of reasoning operations.OP denotes the number of operations or reasoning steps needed to solve a problem.
- 6.5 Performance Across Different Types of Questions: G-LLaVA performs better than baseline models on angle, length, and area questions.
7 Conclusion
The paper addresses limitations in multimodal geometric problem solving by enriching training data with LLM-generated information. Using Geo170K, G-LLaVA outperforms GPT-4-V on MathVista geometry problems with 7B parameters.
- The paper enriches geometric training data with LLM-generated information, producing the Geo170K dataset.
- G-LLaVA outperforms GPT-4-V on the geometric split of MathVista with as few as 7B parameters.