GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning

Jiaqi Chen; Jianheng Tang; Jinghui Qin; Xiaodan Liang; Lingbo Liu; Eric P. Xing; Liang Lin

Source-linked AI summary

GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning

Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric P. Xing, Liang Lin

arXiv:2105.14517v3May 30, 2021cs.AI

TL;DR

GeoQA targets geometry solving that requires coordinating text, diagrams, and theorem knowledge, while prior methods relied on handcrafted rules and small datasets. The paper introduces a large annotated benchmark and NGS, which generates interpretable programs with auxiliary tasks; NGS-Auxiliary outperforms other methods, though difficult high-level cases remain.

Problem
Geometry solving requires joint reasoning over textual descriptions, diagrams, and theorem knowledge, while prior systems depended on handcrafted rules and small datasets.
Method
The paper constructs GeoQA and introduces NGS, which jointly parses text and diagrams, generates executable programs, and uses auxiliary tasks to enhance semantic representation.
Results
NGS-Auxiliary outperforms other methods on GeoQA, and experiments show superiority over other multimodal baselines.
Takeaways & Limitations
GeoQA provides a challenging benchmark with program annotations for studying explicit and interpretable multimodal numerical reasoning.
Takeaways & Limitations
A hard high-level reasoning case remains unresolved by all three evaluated models, motivating better multimodal methods.

Abstract

from arXiv · show

Automatic math problem solving has recently attracted increasing attention as a long-standing AI benchmark. In this paper, we focus on solving geometric problems, which requires a comprehensive understanding of textual descriptions, visual diagrams, and theorem knowledge. However, the existing methods were highly dependent on handcraft rules and were merely evaluated on small-scale datasets. Therefore, we propose a Geometric Question Answering dataset GeoQA, containing 4,998 geometric problems with corresponding annotated programs, which illustrate the solving process of the given problems. Compared with another publicly available dataset GeoS, GeoQA is 25 times larger, in which the program annotations can provide a practical testbed for future research on explicit and explainable numerical reasoning. Moreover, we introduce a Neural Geometric Solver (NGS) to address geometric problems by comprehensively parsing multimodal information and generating interpretable programs. We further add multiple self-supervised auxiliary tasks on NGS to enhance cross-modal semantic representation. Extensive experiments on GeoQA validate the effectiveness of our proposed NGS and auxiliary tasks. However, the results are still significantly lower than human performance, which leaves large room for future research. Our benchmark and code are released at https://github.com/chen-judge/GeoQA .

1 Introduction

GeoQA addresses the underexplored challenge of solving geometry problems from text, diagrams, and theorem knowledge. It introduces a larger annotated benchmark and an interpretable neural solver with auxiliary tasks.

Motivation: Geometry problems require jointly parsing textual descriptions, diagram elements, and theorem knowledge such as the Pythagorean Theorem.Diagrams encode information absent from text, while ambiguous references and implicit relations complicate cross-modal reasoning.
Motivation: Prior geometric solvers relied heavily on handcrafted rules, used small datasets, and produced solving processes difficult for humans to examine.These constraints limited generalization to more complex cases and reduced transparency.
Contributions: GeoQA contains 4,998 real-world Chinese middle-school multiple-choice geometry problems with executable program annotations.The programs represent numerical reasoning steps and are intended to support generalization and interpretability research.
Contributions: NGS fuses text and diagram representations through co-attention and predicts sequential programs that can be executed to obtain answers.The predicted programs expose the numerical reasoning steps for diagnosis.
Contributions: Three auxiliary tasks improve text-diagram semantic representation: diagram jigsaw location prediction, geometric element prediction, and knowledge point prediction.Experiments report that NGS with these auxiliary tasks outperforms other multimodal baselines on GeoQA.

2 Related Work

Related work spans automated geometry solving, multimodal reasoning, and self-supervised auxiliary tasks. GeoQA extends these directions with a larger benchmark and geometry-specific program annotations.

Geometry Problems Solving: GeoS was an early automated geometry system using NLP and computer vision techniques for SAT-style geometry problems.GeoQA is described as 25 times larger than the only public dataset associated with this line of work and adds program annotation.
Multimodal Reasoning: Existing multimodal reasoning methods use implicit joint encoding or domain-specific programs, but their languages generally support only elementary operations.Such operations are not directly suited to geometry’s multi-step numerical calculations and theorem knowledge.
Self-supervised Auxiliary Task: Self-supervised auxiliary tasks commonly use automatically constructed pseudo-labels for pretraining and visual-feature enhancement.Examples include image jigsaw, inpainting, and super-resolution tasks.

3 GeoQA Dataset

GeoQA is a diverse benchmark of real-world Chinese middle-school geometry problems, enriched with structured metadata and executable programs. Its program language represents operations, constants, and variables, including theorem and formula knowledge.

3 GeoQA Dataset: GeoQA contains 4,998 diverse real-world geometry problems collected from Chinese middle-school exams and supplies program annotations for each problem.The dataset is intended to improve evaluation of multimodal numerical reasoning and support applications in smart education.
3.1 Data Description: Each problem includes text, a diagram, four numerical options, an answer index, explanations, problem type, knowledge points, and an annotated program.The formal representation separates the multimodal input from answer and reasoning annotations.
3.1 Data Description: GeoQA covers angle calculation, length calculation, and other problems, with a reported corpus diversity score of 0.47.Knowledge points are normalized into 50 categories using rule-based regular expressions.
3.2 Program Representation: The executable program language models numerical reasoning with operations, predefined constants, problem variables, and process variables.Operations include basic, arithmetic, trigonometric, theorem, and formula categories; examples include the Pythagorean theorem and circle-area formulas.
3.2 Program Representation: Program interpretability comes from the ordered operations, selected constants and variables, and explicit theorem or formula applications.Reading the sequence provides a general view of the problem-solving process.

4 Neural Geometric Solver

The Neural Geometric Solver jointly understands geometric text and diagrams before generating explainable programs. Auxiliary tasks are added to strengthen multimodal understanding.

Neural Geometric Solver: NGS jointly processes textual and diagrammatic information and generates explainable programs for geometric problems.The overall architecture is presented as a neural solver augmented with auxiliary tasks.
Neural Geometric Solver: Auxiliary tasks are used to enhance the solver’s understanding of text-diagram semantics.The paper identifies these tasks as a complementary component of NGS.

4.1 The Architecture of NGS

NGS separately encodes problem text and diagrams, fuses their representations through co-attention, and decodes the resulting multimodal features into executable programs. The decoder generates program tokens sequentially, after which candidate programs are executed to obtain the answer.

Problem Encoder: The text encoder represents each problem token with a word embedding and encodes the sequence using a unidirectional LSTM.
Problem Encoder: The diagram encoder uses the first three stages of ResNet-101 to extract a feature matrix, whose parameters remain fixed during overall NGS training.
Joint Reasoning Module: NGS separately encodes text and diagrams, then feeds both representations into a joint reasoning module for cross-modal fusion.
Joint Reasoning Module: The co-attention module contains 12 self-attention units and 6 guided-attention units to fuse and align textual and diagram representations.
Joint Reasoning Module: The enhanced multimodal representation combines text features with aggregated diagram features before program decoding.
Program Decoder: An attention-based LSTM decoder generates target programs sequentially, using multimodal features to predict the next program token.
Program Decoder: During inference, NGS executes beam-generated programs step by step and selects the first successful program whose grammar and answer are valid.

4.2 Auxiliary Tasks

NGS uses auxiliary tasks to improve diagram perception and problem-level semantic representation. These tasks supervise spatial relations, geometric elements, and knowledge points relevant to solving geometry problems.

Diagram Auxiliary Tasks: Diagram pretraining uses Jigsaw Location Prediction and Geometry Elements Prediction to improve the diagram encoder.
Diagram Auxiliary Tasks: Jigsaw Location Prediction splits a diagram into m × m blocks and trains the encoder to predict shuffled blocks’ relative locations to a target block.
Diagram Auxiliary Tasks: Geometry Elements Prediction trains an N-way classifier to identify diagram elements extracted as labels from the problem text.
Knowledge Points Prediction: Knowledge points prediction adds an extra training signal by predicting one or more of about 50 knowledge points from the gathered multimodal feature.
Knowledge Points Prediction: Unlike diagram pretraining, knowledge points prediction is trained jointly with NGS using a K-way multilabel classifier and binary cross-entropy.

5 Experiments

Experiments evaluate GeoQA models using answer accuracy, comparisons across modalities and program use, beam sizes, auxiliary-task combinations, and representative cases. NGS-Auxiliary performs strongly, but difficult multimodal reasoning remains unresolved.

Experimental Results: 92.3% human accuracy with text and diagrams exceeds the 63.0% text-only result, leaving a substantial model–human gap.The comparison indicates that humans benefit strongly from jointly handling textual and visual information.
Experimental Results: Program-based multimodal models outperform program-free FiLM, RN, and MCAN systems, whose poor results indicate difficulty reasoning over complex geometry as classification.Generated programs provide explicit, interpretable operations for numerical reasoning.
Experimental Results: Adding diagrams to BERT2Prog reduces accuracy from 54.4% to 52.5%, showing that simple feature fusion does not reliably improve program decoding.The authors attribute the decline to the additional diagram disturbing the text pretrained model.
Experimental Results: 60.0% accuracy makes NGS-Auxiliary the best-performing method on GeoQA, with the highest accuracy across all problem types.It improves over Seq2Prog+Diagram by 9.1% on angle problems.
Experimental Results: Larger beam sizes increase accuracy and reduce no-result cases, reaching 63.9% accuracy at beam size 100.The usual testing beam size is 10, while beam size 1 already outperforms baselines significantly.
Ablation Study: All auxiliary tasks improve NGS, with gains of 0.5%, 0.6%, 0.9%, 1.5%, and 2.6% for GE, JL, KP, JL+GE, and all three tasks.NGS-Auxiliary combines geometry-element prediction, jigsaw-location prediction, and knowledge-point prediction.
Case Analysis: In a representative case, NGS-Auxiliary predicts the needed knowledge and generates the correct PythagoreanMinus operation, while baseline and NGS obtain no result.A second case remains unsolved for all models despite correct prediction of the proportion knowledge point.

6 Conclusion

The paper introduces GeoQA, a large-scale geometric question answering dataset with program annotations, and NGS, a neural solver for multimodal geometric reasoning. Experiments show that NGS-Auxiliary outperforms other methods on GeoQA.

6 Conclusion: GeoQA contains 4,998 geometric problems with program annotations for numerical reasoning.The annotations describe problem-solving processes and support interpretable reasoning.
6 Conclusion: NGS jointly reasons over multimodal data and generates interpretable programs for geometric problem solving.The approach is designed to combine textual and diagrammatic information.
6 Conclusion: Multiple auxiliary tasks are proposed to enhance the semantic representation of text and diagrams.These tasks support the solver's multimodal representation learning.
6 Conclusion: NGS-Auxiliary outperforms other methods on the GeoQA benchmark.The paper also reports that GeoQA remains challenging for existing models.

7 Ethical Impact

GeoQA was collected from online education websites for academic research, with copyright retained by the original websites. The work received support from multiple national, provincial, and laboratory funding programs.

7 Ethical Impact: GeoQA was collected from two online education websites and is restricted to academic research use.Copyright belongs to the original websites.
7 Ethical Impact: The work acknowledges support from national, provincial, and laboratory research funding programs.The listed support includes grants from Chinese government agencies and research institutions.