Source-linked AI summary
TheoremQA: A Theorem-driven Question Answering dataset
Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, Tony Xia
TL;DR
TheoremQA addresses the limited evaluation of language models on science problems requiring domain-specific theorem knowledge. The paper constructs an expert-curated benchmark and evaluates 16 models with reasoning prompts, finding GPT-4 strongest while open-source models remain near random-guess performance.
Problem
Existing math word-problem benchmarks emphasize fundamental questions and provide limited evidence about models’ ability to apply domain-specific theorems.
Method
The paper builds an expert-curated theorem-driven dataset and evaluates language and code models with Chain-of-Thoughts and Program-of-Thoughts prompting.
Results
GPT-4 reached 51% accuracy with Program-of-Thoughts prompting, whereas open-source models scored below 15%, barely above the 10% random-guess baseline.
Takeaways & Limitations
TheoremQA provides a benchmark for evaluating theorem application in challenging science questions across multiple domains.
Takeaways & Limitations
The dataset excludes hard-to-evaluate formula, figure, and matrix answers, which may bias its assessment of overall ability.
Abstract
from arXiv · showhide
The recent LLMs like GPT-4 and PaLM-2 have made tremendous progress in solving fundamental math problems like GSM8K by achieving over 90% accuracy. However, their capabilities to solve more challenging math problems which require domain-specific knowledge (i.e. theorem) have yet to be investigated. In this paper, we introduce TheoremQA, the first theorem-driven question-answering dataset designed to evaluate AI models' capabilities to apply theorems to solve challenging science problems. TheoremQA is curated by domain experts containing 800 high-quality questions covering 350 theorems (e.g. Taylor's theorem, Lagrange's theorem, Huffman coding, Quantum Theorem, Elasticity Theorem, etc) from Math, Physics, EE&CS, and Finance. We evaluate a wide spectrum of 16 large language and code models with different prompting strategies like Chain-of-Thoughts and Program-of-Thoughts. We found that GPT-4's capabilities to solve these problems are unparalleled, achieving an accuracy of 51% with Program-of-Thoughts Prompting. All the existing open-sourced models are below 15%, barely surpassing the random-guess baseline. Given the diversity and broad coverage of TheoremQA, we believe it can be used as a better benchmark to evaluate LLMs' capabilities to solve challenging science problems. The data and code are released in https://github.com/wenhuchen/TheoremQA.
1 Introduction
The paper introduces TheoremQA to address the lack of challenging, theorem-driven benchmarks for language models. It evaluates 16 models with multiple prompting strategies and finds a large performance gap between GPT-4 and open-source systems.
- Motivation: Existing math word-problem datasets largely target fundamental, narrow-subject questions and involve limited domain-specific theorem knowledge.GPT-4 and PaLM-2 already exceed 90% accuracy on GSM8K, while MATH remains substantially harder.
- Dataset contribution: TheoremQA is the first theorem-driven question-answering dataset spanning university-level theorems in Math, Physics, EE&CS, and Finance.Its final release contains 800 question-theorem-answer triples curated by domain experts.
- Main results: 51% accuracy was achieved by GPT-4 with Program-of-Thoughts prompting, while ChatGPT reached 35% with the same method.GPT-4 significantly outperformed the other evaluated models.
- Main results: Open-source instruction-tuned language and code models scored below 15% accuracy, barely exceeding the 10% random-guess baseline.The paper attributes many open-source errors to ignorance of the relevant theorem.
- Evaluation: The authors evaluate a wide spectrum of 16 language and code models using Chain-of-Thoughts and Program-of-Thoughts prompting.They also examine theorem integration and multimodal input presentation.
2 Related Work
Related work progresses from arithmetic and algebraic word-problem datasets toward increasingly difficult reasoning benchmarks and increasingly capable large language models. Chain-of-Thought prompting and symbolic execution provide the immediate methodological context.
- Mathematical reasoning datasets: Earlier mathematical reasoning datasets studied arithmetic and algebraic problem solving, with later benchmarks increasing difficulty, diversity, and adversarial challenge.These datasets form the broader progression of mathematical reasoning evaluation.
- Large language models: Large language models advanced from few-shot prediction and instruction following to broader complex reasoning capabilities as model scale, data, and computation increased.The passage describes this development across GPT-3, Instruct-GPT, and later systems including GPT-4.
- Reasoning prompting: Chain-of-Thought prompting asks models to generate intermediate thought processes before producing final answers.TheoremQA extends this reasoning setup with an additional symbolic-execution step.
3 Dataset
TheoremQA is constructed by enumerating university-level theorems, expert-curating question–answer pairs, and restricting outputs to automatically evaluable answer formats. The resulting dataset covers hundreds of theorems, multiple scientific domains, and a multimodal subset.
- Dataset construction: The dataset pipeline first enumerates a broad theorem inventory and then annotates questions associated with those theorems.The process uses LLM suggestions followed by expert refinement and question collection.
- Theorem enumeration: Domain experts refined approximately 400 theorems across Mathematics, Physics, Finance, and Electrical Engineering & Computer Science.A small number were discarded because of evaluation complexity.
- Question annotation: Experts sourced, devised, and modified questions to reduce data contamination and make answers automatically evaluable.Allowed answer forms were integers, floats, lists, booleans, and multiple-choice options.
- Dataset statistics: 800 questions cover 354 theorems, including 442 Math, 146 CS&EE, 131 Physics, and 81 Finance questions.The collection also includes 51 diagram-based questions requiring visual understanding.
- Dataset statistics: Most answers are floats or integers, reducing opportunities for shortcuts associated with multiple-choice datasets.The authors describe this format as more realistic than existing multiple-choice datasets.
- Dataset examples: TheoremQA examples require applying domain-specific results such as Stoke’s theorem or properties of Wiener’s process.These examples illustrate that questions test theorem knowledge rather than only routine calculation.
- Human-level performance: Four undergraduate students scored 12/20, 15/20, 18/20, and 19/20 on a random sample of 20 questions with 24 hours and internet access.The experiment provides a rough estimate of human performance.
4 Method
The method combines standard reasoning prompts with symbolic execution, caption-based handling of diagram inputs, theorem augmentation, and answer normalization. Program-of-Thoughts derives final answers by executing generated programs.
- Method overview: The method is organized into modules for multimodal input, prompting, theorem augmentation, and answer extraction.These modules address image inputs, reasoning procedures, theorem information, and normalized outputs.
- Prompting: Chain-of-Thought prompting elicits a step-by-step thought process before the final answer.It is one of the two established prompting strategies used by the authors.
- Prompting: Program-of-Thought prompting elicits a program whose execution produces the final answer.External execution delegates computational tasks and is intended to improve reliability.
- Answer extraction: Answer extraction uses ChatGPT to identify an answer span and WolframAlpha to normalize it into a float, integer, or list.This addresses embedded and non-normalized answers in model outputs.
- Theorem augmentation: Theorem augmentation supplies retrieved or GPT-4-generated theorem descriptions as additional prompt conditions.The authors evaluate whether these descriptions change model performance.
- Multimodal input: Caption-based multimodal input appends descriptions of supplemental diagrams to the language model’s output as an additional signal.The paper applies this approach because current LLMs do not directly support such multimodal inputs.
5 Experiments
The experiments evaluate multiple instruction-tuned language and code models with CoT and PoT prompting on theorem-driven questions. GPT-4 leads substantially, while open-source models and multimodal systems remain challenged.
- Main Results: GPT-4 outperformed ChatGPT by 13% with CoT and 16% with PoT, showing the strongest performance on challenging tasks.
- Main Results: Open-source models scored 10–14% across prompting strategies, only slightly exceeding the 10% random-guess baseline.
- Program-of-Thoughts Analysis: PoT consistently improved GPT-model accuracy over CoT, with typical gains of 5–8%.
- Program-of-Thoughts Analysis: 92% of GPT-4’s generated Python programs were runnable, compared with roughly 80% for GPT-3.5 and ChatGPT and 40% for StarChat and CodeT5+.
- Theorem Augmentation: Adding theorem descriptions improved performance by no more than 1% across evaluated scenarios.
- Multimodal Questions: None of the models exceeded 10% accuracy on the multimodal subset, with captioning-related information loss identified as the primary difficulty.
- Error Analysis: Human analysis found that nearly half of GPT-4’s errors were minor calculation mistakes, whereas Alpaca’s errors mainly reflected not knowing the theorem.
6 Conclusion
The paper introduces TheoremQA and finds strong GPT-4 performance alongside persistent difficulties for open-source and multimodal models. It points toward science-focused training and specialized visual encoders as future directions.
- TheoremQA is a theorem-driven science question-answering dataset used to evaluate different LLMs.
- GPT-4 performs strongly on TheoremQA, while existing open-source LLMs struggle to achieve reasonable performance.
- The authors suggest more science-related pre-training or fine-tuning to close the performance gap.
- Multimodal science questions remain extremely challenging, and specialized visual encoding models may better represent their diagrams.
Limitations
TheoremQA has limitations in answer extraction, dataset scope, and multimodal-question performance.
- Answer extraction errors make the reported final accuracy an approximate lower bound.Some answers are not located by the extractor.
- The dataset excludes hard-to-evaluate questions whose answers are formulas, figures, or matrices.This selection may bias evaluation of overall ability.
- Existing models did not achieve reasonable performance on TheoremQA's multimodal questions.