Source-linked AI summary
Mathematical Capabilities of ChatGPT
Simon Frieder, Luca Pinchetti, Alexis Chevalier, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Julius Berner
TL;DR
The paper addresses limited coverage of advanced natural-language mathematics benchmarks and asks whether language models can support professional mathematical work. It introduces GHOSTS and miniGHOSTS and evaluates ChatGPT versions and GPT-4 with fine-grained metrics. ChatGPT is strongest at querying mathematical facts, GPT-4 additionally handles undergraduate mathematics, but advanced performance remains below graduate-student level.
Problem
Existing natural-language mathematics datasets mostly cover elementary mathematics or standard-test-like problems, offering little insight into language-model usefulness for professional mathematicians.
Method
The authors release GHOSTS and miniGHOSTS and evaluate ChatGPT versions and GPT-4 using sourced and hand-crafted prompts, ratings, and fine-grained error codes.
Results
Across advanced mathematics, ChatGPT performs best on fact retrieval and related search-engine tasks, while GPT-4 improves performance and additionally supports undergraduate-level mathematics but fails at graduate-level difficulty.
Takeaways & Limitations
For mathematically proficient users who can check correctness, (Chat)GPT can serve as a search engine or knowledge base to accelerate lookup tasks in mathematical research.
Takeaways & Limitations
GHOSTS is not yet large enough to significantly improve language models through fine-tuning because annotation is prohibitively expensive.
Abstract
from arXiv · showhide
We investigate the mathematical capabilities of two iterations of ChatGPT (released 9-January-2023 and 30-January-2023) and of GPT-4 by testing them on publicly available datasets, as well as hand-crafted ones, using a novel methodology. In contrast to formal mathematics, where large databases of formal proofs are available (e.g., the Lean Mathematical Library), current datasets of natural-language mathematics, used to benchmark language models, either cover only elementary mathematics or are very small. We address this by publicly releasing two new datasets: GHOSTS and miniGHOSTS. These are the first natural-language datasets curated by working researchers in mathematics that (1) aim to cover graduate-level mathematics, (2) provide a holistic overview of the mathematical capabilities of language models, and (3) distinguish multiple dimensions of mathematical reasoning. These datasets also test whether ChatGPT and GPT-4 can be helpful assistants to professional mathematicians by emulating use cases that arise in the daily professional activities of mathematicians. We benchmark the models on a range of fine-grained performance metrics. For advanced mathematics, this is the most detailed evaluation effort to date. We find that ChatGPT can be used most successfully as a mathematical assistant for querying facts, acting as a mathematical search engine and knowledge base interface. GPT-4 can additionally be used for undergraduate-level mathematics but fails on graduate-level difficulty. Contrary to many positive reports in the media about GPT-4 and ChatGPT's exam-solving abilities (a potential case of selection bias), their overall mathematical performance is well below the level of a graduate student. Hence, if your goal is to use ChatGPT to pass a graduate-level math exam, you would be better off copying from your average peer!
1 Introduction
The paper introduces GHOSTS to evaluate advanced mathematical abilities and professional-use cases for language models. It also proposes benchmarking ChatGPT versions and GPT-4 to identify capabilities, failure modes, and improvements over time.
- Professional mathematical use: The benchmark tests skills relevant to professional mathematics, including computation, proof completion, insight-based problem solving, and literature surveying across domains.These use cases are intended to assess whether language models can emulate tasks mathematicians encounter in daily work.
- Evaluation methodology: The authors design a thorough evaluation methodology with warning and error codes, response scoring, and comparisons with models trained for mathematical comprehension.The methodology captures failure modes and enables fine-grained reporting of model behavior.
- Dataset and benchmark: GHOSTS is introduced as a natural-language mathematics dataset for testing advanced mathematical comprehension across multiple aspects.The dataset includes problems selected partly for a high probability of being absent from ChatGPT’s training data.
- Practical implications: The analysis summarizes where (Chat)GPT may be useful for mathematicians while identifying capability limits and failure modes.The authors characterize this analysis as a mathematical model card of strengths and weaknesses.
- Model comparisons: The study evaluates ChatGPT’s 9-January-2023 and 30-January-2023 versions and investigates performance increases brought by GPT-4.The January 30 version was released with claims of improved factuality and mathematical capabilities.
2 Related Work
Prior natural-language mathematics benchmarks largely emphasize elementary reasoning, while existing graduate-level datasets use representations tailored to proof graphs. The paper positions its benchmark as broader and less reference-centric than these alternatives.
- Approaches to mathematical reasoning: Automated mathematical reasoning also includes classical symbolic approaches and newer supervised-learning or language-model methods.The paper notes that classical symbolic approaches have reached a plateau, while learned methods target mathematical relationships in different representations.
- Natural-language mathematics benchmarks: Recent LLMs were commonly evaluated on elementary datasets such as MathQA and GSM8K, leaving advanced natural-language mathematical reasoning less covered.The related work notes that PaLM, GPT-3, and fine-tuned BERT variants achieved limited performance on these simpler benchmarks.
- Advanced benchmark examples: Minerva extended evaluation to the harder MATH competition dataset and achieved roughly 50% there.Minerva was trained partly on MathJax-containing websites and arXiv preprints.
- Graduate-level datasets: NaturalProofs and NaturalProofs-Gen cover graduate-level mathematics using annotated theorems and proof graphs extracted from mathematical resources.Their evaluation schemas are strongly tailored to reference-graph representations.
- Distinction from prior work: GHOSTS is not reference-centric and therefore supports evaluation of proofs beyond the reference-graph representation, including computational proofs.This broadens the types of mathematical reasoning that can be assessed.
3 GHOSTS and miniGHOSTS Dataset
GHOSTS combines sourced and hand-crafted prompts to evaluate mathematical reasoning across difficulty, question type, and out-of-distribution dimensions. miniGHOSTS provides GPT-4 with a representative 170-prompt subset, while the evaluation records ratings and fine-grained error information.
- Dataset scale and evaluation: The authors collected 709 prompts and evaluated ChatGPT versions and GPT-4 on subsets while recording ratings, error types, output lengths, and prompt-engineering stability.GPT-4 was evaluated on a 170-prompt miniGHOSTS subset selected to match each source file’s mean rating and standard deviation.
- Subdataset composition: GHOSTS is divided into six subdatasets covering graduate texts, proof gaps, olympiad problems, symbolic integration, MATH problems, and search-engine aspects.The table summarizes the component files, prompt counts, and associated tags.
- Coverage dimensions: The dataset spans mathematical difficulty from elementary arithmetic through olympiad problems, question types from fact retrieval to proof completion, and three out-of-distribution dimensions.The dimensions are encoded as M_i, Q_i, and D_i categories.
- Scope and assumptions: Non-text-based questions, such as diagrams without explanatory text, are excluded because the models’ input was purely textual.The benchmark instead categorizes prompts using textual difficulty, question-type, and distributional dimensions.
- Evaluation format: The rating key compresses model capability into a one-dimensional score from 1 to 5, while warning and error codes provide finer-grained information.The dataset requires a correct answer from the initial prompt without subsequent interaction.
- Human effort: Hand-crafted prompts were required because advanced mathematical insight was needed both to create prompts and to evaluate non-token-level answers such as proofs.The authors report 1,636 prompt evaluations requiring several hundreds of person-hours.
4 Results
The two ChatGPT versions perform similarly overall and generally fail the 3.5 passing threshold on most subdatasets, whereas GPT-4 performs substantially better but remains imperfect on graduate-level tasks. ChatGPT is strongest on factual and gap-filling questions, while proof-based graduate exercises and complex symbolic calculations remain difficult.
- Passing threshold: For a 3.5 passing threshold, both ChatGPT versions fail on a majority of subdatasets, while GPT-4 passes most miniGHOSTS files but not graduate-level or Olympiad problem-solving files.The 3.5 threshold is defined as the midpoint between the lowest and highest ratings.
- ChatGPT results: 3.20 average rating (standard deviation 1.23) for 9-January-2023 ChatGPT, with particularly poor performance on graduate-level proofs, olympiad problems, and complex symbolic calculations.Prompt engineering only slightly improved results on these complex questions.
- Task-specific strengths: ChatGPT generally exceeds 3.5 on tasks requiring factual statements or filling gaps, while its context recognition and output notation are usually strong.Prompt length has no clear effect on ratings.
- Comparisons with mathematical models: 2.51 average rating (standard deviation 0.87) for 9-January-2023 ChatGPT on the sampled integral dataset, compared with nearly 100% correct solutions reported for the reference model.GPT-4 reached 3.50 average rating (standard deviation 1.43) on the corresponding miniGHOSTS subset.
- GPT-4 output characteristics: GPT-4 nearly doubles the percentage of perfect ratings and approximately halves ratings of 1 or 2 relative to the ChatGPT versions, although its answers are longer and can reduce readability.GPT-4 also remains vulnerable to mistakes in seemingly simple calculations.
5 Conclusion
The evaluation finds that (Chat)GPT is not consistently reliable for advanced mathematical proofs or calculations, although it sometimes produces insightful proofs. Its most useful role is as a mathematical search engine or knowledge base for proficient users who can assess its output.
- (Chat)GPT is not yet ready to deliver high-quality proofs or calculations consistently, although it produces insightful proofs in a few cases.
- (Chat)GPT’s capabilities generally decline as mathematical difficulty increases.
- (Chat)GPT falls short of models trained for single mathematical tasks, but offers greater flexibility across mathematical domains.
- For mathematically proficient users, (Chat)GPT can function as a search engine or knowledge base to accelerate lookup tasks in mathematical research.
- The GHOSTS dataset is sufficiently comprehensive for evaluation and comparison but too small to significantly improve models through fine-tuning.
B Dataset Creation
GHOSTS organizes advanced mathematical prompts across question types, difficulty levels, and out-of-distribution characteristics. Its records also capture model outputs, ratings, error and warning codes, comments, and subject classifications.
- Mathematical difficulty: The dataset spans ascending difficulty from elementary arithmetic and symbolic integration to undergraduate exercises and mathematical olympiad problems.
- Question types: GHOSTS categorizes prompts by review, overview, computational, proof-based, and proof-completion questions.
- Out-of-distribution design: Its prompts include nontrivial encodings, succinct solutions, and spoken-dialogue formulations intended to reduce the likelihood that models encountered them during training.
- Scope: The dataset’s exhaustive coverage is limited because constructing all attribute combinations would require 60 manually curated subdatasets, while comprehensive MSC evaluation would require a community effort.
- Record format: Each data point records the prompt, raw output, 1–5 rating, optional error and warning codes, comments, and MSC classifications.
- Subject classification: MSC classifications cover 78 mathematics subject codes and can help users identify areas where the model performs better.
B.4 Data Collection and Labeling Policies
The study uses standardized ratings and fine-grained error and warning codes to evaluate whether model responses understand prompts, answer correctly, and avoid problematic presentation.
- Ratings: Responses receive ratings from 1 to 5, ranging from failure to understand the query to a completely correct answer.
- Error codes: Error codes distinguish missing information, wrong statements, incorrect computations, faulty logic, unsupported operations, and related failure modes.
- Logical errors: The logic category includes unrelated proof steps, unsupported inferences, circular arguments, and rule violations.
- Rating policy: Intermediate ratings require error codes, while perfect scores require no errors and lowest scores may omit them.
- Warning codes: Warning codes identify withheld essential information, rambling, unrelated information, strange behavior, and unrequested notation changes.
B.5 Mitigating Human Errors
The authors use multiple procedural safeguards to reduce copying, formatting, encoding, comparison, and assessment errors. They also document evaluation scope and acknowledge limits related to model identification and coverage.
- Data integrity: Authors used repeated dataset checks and automatic consistency checks to mitigate copy-paste errors and missing or mismatched records.
- Assessment checks: Random reassessments and statistical checks were used to identify problematic ratings and unusual prompts.
- Caveat: The authors state that their safeguards are not exhaustive and were selected within a fixed time budget.
- Assessment procedure: Prompts were assigned according to difficulty, with more senior mathematicians receiving more difficult prompts.
- Evaluation scope: The evaluation involved 1,636 advanced-mathematics prompt evaluations performed by graduate-level researchers.
- Dataset checks: Some prompts from the MATH dataset were found to have subject mismatches, such as an arithmetic problem in the Counting and Probability file.
- Model access: Using the GUI web interface prevents precise control over model versions and API settings, and no official document links the interface to those versions.
D Further Results
Prompt engineering was tested on Olympiad-Problem-Solving questions by asking ChatGPT to reason step-by-step or by making the task more explicit. It slightly improved overall ratings while reducing some computational errors but increasing faulty-logic errors.
- About 20% of questions were prompt-engineered with step-by-step instructions or explicit “Prove that...” and “Show that...” formulations.
- Prompt engineering decreased wrong statements and computations, corresponding to error codes e2, e3, and e4.
- Prompt engineering increased errors rooted in faulty logic, corresponding to error code e5.
- Overall, prompt engineering improved the average rating only slightly for the 9-January-2023 model.
D.2 Qualitative Analysis of Subdatasets on ChatGPT 9-January-2023
The 9-January-2023 ChatGPT model showed uneven qualitative performance: it handled some definitions and basic concepts, but struggled with advanced proofs, symbolic computation, arithmetic, and unconventional constraints.
- Grad-Text: ChatGPT performed best on simple set-theory and logic questions from the first chapter of Munkres’ Topology.
- Grad-Text: On other graduate-text materials, ChatGPT performed substantially worse and could be deceptively confident while students used it for study.
- Holes-in-Proofs: ChatGPT recognized many well-known results in proof-completion tasks, but its algebraic manipulations were inconsistent, failing even on simple operations.
- Olympiad-Problem-Solving: Olympiad-Problem-Solving was the poorest-performing subdataset: 75% of scores were 2, with no rating of 5 and only one rating of 4.
- Olympiad-Problem-Solving: ChatGPT often mishandled induction, unconventional puzzles, and unusual constraints in Olympiad-Problem-Solving questions.
- Symbolic-Integration: In symbolic integration, ChatGPT frequently produced incorrect concrete computations and sometimes confidently claimed that elementary integrals lacked closed forms.
- MATH: In Algebra and Probability questions, reasoning was often correct, but error code e4 occurred 36% of the time, commonly involving standard arithmetic operations.
- Search-Engine-Aspects: For search-engine and definition-retrieval tasks, ChatGPT usually knew theorems and recited most definitions correctly, though it sometimes stated theorems inaccurately or confused senses of terms.
D.3 (Chat)GPT’s Confidence
The paper examines ChatGPT’s confidence and presents figures organizing performance by mathematical fields and error or warning codes. These displays emphasize both uncertainty calibration and the limits of cross-field comparisons.
- Confidence: ChatGPT typically expressed high confidence even when its mathematical response was wrong, unlike Codex, which voiced doubt.
- Figures: Figure 4 plots average ratings by mathematical field, colors MSC-code occurrence, shows codes with at least five occurrences, and includes 95% confidence intervals.
- Figures: The Figure 4 ranking does not indicate field complexity because the exercises are not equally complicated across fields.
- Figures: Figure 5 encodes counts as annotations and relative frequencies as colors for error and warning codes across subdatasets and files.
D.5 Comparison of (Chat)GPT Versions
The 30-January-2023 ChatGPT version performs very similarly to the 9-January-2023 version, while GPT-4 strictly dominates both in performance but produces more rambling warnings.
- The 30-January-2023 version performs very similarly to the 9-January-2023 version, with differences in rating, error-code, and warning-code distributions.These distributions are compared in Figure 6.
- GPT-4 strictly dominates the ChatGPT versions in performance.GPT-4 also provides contextual explanations and often gives useful, correct pointers to relevant theorems.
- GPT-4 receives significantly more w2 warning codes because its greater verbosity can become rambling.The optimal verbosity level depends on the mathematical background of the user.
- Figure 7 compares the three models’ rating percentages across the different subdatasets.The 9-January-2023 and 30-January-2023 models are evaluated on GHOSTS, while GPT-4 is evaluated on miniGHOSTS.
- Figure 8 compares output lengths using intervals containing 20% of prompts and 95% confidence intervals as error bars.The figure also relates output verbosity to achieved rating.
- Figure 9 compares prompt lengths using intervals containing 10% of GHOSTS prompts and 95% confidence intervals as error bars.
E Datasheet for the GHOSTS Dataset
The GHOSTS datasheet documents a dataset created to address the limited coverage of natural-language mathematics benchmarks and to evaluate mathematical assistance in professional use cases. GHOSTS contains evaluated mathematical prompts and model outputs, with a 170-prompt subset forming miniGHOSTS.
- Existing natural-language mathematics datasets mostly cover elementary mathematics or resemble standard tests, leaving professional mathematicians’ daily tasks insufficiently represented.This gap makes it unclear whether language models can help mathematicians in practice.
- The dataset was created by the authors of the work and received no associated grant or funding.
- GHOSTS consists of natural-language mathematical prompts paired with one or more (Chat)GPT outputs and fine-grained evaluations by the authors.
- GHOSTS contains 709 prompts, while miniGHOSTS contains a selected 170-prompt subset.Twenty-four GHOSTS questions also received light prompt-engineering variations.
Q7. Does the dataset contain all possible instances or is it a sample (not necessarily random)
GHOSTS is a curated dataset rather than a complete collection of mathematical instances. It spans 78 MSC codes, but cost and evaluation constraints prevent representation of all fields, question types, and dimensions of mathematical behavior.
- GHOSTS is not fully representative of all mathematical fields, question types, or dimensions of mathematical behavior.The authors attribute this boundary to the prohibitive cost of human evaluation, which cannot be fully automated away.
- The dataset covers mathematical questions from 78 different MSC codes.
- The study collected and constructed prompts from various sources and captured and rated the corresponding model outputs.
- The MATH and Symbolic-Integration subdatasets were randomly sampled from existing datasets across different files.
Q49. Have any third parties imposed IP-based or other restrictions on the data associated
The dataset is self-contained, but copyright restrictions limit public release of some prompts. Author-created prompts are covered by CC BY-NC 4.0, while other prompts remain subject to intellectual-property restrictions.
- IP restrictions apply only to prompts not solely created by the authors; author-created prompts are under CC BY-NC 4.0.